At 10:46 AM 4/9/2007 -0400, A.M. Kuchling wrote: >On Sat, Apr 07, 2007 at 12:44:56PM -0400, Phillip J. Eby wrote: > > I don't know whether this will actually solve any problems the cheeseshop > > itself is having; it may be that ill-behaved web spiders are at fault, or > > something else altogether. > >In this recent case, two different spiders were crawling the wiki very >quickly, the machine's load average was in the 70s, and the >out-of-memory killer was killing off PostgreSQL processes. > >I don't think the load caused from people running easy_install is >especially high -- it's certainly not a source of problems -- but >making static pages would still be good to make mirroring the package >archive more useful. Right now people could mirror >http://cheeseshop.python.org/packages/, but there's nothing there for >easy_install or for human readers; it's just a tree of package >directories.
Hm. Well, actually, if that directory structure were something I could code to, easy_install could sure as heck be *made* to use it. The only thing easy_install couldn't get from it was external links to downloads, SVN versions, etc. Notice, for example, that if you use "easy_install -f http://cheeseshop.python.org/packages/source/s/simple_json/ simple_json", easy_install won't look at the main package index, but just download directly. So an automated form of that calculation could easily be added to easy_install. What I had in mind for an easy_install mirror, however, was a script that would just create a /packagename/index.html file with links gathered from all versions of a package on the original Cheeseshop, and with packagename generated as a setuptools "safe name" in lower case. This pattern would allow easy_install to find every possible relevant link in just one (static) web hit. _______________________________________________ Distutils-SIG maillist - [email protected] http://mail.python.org/mailman/listinfo/distutils-sig
