On Fri, Jun 18, 2010 at 6:44 PM, Ian Bicking <[email protected]> wrote: > With all the reliability discussion, I thought I'd offer a kind of > counterproposal, that we rewrite PyPI to use App Engine. > > Of course, this means writing code, etc., but I believe this is a reasonable > goal. I think if "we" (Catalog-SIG? PyPI maintainers?) committed to using > such an implementation (assuming it was of good quality) that we could find > people (probably not on this list) to write and maintain the code. People > have already rewritten PyPI a couple times, but no one knows what exactly to > *do* with the rewrite so they haven't gone anywhere. And PyPI is not a > particularly complicated application. I think we can set the bar high on > the implementation quality and that people will meet it, so long as they > know their effort won't be in vain.
Out of curiosity : have you ever worked with the current implementation ? I have hard time to understand why some people say it's hard to work with it, I don't think its a valid argument. > > Why App Engine? The primary reason I'm proposing it is because it will be > much easier to manage. If it runs out of memory it won't bring down a > machine. If new people maintain the system it's easy to describe how to do > deployments, for instance. It's easy for people to install their own PyPI > instances for development and to generate patches. Hosted services can have > downtimes of course, but unlike currently there are other people (the App > Engine maintainers) who will resolve those problems. There's still a class > of bugs like badly indexed tables or weird locking issues that could bring > PyPI down and "we" would have to fix it, and with a rewrite there's more of > a risk of that, but... it'll just take some testing to make sure things are > okay. > > In terms of cost, I expect we can get free hosting, and packages can be > stored directly in the data store. That doesn't preclude using a CDN like > CloudFront, but that can be handled separately. Also since the index just > links to packages, packages can be incrementally uploaded to a CDN. Even if I don't think its a priority in our concerns (community mirrors come first), I wouldn't mind having the main PyPI UI in GAE. Although, if PyPI was to be ported to GAE, couldn't we reuse the existing code instead of rewriting from scratch ? we would just have to rewrite the DB layer. > Besides a commitment to using the code (which I think is really important to > motivate people), a scrubbed dump of the database would be really helpful > for development. I know we've passed around complete dumps to people, but > it contains private information so we can't put it up publicly which creates > a speed bump for developers. Private information could be easily removed from those dumps; But I don't think it's so helpful since you have all the .sql scripts to create your own DB. But we could add a script to create some sample data on the top of those scripts. > > > Linkage... > A buzz post where I asked about it: > http://www.google.com/buzz/ianbicking/BRWDjsMCGWQ/I-like-the-original-proposal-move-PyPI-stuff-into > > A PyPI *mirror* written for App Engine: > http://pypi.appspot.com/ > > A PyPI implementation in Django (one is a fork of the other?), > database-backed (would take some work to get it on App Engine): > http://pypi.python.org/pypi/djangopypi/ > http://github.com/benliles/chishop > > > -- > Ian Bicking | http://blog.ianbicking.org > > _______________________________________________ > Catalog-SIG mailing list > [email protected] > http://mail.python.org/mailman/listinfo/catalog-sig > > -- Tarek Ziadé | http://ziade.org _______________________________________________ Catalog-SIG mailing list [email protected] http://mail.python.org/mailman/listinfo/catalog-sig
