[Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. I intend to post the proposal to the PSF board (of which I'm a member, in case you shouldn't know) and to have it vote

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread M.-A. Lemburg
Mathieu Leduc-Hamel wrote: To continue the discussion about a rewrite or a cleanup of the Pypi codebase, I'm from Montreal-Python usergroup and I'm say that yes at the first the current codebase of pypi seem to be very unclear and difficult to maintain. But it's not an impossible mission

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Michael Crute
On Tue, Jun 15, 2010 at 7:49 AM, M.-A. Lemburg m...@egenix.com wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. What about a set of volunteer mirrors of PyPi

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Steven D'Aprano
On Tue, 15 Jun 2010 09:49:03 pm M.-A. Lemburg wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. I think the most important missed fact is, just how unreliable

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Steven D'Aprano wrote: On Tue, 15 Jun 2010 09:49:03 pm M.-A. Lemburg wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. I think the most important missed fact

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure hosted on a cloud platform may be confortable, and probably needed to have a 24/7 running system, but we need to take

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Michael Crute wrote: On Tue, Jun 15, 2010 at 7:49 AM, M.-A. Lemburg m...@egenix.com wrote: As mentioned, I've been working on a proposal text for the cloud idea. Here's a first draft. Please have a look and let me know whether I've missed any important facts. Thanks. What about a set of

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 6:02 PM, M.-A. Lemburg m...@egenix.com wrote: Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure hosted on a cloud platform may be confortable, and

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Ronald Oussoren
On 15 Jun, 2010, at 19:02, Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 6:02 PM, M.-A. Lemburg m...@egenix.com wrote: Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 6:02 PM, M.-A. Lemburg m...@egenix.com wrote: Alexis Métaireau wrote: Hello, Firstly, as Tarek said in another thread, I'm afraid this kill the PEP381 about making a mirroring infrastructure. Having a infrastructure hosted on a cloud platform may

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Jesus Cea wrote: On 15/06/10 13:49, M.-A. Lemburg wrote: Server side: upload cronjobs Since the /simple index tree is currently being created dynamically, we'd need to create static copies of it at regular intervals in order to upload the content to the S3

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 16:33, Steven D'Aprano wrote: For example, if a single edge server in (say) Australia goes down, Amazon might not count it as an outage for the purpose of calculating their 99.99% reliability since the system as a whole is still up,

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 19:45, M.-A. Lemburg wrote: Note that with community servers that only mirror once a day, you'd have to wait up to a whole day for your package updates to become visible worldwide. But TODAY mirror use is voluntary and per-user. That

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 7:34 PM, M.-A. Lemburg m...@egenix.com wrote: [..] So I think it would be better to focus on PEP 381, and make those existing mirrors comply with it. And maybe work on the legal issues you've mentioned That can all happen in parallel. I really doubt it. You have come

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 8:21 PM, Jesus Cea j...@jcea.es wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 19:45, M.-A. Lemburg wrote: Note that with community servers that only mirror once a day, you'd have to wait up to a whole day for your package updates to become visible

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 7:43 PM, M.-A. Lemburg m...@egenix.com wrote: Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 7:15 PM, Ronald Oussoren ronaldousso...@mac.com wrote: On 15 Jun, 2010, at 19:02, Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 6:02 PM, M.-A. Lemburg m...@egenix.com wrote:

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Martin v. Löwis
Hi Martin, Notice that you actually replied to Marc-Andre Lemburg. You should be able to get access to the Python sandbox repository and add your project there: http://svn.python.org/projects/sandbox/trunk/ If that's not an option, I'd suggest you have a look at one of the

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 7:34 PM, M.-A. Lemburg m...@egenix.com wrote: [..] So I think it would be better to focus on PEP 381, and make those existing mirrors comply with it. And maybe work on the legal issues you've mentioned That can all happen in parallel. I really

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Martin v. Löwis wrote: I read pep 381 long time ago and I don't remember how/when a mirror would update, but I do remember it doesn't mandate digital signatures (signed by pypi central node, verified by setuptoolsfriends). That is a big gap, in my opinion. The PEP doesn't explain the

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
* How will clients be sure that they are getting the correct key ? They should initially download it from the master server (when that is online) and cache it. * What would a client do if the PyPI server is down ? Isn't that straight-forward? * How would clients protect their local

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Martin v. Löwis wrote: PyPI itself has in recent months been mostly maintained by one developer: Martin von Loewis. Projects are underway to enhance PyPI in various ways, including a proposal to add external mirroring (PEP 381), but these are all far from being finalized or implemented.

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 9:02 PM, Martin v. Löwis mar...@v.loewis.de wrote: [..] Alternatively, you could start submitting patches. Some work Matthieu did is already integrated via the branch I worked on for PEP 345. And we were considering using the same workflow since I can commit. Of course,

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Tarek Ziadé
2010/6/15 Tarek Ziadé ziade.ta...@gmail.com: On Tue, Jun 15, 2010 at 9:02 PM, Martin v. Löwis mar...@v.loewis.de wrote: [..] Alternatively, you could start submitting patches. Some work Matthieu did is already integrated via the branch I worked on for PEP 345. And we were considering using

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread M.-A. Lemburg
Tarek Ziadé wrote: On Tue, Jun 15, 2010 at 10:14 PM, M.-A. Lemburg m...@egenix.com wrote: I'm not trying to compete with your mirror PEP, just trying to solve a problem. We are trying to solve the same problem, aren't we ? Sure, but the intent is not to compete with the PEP. Even with the

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Martin v. Löwis
As a maintainer of the PyPI project, it makes your workflow simpler, - contributors can clone the repo, change the code and ask you for a pull - you can pull changes by direct hg commands, and merge them After using Mercurial in one project, I'm skeptical that this really makes things

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
* How will clients be sure that they are getting the correct key ? They should initially download it from the master server (when that is online) and cache it. So they'll use HTTPS and check the server certificate as well ? No. But they trust that the package contents is untampered when

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Tarek Ziadé
2010/6/15 Martin v. Löwis mar...@v.loewis.de: As a maintainer of the PyPI project, it makes your workflow simpler, - contributors can clone the repo, change the code and ask you for a pull - you can pull changes by direct hg commands, and merge them After using Mercurial in one project, I'm

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 20:52, Tarek Ziadé wrote: Do you trust the package you are installing more than an official mirror ? if so, why ? If a package is signed by the author, I only need to trust the author. If a package is not signed in PYPI, I must trust

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 21:52, Martin v. Löwis wrote: As for timeliness: it would be reasonable to setup the mirrors so that they won't be behind more than one minute (by polling for changes every minute). On the one hand, some people claim that this would be

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 22:04, Martin v. Löwis wrote: I read pep 381 long time ago and I don't remember how/when a mirror would update, but I do remember it doesn't mandate digital signatures (signed by pypi central node, verified by setuptoolsfriends). That

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
What's important also, is to make sure z3c.pypimirror includes the server-side work, so existing mirrors can be upgraded. Not really. z3c.pypimirror has a completely different function. Operators providing one of the official PyPI mirrors should use pep381client instead. Of course, if

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
Could I ask pep381 to be updated?. Sure you can ask. So did I. Regards, Martin ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 22:33, M.-A. Lemburg wrote: * How will clients be sure that they are getting the correct key ? Err... Download from a HTTPS server, with certificate verification in the client, would be nice :). * What would a client do if the PyPI

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
WebHooks:http://webhooks.pbworks.com/ Exactly so. Still, it requires a non-static web server. Also, with a push model, it's more difficult for the client to determine whether the server is current. In a pull model, the client can look at the last synchronization timestamp, and determine

Re: [Catalog-sig] PyPI down again...

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 11:55 PM, Mathieu Leduc-Hamel marra...@gmail.com wrote: Just be prepared to provide the code as separately-reviewable chunks of modifications. That's exactly the point. I may be wrong but me and people want to contribute and it's exactly what project like Bitbucket

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Steven D'Aprano
On Wed, 16 Jun 2010 03:44:05 am Jesus Cea wrote: 2. Packages MUST be digitally signed. Ideally by the owner -1 on requiring that by the package owner. While digitally signing packages is a good idea, the state of the art is not yet so simple that this will be anything but a barrier to entry

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Justin Cappos
On Tue, Jun 15, 2010 at 2:55 PM, Jesus Cea j...@jcea.es wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 20:52, Tarek Ziadé wrote: Do you trust the package you are installing more than an official mirror ? if so, why ? If a package is signed by the author, I only need to

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
On Tue, Jun 15, 2010 at 11:55 PM, Jesus Cea j...@jcea.es wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 15/06/10 20:52, Tarek Ziadé wrote: Do you trust the package you are installing more than an official mirror ? if so, why ? If a package is signed by the author, I only need to

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Fred Drake
On Tue, Jun 15, 2010 at 6:24 PM, Steven D'Aprano st...@pearwood.info wrote: A digital signature is not an MD5 checksum, it may have actual legal meaning in many countries equivalent to a pen and paper signature. I would expect that verifying a package was signed by PyPI to mean no more than

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Tarek Ziadé
2010/6/16 Martin v. Löwis mar...@v.loewis.de: What's important also, is to make sure z3c.pypimirror includes the server-side work, so existing mirrors can be upgraded. Not really. z3c.pypimirror has a completely different function. It's a mirroring script for PyPI. Why do you say it has a

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Jesus Cea
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 16/06/10 00:24, Steven D'Aprano wrote: I would not be digitally signing anything I didn't create unless I had good legal advice that it was safe to do so. The pypi signature certifies that the package has not been tampered with. It DO NOT

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
I would not be digitally signing anything I didn't create unless I had good legal advice that it was safe to do so. I'm actually not worried about this. In my own country, a valid digital signature requires much more than invocation of the RSA algorithm. E.g. available of certain certified

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
Is the plan to use what is proposed in http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html in practice? You mean, is it implemented and deployed? Sure - just try for yourself. Is more information available about this? This is not a very specific question. The answer is

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
Am 16.06.2010 00:38, schrieb Tarek Ziadé: 2010/6/16 Martin v. Löwismar...@v.loewis.de: What's important also, is to make sure z3c.pypimirror includes the server-side work, so existing mirrors can be upgraded. Not really. z3c.pypimirror has a completely different function. It's a mirroring

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread Martin v. Löwis
Am 16.06.2010 00:37, schrieb Fred Drake: On Tue, Jun 15, 2010 at 6:24 PM, Steven D'Apranost...@pearwood.info wrote: A digital signature is not an MD5 checksum, it may have actual legal meaning in many countries equivalent to a pen and paper signature. I would expect that verifying a package

Re: [Catalog-sig] Proposal: Move PyPI static data to the cloud for better availability

2010-06-15 Thread geremy condra
On Tue, Jun 15, 2010 at 3:55 PM, Martin v. Löwis mar...@v.loewis.de wrote: Is the plan to use what is proposed in http://mail.python.org/pipermail/catalog-sig/2009-March/002018.html in practice? You mean, is it implemented and deployed? Sure - just try for yourself. Is more information