[Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However there is still another massive area of insecurity within the packaging tool chain.

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Jesse Noller
On Wednesday, February 27, 2013 at 10:26 AM, Donald Stufft wrote: PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However there is

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread M.-A. Lemburg
On 27.02.2013 16:26, Donald Stufft wrote: PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However there is still another massive area

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: -1. There are many reasons for not hosting packages and distributions on PyPI itself. If you use and trust a package, you also have to know and trust its dependencies, no matter where they are hosted, so you're not gaining

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Wed, Feb 27, 2013 at 8:26 AM, Donald Stufft donald.stu...@gmail.com wrote: PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and the parties involved should all be thanked. However

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Ronald Oussoren
On 27 Feb, 2013, at 16:42, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: -1. There are many reasons for not hosting packages and distributions on PyPI itself. If you use and trust a package, you also have to know and trust

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Marcus Smith
pip/easy_install into installing the right thing by version number naming (which is completely broken btw. It's impossible to name separate Python 2 and Python 3 packages so that both pip and easy_install will do the right thing in every case. See

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 11:34 AM, M.-A. Lemburg wrote: On 27.02.2013 16:42, Donald Stufft wrote: On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: -1. There are many reasons for not hosting packages and distributions on PyPI itself. If you use and

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread M.-A. Lemburg
On 27.02.2013 17:43, Donald Stufft wrote: On Wednesday, February 27, 2013 at 11:34 AM, M.-A. Lemburg wrote: On 27.02.2013 16:42, Donald Stufft wrote: On Wednesday, February 27, 2013 at 10:39 AM, M.-A. Lemburg wrote: -1. There are many reasons for not hosting packages and distributions on

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 12:10 PM, M.-A. Lemburg wrote: Package installers only need access to the static files in the /simple/ index. Those can be put behind a CDN to increase uptime. PyPI itself doesn't have to be up and running if you just want to download the files

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 12:22 PM, holger krekel wrote: The main means of securing against tampering is author-signatures and verification by installers. If we have that, the download location does not matter (pypi/CDN/google/...). Again we don't have that yet, It's only 1 layer, and

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Feb 27, 2013, at 10:22 AM, holger krekel hol...@merlinux.eu wrote: On Wed, Feb 27, 2013 at 10:26 -0500, Donald Stufft wrote: PyPI is now being served with a valid SSL certificate, and the tooling has begun to incorporate SSL verification of PyPI into the process. This is _excellent_ and

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Jesse Noller
2. External links decrease the expected uptime for a particular set of requirements. PyPI itself has become very stable, however the same cannot be said for all of the hosts linked that the toolchain processes. Each new host is an additional SPOF. Ex: I depend on PyPI and 10 other

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Justin Cappos
Having different sources for package metadata does pose security concerns, for example version mismatch attacks by a MITM. Unless we co-locate all package metadata at a single source that is trusted for protecting against these issues, this will be an issue.(However, possibly not the biggest

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
Which in particular means that metadata needs to come from PyPI itself, not from the tarball file name. Aaron Meurer On Feb 27, 2013, at 11:06 AM, Justin Cappos jcap...@poly.edu wrote: Having different sources for package metadata does pose security concerns, for example version mismatch

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Noah Kantrowitz
On Feb 27, 2013, at 9:28 AM, M.-A. Lemburg wrote: On 27.02.2013 18:05, Noah Kantrowitz wrote: M.-A. Lemburg m...@egenix.com wrote: I propose we deprecate the external links that PyPI has published on the /simple/ indexes which exist because of the history of PyPI. Ideally in some number

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Giovanni Bajo
Il giorno 27/feb/2013, alle ore 19:23, Donald Stufft donald.stu...@gmail.com ha scritto: On Wednesday, February 27, 2013 at 12:44 PM, Donald Stufft wrote: Why not first have an a good infrastructure and capacity with pypi.python.org so that people *want* to move their files there? PyPI has

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 1:32 PM, Giovanni Bajo wrote: In fact, Python is a big-enough brand name that we could even get a CDN service almost for free in exchange of an acknowledge of the CDN company being used. As far as I know this has already have been offered in some form

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 1:34 PM, holger krekel wrote: On Wed, Feb 27, 2013 at 13:00 -0500, Jesse Noller wrote: 2. External links decrease the expected uptime for a particular set of requirements. PyPI itself has become very stable, however the same cannot be said for all of

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Jesse Noller
On Wednesday, February 27, 2013 at 1:33 PM, Donald Stufft wrote: On Wednesday, February 27, 2013 at 1:32 PM, Giovanni Bajo wrote: In fact, Python is a big-enough brand name that we could even get a CDN service almost for free in exchange of an acknowledge of the CDN company being

[Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Monty Taylor
Hey all, OpenStack recently ran in to a problem where one of our depends released a new version that only works with Python 3 and not Python 2. While I wholeheartedly support the gusto of that, and also can't wait until we can move to Python 3, there's a tooling issue here. If I'm doing pip

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: Hey all, OpenStack recently ran in to a problem where one of our depends released a new version that only works with Python 3 and not Python 2. While I wholeheartedly support the gusto of that, and also can't wait until we can

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 2:05 PM, Donald Stufft wrote: On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: Hey all, OpenStack recently ran in to a problem where one of our depends released a new version that only works with Python 3 and not Python 2. While I

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Monty Taylor
On 02/27/2013 02:05 PM, Donald Stufft wrote: On Wednesday, February 27, 2013 at 2:03 PM, Monty Taylor wrote: Hey all, OpenStack recently ran in to a problem where one of our depends released a new version that only works with Python 3 and not Python 2. While I wholeheartedly support the

Re: [Catalog-sig] Problems with pep381client

2013-02-27 Thread Brian Long
Hello all, I've heard nothing in response to my email and with all the discussion taking place on this list regarding the security posture of Pypi, I assume private mirrors need to hold off until all the issues are resolved. Am I correct in the assumption that I should avoid using pep381client

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Wed, Feb 27, 2013 at 11:37 AM, holger krekel hol...@merlinux.eu wrote: On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg m...@egenix.com wrote: I'm not saying that it's not a good idea to host packages on PyPI, but forcing the

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On a general note: It really warms my heart to see that people are warming up to the idea of using CDN's and getting rid of external downloads. I'm all for that. //Lennart ___ Catalog-SIG mailing list Catalog-SIG@python.org

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Jesse Noller
On Wednesday, February 27, 2013 at 2:47 PM, Lennart Regebro wrote: On a general note: It really warms my heart to see that people are warming up to the idea of using CDN's and getting rid of external downloads. I'm all for that. Excellent. So it's a date!

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Noah Kantrowitz
On Feb 27, 2013, at 11:47 AM, Lennart Regebro wrote: On a general note: It really warms my heart to see that people are warming up to the idea of using CDN's and getting rid of external downloads. I'm all for that. Just to be clear on this point 1) Moving PyPI and other PSF properties behind

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Daniel Holth
Would it be wrong to ask for a /complex API at the same time? The simple api, with 28k package names on one page, is getting a little silly. ___ Catalog-SIG mailing list Catalog-SIG@python.org http://mail.python.org/mailman/listinfo/catalog-sig

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Monty Taylor
On 02/27/2013 02:46 PM, Daniel Holth wrote: The Classifier: Python ... is another tag that could be more likely to tell you what you want to know. Hopefully the new, broken package advertises Classifier: Python 3 only. In theory I don't like the idea of using the Trove classifiers for

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor mord...@inaugust.com (mailto:mord...@inaugust.com) wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 11:37 AM, holger krekel hol...@merlinux.eu

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Aaron Meurer
We had a problem with SymPy where we have separate tarballs for Python 2 and Python 3 (our 2to3 process uses git metadata, so we prefer to do it at release time rather than have the user do it). But pip was attempting to install the Python 2 version in both Python 2 and Python 3. The solution

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Noah Kantrowitz
On Feb 27, 2013, at 11:59 AM, Monty Taylor wrote: On 02/27/2013 02:46 PM, Daniel Holth wrote: The Classifier: Python ... is another tag that could be more likely to tell you what you want to know. Hopefully the new, broken package advertises Classifier: Python 3 only. In theory I don't

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 2:59 PM, Monty Taylor wrote: Yeah - it's not so much that _I_ want to use something to know - it's more that I want pip to not try to install python3-only packages when I'm clearly running in python2. The reverse should also be true of course. I know we all

Re: [Catalog-sig] Python Version support in PyPI

2013-02-27 Thread Daniel Holth
On Wed, Feb 27, 2013 at 3:02 PM, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 27, 2013 at 2:59 PM, Monty Taylor wrote: Yeah - it's not so much that _I_ want to use something to know - it's more that I want pip to not try to install python3-only packages when I'm clearly

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Feb 27, 2013, at 1:01 PM, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor mord...@inaugust.com wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 11:37 AM,

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Daniel Holth
On Wed, Feb 27, 2013 at 3:08 PM, Aaron Meurer asmeu...@gmail.com wrote: On Feb 27, 2013, at 1:01 PM, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 27, 2013 at 2:56 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 12:49 PM, Monty Taylor mord...@inaugust.com wrote: On

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread holger krekel
On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 11:37 AM, holger krekel hol...@merlinux.eu wrote: On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Noah Kantrowitz
On Feb 27, 2013, at 12:16 PM, holger krekel wrote: On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 11:37 AM, holger krekel hol...@merlinux.eu wrote: On Wed, Feb 27, 2013 at 19:34 +0100, Lennart Regebro wrote: On

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Marcus Smith
As far as I'm concerned, pip is broke too, in the sense that the method we use to make pip work in Python 3 is a bit of an annoying hack (namely, upload a separate tarball for each minor Python 3 version). I agree it's a hack. but only =1.2 package metadata supports requires-python and

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Richard Jones
On Feb 28, 2013 2:26 AM, Donald Stufft donald.stu...@gmail.com wrote: I propose we deprecate the external links that PyPI has published on the /simple/ indexes which exist because of the history of PyPI. +1 ___ Catalog-SIG mailing list

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Chris Lambacher
On Wed, Feb 27, 2013 at 3:27 PM, Donald Stufft donald.stu...@gmail.comwrote: I'm not asking for this to be shutoff immediately, it will be phased, particularly so project maintainers can be made aware that it's going away and can upload versions to PyPI to prevent this kind of wide spread

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor mord...@inaugust.com wrote: But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could explicitly break the external links by having them point to nothing if you are worried about the security or if it's some

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Monty Taylor
On 02/27/2013 04:04 PM, Lennart Regebro wrote: On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor mord...@inaugust.com wrote: But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could explicitly break the external links by having them point to nothing if you are

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread PJ Eby
On Wed, Feb 27, 2013 at 1:34 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg m...@egenix.com wrote: I'm not saying that it's not a good idea to host packages on PyPI, but forcing the community into doing this is not a good idea. I still don't

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On Wed, Feb 27, 2013 at 9:01 PM, Donald Stufft donald.stu...@gmail.com wrote: Modify the PyPI software to no longer link to those urls. Well, I guess we can remove the software home page and the download URL's from the simple index. For example, in PIL's case the simple index looks like this:

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread PJ Eby
On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor mord...@inaugust.com wrote: But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could explicitly break the external links by having them

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On Wed, Feb 27, 2013 at 10:17 PM, PJ Eby p...@telecommunity.com wrote: I haven't seen anybody mention it yet, but checkouts of development versions are a use case that can't currently be addressed without support for multiple external links. For example, setuptools itself offers SVN checkout

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Noah Kantrowitz
On Feb 27, 2013, at 1:31 PM, PJ Eby wrote: On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor mord...@inaugust.com wrote: But wouldn't this only be a change in pip/easy_install, not PyPI itself? I suppose you could

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 4:17 PM, PJ Eby wrote: On Wed, Feb 27, 2013 at 1:34 PM, Lennart Regebro rege...@gmail.com (mailto:rege...@gmail.com) wrote: On Wed, Feb 27, 2013 at 5:34 PM, M.-A. Lemburg m...@egenix.com (mailto:m...@egenix.com) wrote: I'm not saying that it's not a

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Ɓukasz Langa
On 27 lut 2013, at 21:16, holger krekel hol...@merlinux.eu wrote: On Wed, Feb 27, 2013 at 14:49 -0500, Monty Taylor wrote: On 02/27/2013 02:47 PM, Aaron Meurer wrote: If we don't remove the feature from pypi itself, then it won't help the folks for whom its a problem, because there will be

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On Wed, Feb 27, 2013 at 10:31 PM, PJ Eby p...@telecommunity.com wrote: Replacing that would be a PITA because PyPI only lets you upload and register new releases from distutils' command line. You can upload files, but not create new releases. But that seems like a pretty minor addition, or?

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 4:31 PM, PJ Eby wrote: So far, I don't think anybody's talking to the right we for stopping it. It's the tools that control this, not PyPI. (PyPI can't actually stop the tools from using this information without also making itself a lot less useful to

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Richard Jones
On 28 February 2013 08:31, PJ Eby p...@telecommunity.com wrote: OTOH, I currently make development snapshots of setuptools and other projects available by dumping them in a directory that's used as an external download URL. Replacing that would be a PITA because PyPI only lets you upload and

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Lennart Regebro
On Wed, Feb 27, 2013 at 11:48 PM, Richard Jones rich...@python.org wrote: I've advocated us having the upload/register/whatever functionality in a separate tool for a while, but that doesn't seem to have gained any traction. Of course issues around the complexity introduced by setup.py make it

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Wed, Feb 27, 2013 at 2:31 PM, PJ Eby p...@telecommunity.com wrote: On Wed, Feb 27, 2013 at 4:04 PM, Lennart Regebro rege...@gmail.com wrote: On Wed, Feb 27, 2013 at 8:49 PM, Monty Taylor mord...@inaugust.com wrote: But wouldn't this only be a change in pip/easy_install, not PyPI itself? I

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread PJ Eby
On Wed, Feb 27, 2013 at 4:50 PM, Donald Stufft donald.stu...@gmail.com wrote: Development snapshots are a use case that i'm not sure makes sense for PyPI, but if they do should require specific opt-in to install them. Does easy_install have a command line flag that adds extra links? *chuckle*.

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 7:08 PM, PJ Eby wrote: On Wed, Feb 27, 2013 at 6:16 PM, Aaron Meurer asmeu...@gmail.com (mailto:asmeu...@gmail.com) wrote: As far as I'm concerned, this is all about helping package maintainers. The way pip works now, every time I do a release candidate,

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Aaron Meurer
On Wed, Feb 27, 2013 at 6:24 PM, Donald Stufft donald.stu...@gmail.com wrote: On Wednesday, February 27, 2013 at 8:13 PM, PJ Eby wrote: On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft donald.stu...@gmail.com wrote: This seems a bit complicated, people in general don't even know the external

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Wednesday, February 27, 2013 at 8:34 PM, Aaron Meurer wrote: On Wed, Feb 27, 2013 at 6:24 PM, Donald Stufft donald.stu...@gmail.com (mailto:donald.stu...@gmail.com) wrote: On Wednesday, February 27, 2013 at 8:13 PM, PJ Eby wrote: On Wed, Feb 27, 2013 at 7:36 PM, Donald Stufft

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Marcus Smith
maintainers. The way pip works now, every time I do a release candidate, pip automatically installs it, even though I only upload it an option to exclude pre-releases (or in reverse, an option to allow them) does seem overdue. reasons not to do this? anyone? links to the most relevant

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Nick Coghlan
On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft donald.stu...@gmail.com wrote: Sometimes you need to break things. The goal is to do it with ample warning and migration time so that people have a chance to move to the new way of doing things. Again, I am not suggesting we delete all external

Re: [Catalog-sig] Deprecate External Links

2013-02-27 Thread Donald Stufft
On Thursday, February 28, 2013 at 1:39 AM, Nick Coghlan wrote: On Thu, Feb 28, 2013 at 6:27 AM, Donald Stufft donald.stu...@gmail.com (mailto:donald.stu...@gmail.com) wrote: Sometimes you need to break things. The goal is to do it with ample warning and migration time so that people have a