Re: [Catalog-sig] A 90% Solution

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 03:46, PJ Eby wrote: On Mon, Mar 11, 2013 at 8:28 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 00:39, Donald Stufft wrote: On Mar 11, 2013, at 7:04 PM, PJ Eby p...@telecommunity.com wrote: Just a thought, but... If 90% of PyPI projects do not have any external files

Re: [Catalog-sig] A 90% Solution

2013-03-12 Thread holger krekel
On Mon, Mar 11, 2013 at 19:04 -0400, PJ Eby wrote: Just a thought, but... If 90% of PyPI projects do not have any external files to download, then, wouldn't it make sense to: sidenote: we need to verify and clarify the 90/10 ratio. It would be the basis for action/changing pypi-state so we

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jesse Noller
On Mar 12, 2013, at 1:25 AM, Lennart Regebro rege...@gmail.com wrote: On Mon, Mar 11, 2013 at 8:57 PM, PJ Eby p...@telecommunity.com wrote: On Mon, Mar 11, 2013 at 1:45 PM, Lennart Regebro rege...@gmail.com wrote: So, we should not remove the links for external packages until somebody

Re: [Catalog-sig] A 90% Solution

2013-03-12 Thread Jesse Noller
On Mar 12, 2013, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 03:46, PJ Eby wrote: On Mon, Mar 11, 2013 at 8:28 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 00:39, Donald Stufft wrote: On Mar 11, 2013, at 7:04 PM, PJ Eby p...@telecommunity.com wrote: Just a

Re: [Catalog-sig] A 90% Solution

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 10:20, Jesse Noller wrote: On Mar 12, 2013, at 3:57 AM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 03:46, PJ Eby wrote: On Mon, Mar 11, 2013 at 8:28 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 00:39, Donald Stufft wrote: On Mar 11, 2013, at 7:04 PM, PJ

[Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
Hi all, below is the new PEP pre-submit version (V2) which incorporates the latest suggestions and aims at a rapidly deployable solution. Thanks in particular to Philip, Donald and Marc-Andre. I also added a few notes on how installers should behave with respect to non-PYPI crawling. I think

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread Nick Coghlan
That looks pretty good to me. My only comment is that qualifiers like new don't age well in an API. The explicit nocrawlhomepage and nocrawldownload might be a better choice. Cheers, Nick. ___ Catalog-SIG mailing list Catalog-SIG@python.org

Re: [Catalog-sig] A 90% Solution

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 5:50 AM, M.-A. Lemburg m...@egenix.com wrote: Not hard to do: we'd just need to keep the old index in place using a different URL, e.g. /simple-v1/. That's not necessary: the XML-RPC API lets you query those URLs directly. They're part of the metadata standard, after

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 1:25 AM, Lennart Regebro rege...@gmail.com wrote: Externally hosted files are a real world actual problem. You're leaving out some important words from that sentence. Words like, for some people and who choose to depend on projects using them. PyPI isn't your private

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 10:38 AM, PJ Eby p...@telecommunity.com wrote: I'll ask it again: why should *thousands* of projects be censored or made to change their release processes, because *you* can't be bothered to cache the distributions of the projects you depend on? Because

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 10:38 AM, PJ Eby p...@telecommunity.com wrote: AFAICT, you're the ones stopping things moving forward here, filibustering against every possible compromise. Sorry, one more thing: I'm interested in what your comprise would be. Can you write up a counter-proposal to

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 7:38 AM, holger krekel hol...@merlinux.eu wrote: In addition, maintainers of installation tools are asked to release two updates. The first one shall provide clear warnings if external crawling needs to happen, A clarification here: needs to happen is not

Re: [Catalog-sig] Inconsistency on f.pypi.python.org with Products.PluggableAuthService

2013-03-12 Thread Maurits van Rees
Op 05-03-13 16:34, Christian Theune schreef: Hi, it seems my fight to keep f.pypi.python.org is at least keeping the pypi-mirrors.org page happy. Unfortunately one ouf our users detected another inconsistency that the mirror script doesn't find or clean up by itself. I also don't know how to

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 12:38, holger krekel wrote: Hi all, below is the new PEP pre-submit version (V2) which incorporates the latest suggestions and aims at a rapidly deployable solution. Thanks in particular to Philip, Donald and Marc-Andre. I also added a few notes on how installers should

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 16:42, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 10:38 AM, PJ Eby p...@telecommunity.com wrote: I'll ask it again: why should *thousands* of projects be censored or made to change their release processes, because *you* can't be bothered to cache the distributions of the

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
On Wed, Mar 13, 2013 at 01:19 +1000, Nick Coghlan wrote: That looks pretty good to me. My only comment is that qualifiers like new don't age well in an API. The explicit nocrawlhomepage and nocrawldownload might be a better choice. Right, we might also consider dropping rel-attributing given

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 11:19 AM, M.-A. Lemburg m...@egenix.com wrote: So let's do this carefully and find a good solution before jumping to conclusions. Completely agreed; rushing is a bad idea. But so is not starting. What I'm seeing — as a total outsider, a user of these tools, not someone

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 11:53 -0400, PJ Eby wrote: On Tue, Mar 12, 2013 at 7:38 AM, holger krekel hol...@merlinux.eu wrote: In addition, maintainers of installation tools are asked to release two updates. The first one shall provide clear warnings if external crawling needs to happen, A

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 17:29, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 11:19 AM, M.-A. Lemburg m...@egenix.com wrote: So let's do this carefully and find a good solution before jumping to conclusions. Completely agreed; rushing is a bad idea. But so is not starting. What I'm seeing — as a

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread Carl Meyer
Hi Holger, I am confused about the discrepancy between the title of this pre-PEP (transition to release file hosting on PyPI) and the contents of the PEP, which describe a transition to not crawling _HTML pages_ on external sites looking for distribution download links. These are not the same

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
Hi Marc-Andre, all, On Tue, Mar 12, 2013 at 17:06 +0100, M.-A. Lemburg wrote: On 12.03.2013 12:38, holger krekel wrote: Hi all, below is the new PEP pre-submit version (V2) which incorporates the latest suggestions and aims at a rapidly deployable solution. Thanks in particular to

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
Hi Carl, On Tue, Mar 12, 2013 at 10:48 -0600, Carl Meyer wrote: Hi Holger, I am confused about the discrepancy between the title of this pre-PEP (transition to release file hosting on PyPI) and the contents of the PEP, which describe a transition to not crawling _HTML pages_ on external

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 12:29 PM, Jacob Kaplan-Moss ja...@jacobian.org wrote: On Tue, Mar 12, 2013 at 11:19 AM, M.-A. Lemburg m...@egenix.com wrote: So let's do this carefully and find a good solution before jumping to conclusions. Completely agreed; rushing is a bad idea. But so is not

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 13:18 -0400, PJ Eby wrote: On Tue, Mar 12, 2013 at 12:29 PM, Jacob Kaplan-Moss ja...@jacobian.org wrote: On Tue, Mar 12, 2013 at 11:19 AM, M.-A. Lemburg m...@egenix.com wrote: So let's do this carefully and find a good solution before jumping to conclusions.

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jesse Noller
And I've put multiple compromise proposals out there to begin mitigating the problem *now* (i.e. for non-updated versions of setuptools), and every time, the objection is, no, we need to ban it all now, no discussion, no re-evaluation, no personal choice, everyone must do as we say, no

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 1:33 PM, Jesse Noller jnol...@gmail.com wrote: There's not much to understand: external hosting of packages is *actively harmful*, period. End users of easy_install and pip *don't even realize* 99% of the time that these tools are following links off of PyPi and

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread M.-A. Lemburg
Just a quick note (more later, if time permits)... On 12.03.2013 18:05, holger krekel wrote: Hi Marc-Andre, all, - Prepare PYPI implementation to allow a per-project hosting mode, effectively enabling or disabling external crawling. When enabled nothing changes from the current

[Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm

2013-03-12 Thread M.-A. Lemburg
I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip into the same directory and let easy_install running on Linux scan this, it considers the

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Donald Stufft
On Mar 12, 2013, at 12:41 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 17:29, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 11:19 AM, M.-A. Lemburg m...@egenix.com wrote: So let's do this carefully and find a good solution before jumping to conclusions. Completely agreed;

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Carl Meyer
It seems to me that there's a remarkable level of consensus developing here (though it may not look like it), and a small set of remaining open questions. The consensus (as I see it): - Migrate away from scraping external HTML pages, with package owners in control of the migration but a deadline

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Robert Collins
On 13 March 2013 07:18, Carl Meyer c...@oddbird.net wrote: It seems to me that there's a remarkable level of consensus developing here (though it may not look like it), and a small set of remaining open questions. The consensus (as I see it): I think that is a fair summary. One thing I'd

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 12:54 PM, PJ Eby p...@telecommunity.com wrote: This is a rationale for secure defaults for various options, like the ones I outlined in the portions of my post that you *didn't* quote. It's not a rationale for removing the options themselves. Exactly; thanks for saying

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 1:00 PM, M.-A. Lemburg m...@egenix.com wrote: The whole Python package eco-system works based on trust and injecting fear into this system is not helpful, IMO. I'm sorry if my words came across that way; I'm not trying to scare anyone. I'm trying to emphasize that this

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jesse Noller
On Tuesday, March 12, 2013 at 2:56 PM, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 1:00 PM, M.-A. Lemburg m...@egenix.com (mailto:m...@egenix.com) wrote: The whole Python package eco-system works based on trust and injecting fear into this system is not helpful, IMO. I'm

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 1:58 PM, Jesse Noller jnol...@gmail.com wrote: Nah, that was me injecting fear. I call dibs on that one. Aw, man! Can I have Uncertainty and Doubt then? Jacob ___ Catalog-SIG mailing list Catalog-SIG@python.org

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jesse Noller
On Tuesday, March 12, 2013 at 2:59 PM, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 1:58 PM, Jesse Noller jnol...@gmail.com (mailto:jnol...@gmail.com) wrote: Nah, that was me injecting fear. I call dibs on that one. Aw, man! Can I have Uncertainty and Doubt then? Jacob

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Monty Taylor
On 03/12/2013 11:00 AM, M.-A. Lemburg wrote: On 12.03.2013 18:33, Jesse Noller wrote: And I've put multiple compromise proposals out there to begin mitigating the problem *now* (i.e. for non-updated versions of setuptools), and every time, the objection is, no, we need to ban it all now,

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 12:18 -0600, Carl Meyer wrote: It seems to me that there's a remarkable level of consensus developing here (though it may not look like it), and a small set of remaining open questions. The consensus (as I see it): - Migrate away from scraping external HTML pages,

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 19:07 +0100, M.-A. Lemburg wrote: Just a quick note (more later, if time permits)... On 12.03.2013 18:05, holger krekel wrote: Hi Marc-Andre, all, - Prepare PYPI implementation to allow a per-project hosting mode, effectively enabling or disabling external

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 2:18 PM, Carl Meyer c...@oddbird.net wrote: It seems to me that there's a remarkable level of consensus developing here (though it may not look like it), and a small set of remaining open questions. The consensus (as I see it): - Migrate away from scraping external

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 2:43 PM, Robert Collins robe...@robertcollins.net wrote: This takes an age when each new web host to talk to is a new DNS lookup (say 0.3 seconds) + HTTP request (0.6 seconds) with possible HTTPS setup in there too (up to 1.2 seconds). A project with dozens of

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 20:17, holger krekel wrote: On Tue, Mar 12, 2013 at 19:07 +0100, M.-A. Lemburg wrote: Just a quick note (more later, if time permits)... On 12.03.2013 18:05, holger krekel wrote: Hi Marc-Andre, all, - Prepare PYPI implementation to allow a per-project hosting mode,

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 2:21 PM, PJ Eby p...@telecommunity.com wrote: The *only* thing I object to is the part where some people want to ban external links from /simple, always and forever, regardless of the package authors' choice in the matter. Here's the thing though, there are already a

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 2:07 PM, M.-A. Lemburg m...@egenix.com wrote: Just a quick note (more later, if time permits)... On 12.03.2013 18:05, holger krekel wrote: Hi Marc-Andre, all, - Prepare PYPI implementation to allow a per-project hosting mode, effectively enabling or disabling

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 14:36 -0500, Jacob Kaplan-Moss wrote: On Tue, Mar 12, 2013 at 2:21 PM, PJ Eby p...@telecommunity.com wrote: The *only* thing I object to is the part where some people want to ban external links from /simple, always and forever, regardless of the package authors'

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread holger krekel
On Tue, Mar 12, 2013 at 15:21 -0400, PJ Eby wrote: On Tue, Mar 12, 2013 at 2:18 PM, Carl Meyer c...@oddbird.net wrote: It seems to me that there's a remarkable level of consensus developing here (though it may not look like it), and a small set of remaining open questions. The consensus

Re: [Catalog-sig] V2 pre-PEP: transitioning to release file hosting on PYPI

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 20:46, PJ Eby wrote: On Tue, Mar 12, 2013 at 2:07 PM, M.-A. Lemburg m...@egenix.com wrote: Just a quick note (more later, if time permits)... On 12.03.2013 18:05, holger krekel wrote: Hi Marc-Andre, all, - Prepare PYPI implementation to allow a per-project hosting mode,

Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm

2013-03-12 Thread M.-A. Lemburg
On 12.03.2013 19:15, M.-A. Lemburg wrote: I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg egenix-mxodbc-connect-client-2.0.2.win32-py2.6.prebuilt.zip into the same directory and let

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 3:36 PM, Jacob Kaplan-Moss ja...@jacobian.org wrote: On Tue, Mar 12, 2013 at 2:21 PM, PJ Eby p...@telecommunity.com wrote: The *only* thing I object to is the part where some people want to ban external links from /simple, always and forever, regardless of the package

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Donald Stufft
On Mar 12, 2013, at 4:14 PM, Carl Meyer c...@oddbird.net wrote: On 03/12/2013 01:21 PM, PJ Eby wrote: - In some way, migrate to a situation where the popular installer tools install only release files from PyPI by default, but are capable of installing from other locations if the user

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 3:16 PM, PJ Eby p...@telecommunity.com wrote: I'm confused by this statement. never access an external host is not consistent with have the option to specify what hosts you trust, while still keeping PyPI as a universal index of Python software. Sorry to be confusing!

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Jacob Kaplan-Moss
On Tue, Mar 12, 2013 at 3:30 PM, Jacob Kaplan-Moss ja...@jacobian.org wrote: As I've said, the implementation details aren't of a concern to me; the result is. You know what though, I kinda lied. While I don't care about the implementation, I *do* care about keeping this process moving

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 4:14 PM, Carl Meyer c...@oddbird.net wrote: You say below that nobody has proposed a 'trust everything' flag. If there is no trust everything flag, then it seems to me that with either option A or option B the user needs to specify what they intend to trust. I.e. if you

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Trishank Karthik Kuppusamy
Hello Jacob, Good to hear from you! Thanks for stating your concerns so clearly, and we do understand them. We agree that inertia is important to maintain. In fact, we are excited to show this in person to the PyPI community on Friday. We expect to release a design document and a demo in a

Re: [Catalog-sig] setuptools/distribute/easy_install/pkg_resource sorting algorithm

2013-03-12 Thread PJ Eby
On Tue, Mar 12, 2013 at 3:59 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.03.2013 19:15, M.-A. Lemburg wrote: I've run into a weird issue with easy_install, that I'm trying to solve: If I place two files named egenix_mxodbc_connect_client-2.0.2-py2.6.egg

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Reinout van Rees
On 11-03-13 11:44, Lennart Regebro wrote: That's now all the energy I'm willing to spend on discussing this topic. Third-party hosting needs to go. I believe there is a broad consensus on this. Let's instead discuss*how* to implement it. Hear hear! I'm so fed up with other people's non-pypi

Re: [Catalog-sig] pre-PEP: transition to release-file hosting at pypi site

2013-03-12 Thread Reinout van Rees
On 12-03-13 16:38, PJ Eby wrote: I'll ask it again: why should*thousands* of projects be censored or made to change their release processes, because*you* can't be bothered to cache the distributions of the projects you depend on? So... everyone that uses pypi should be *forced* to use their