Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Ralf Gommers
On Wed, Jan 29, 2014 at 5:22 PM, Vinay Sajip vinay_sa...@yahoo.co.ukwrote: On Wed, 29/1/14, Paul Moore p.f.mo...@gmail.com wrote: That package installation utilities should not dabble in sys.path manipulation. The import path is the user's

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Vinay Sajip
On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever reason, that should be supported. Setuptools inserting stuff to sys.path that

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Noah Kantrowitz
On Jan 30, 2014, at 1:09 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I want to use inplace builds and PYTHONPATH instead of virtualenvs for whatever

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Donald Stufft
On Jan 30, 2014, at 1:21 PM, Noah Kantrowitz n...@coderanger.net wrote: On Jan 30, 2014, at 1:09 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Thu, 30/1/14, Ralf Gommers ralf.gomm...@gmail.com wrote: Also end user. If, as a user, I

Re: [Distutils] Using Wheel with zipimport

2014-01-30 Thread Nick Coghlan
Thanks for pointing that out Noah - I was planning to come back and check if that wording was considered more acceptable. On 31 Jan 2014 07:12, Donald Stufft don...@stufft.io wrote: The updated text is fine with me as it at least accurately documents the fact that using that feature is fraught

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Donald Stufft donald at stufft.io writes: 1. That unpacking into the home directory is problematic because users that run services often don’t have home directories. This you waved away by saying that the home directory isn’t a required place, to which Jim responded that unpacking

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 05:52, Donald Stufft don...@stufft.io wrote: Which further shows that at the time it was cool that it worked but that the weird failures being a reason that Wheel was an installation format. Having cool features in a format is not a bad thing. Zip imports are the cool

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 04:14, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: When this topic came up before, I asked for specific failure modes which were causing concern, but I never got a response IIRC. From what I recall, the topic came up before in relation to distlib's wheel mount functionality.

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Paul Moore p.f.moore at gmail.com writes: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and accepted limitation of zipimport, and as a workaround, it has issues. If

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 20:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: You don't want it to be there, even if it might be useful to others, just because it isn't useful to you? It's not as if distlib is forcing that functionality on anyone. I believe

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately) runnable as a zip file, so that you can bootstrap into the wheel

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 10:36, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: 1. It does *not* just use the fact that wheels are importable. It goes beyond that and *extracts* C extensions to make them importable, too. That is a workaround for a known and

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 2:32 AM, Nick Coghlan ncogh...@gmail.com wrote: All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages*

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 6:47 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 2:32 AM, Nick Coghlan ncogh...@gmail.com wrote: All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Chris Jerdonek
On Wed, Jan 29, 2014 at 3:41 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
That doesn't really speak to the fact they were designed for that. If I read that it looks like some commenting that they are importable (which as it stands they are) and not someone calling it a supported feature of the format. It even states that the format is designed primarily for

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 11:41, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended feature. The wheel file for the wheel project itself is (deliberately)

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 6:59 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 11:41, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 4:23 AM, Paul Moore p.f.mo...@gmail.com wrote: As I recall, it was in the original version of the PEP/spec and it was always an intended

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 12:18, Donald Stufft don...@stufft.io wrote: But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in without any chance for discussion. Fair point. I'm sure Nick did just think

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 7:43 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:18, Donald Stufft don...@stufft.io wrote: But more importantly, while I'm against officially supporting importable Wheels because of various reasons, my biggest problem is that this was added in

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 21:50, Donald Stufft don...@stufft.io wrote: As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the chance to discuss it. It should not be added ex post facto by a committer

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 7:58 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 21:50, Donald Stufft don...@stufft.io wrote: As a follow up, even if this becomes something we document and actually support, then it should be done in the next version of Wheel when people have the

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the ability to run wheels from sys.path. I don't believe that virtualenv

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 22:57, Donald Stufft don...@stufft.io wrote: Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then in the next version of the Wheel spec it should be added when

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:14 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 22:57, Donald Stufft don...@stufft.io wrote: Also to be specific, what I believe should be done is that the change should be reverted, and if it is desired that Wheels officially support zip import then

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:10, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:17 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:10, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the wheel format that is incompatible with it. Note that virtualenv uses the

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html Donald, I was *there*. I know what

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:31 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:16, Donald Stufft don...@stufft.io wrote: So basically even though the text of the PEP specifically points out that a difference of Wheel and Egg is that Eggs are importable it somehow supports that?

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:32 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature.

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Matthew Iversen
On the one hand, it's easy to see the allure of a zipimport-able format. Grab a file, import it, use functionality straight away. It has an attraction of self-containedness. On the other hand, python's dynamic nature means that things are not as simple as Java jar's as Donald points out. I agree

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:34 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:32 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:43 AM, Daniel Holth dho...@gmail.com wrote: On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Daniel Holth
On Wed, Jan 29, 2014 at 8:44 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:43 AM, Daniel Holth dho...@gmail.com wrote: On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:46 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:34, Donald Stufft don...@stufft.io wrote: If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong. You have the burden of proof backwards there. You're the one

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Daniel Holth
On Wed, Jan 29, 2014 at 8:31 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 8:10 AM, Paul Moore p.f.mo...@gmail.com wrote: On 29 January 2014 12:58, Nick Coghlan ncogh...@gmail.com wrote: You can campaign to deprecate that feature *if* we ever want to make a change to the

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:34, Donald Stufft don...@stufft.io wrote: If I'm wrong, then by all means show me where it was discussed so I can admit I was wrong. You have the burden of proof backwards there. You're the one asking me to break backwards compatibility, and to let people continue to

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
Nick Coghlan ncoghlan at gmail.com writes: I believe Paul's concern is with anything that suggests that arbitrary *third party* code can be run from wheel files, when the reality is that it is fairly easy to accidentally write code that assumes it is installed on the filesystem in a way that

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word:

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 8:59 AM, Nick Coghlan ncogh...@gmail.com wrote: On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a consenting adult who understands the caveats you may do so. What

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 9:25 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote: Mitre’s rules for CVEs are not entirely obvious to people who are not familiar with them. Generally if the feature *can* be used securely or there was no evidence that the author intended that

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Brett Cannon
On Wed, Jan 29, 2014 at 9:40 AM, Donald Stufft don...@stufft.io wrote: On Jan 29, 2014, at 9:25 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 30 Jan 2014 00:28, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: It may be useful to understand that wheel has *political features* or if you prefer *setting the defaults based on what we have learned from eggs*. I don't recommend that they be zip-imported generally but if you are a

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote: It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I just recently redid the get-pip.py installer to use a zip file (not a Wheel

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
I went through this with Chris McDonough back when packaging was dropped from 3.3, and he really helped me focus on what I found to be the two closely related core problems: - implicit sys.path manipulation - installing as eggs by default That was due to easy_install defaults being chosen

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 13:31, Donald Stufft don...@stufft.io wrote: Here's Paul explicitly mentioning that Wheels being used with zip import is an incidental benefit and not a core feature. https://mail.python.org/pipermail/distutils-sig/2013-March/020379.html That's quoted out of context. It was

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Donald Stufft
On Jan 29, 2014, at 10:00 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: On Wed, 29/1/14, Donald Stufft don...@stufft.io wrote: It’s hard to pin down because the failure modes of zipped eggs are nebulous themselves. For instance take pip. I

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Paul Moore
On 29 January 2014 14:25, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: What *else* have we learned from eggs? That designing modules to support zipimport correctly is non-trivial. And that assuming that things work unless told otherwise is a bad default behaviour. That packaging solutions should

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Vinay Sajip
On Wed, 29/1/14, Paul Moore p.f.mo...@gmail.com wrote: That designing modules to support zipimport correctly is non-trivial. It's not trivial, but it's not especially hard, either. Mostly, it's about remembering to consider zipimport, since that

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Matthew Iversen
On 30/01/14 00:59, Nick Coghlan wrote: On 29 January 2014 23:48, Donald Stufft don...@stufft.io wrote: So what did you mean when you said We discussed it extensively before PEP 427 was approved if you're now saying that it wasn't discussed. Explicitly would be a better word:

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Greg Ewing
Nick Coghlan wrote: Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and we wanted the wheel format to offer a strict superset of the egg

Re: [Distutils] Using Wheel with zipimport

2014-01-29 Thread Nick Coghlan
On 30 Jan 2014 07:50, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Nick Coghlan wrote: Otherwise we'd have to define a whole new format for something that can be adequately handled by a wheel that meets certain restrictions, and that would be pointless (we already have too many formats, and

[Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
I noticed yesterday that Nick made a commit to PEP427 that Wheels were designed to be compatible with zipimport so that you could directly add them to sys.path (http://hg.python.org/peps/rev/811e34eda04c). I was not aware that this was the case, and had thought otherwise, and have been telling

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's a useful capability if used responsibly (as we do in the ensurepip

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on sys.path - whether or not it will work depends on the context, but it's

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Daniel Holth
On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on sys.path is no different from putting any other zipfile directly on

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Vinay Sajip
Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it. I know that people have had bad experiences in the past with eggs, for the

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
On 29 Jan 2014 11:19, Daniel Holth dho...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think you're reading too much into that comment. Putting a wheel file directly on

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 11:54 PM, Nick Coghlan ncogh...@gmail.com wrote: On 29 Jan 2014 11:19, Daniel Holth dho...@gmail.com wrote: On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft don...@stufft.io wrote: On Jan 28, 2014, at 6:38 PM, Nick Coghlan ncogh...@gmail.com wrote: I think

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Donald Stufft
On Jan 28, 2014, at 11:14 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Donald Stufft donald at stufft.io writes: I think we need to reconsider this. I cannot stress how badly an idea I think it is for Wheels to concern itself with the ability to add the Wheel to sys.path and import it.

Re: [Distutils] Using Wheel with zipimport

2014-01-28 Thread Nick Coghlan
All of those arguments are against *recommending* directly importing from wheels. Yes, there are lots of problems with running directly from a zip archive, which is why the fact that easy_install *actively encourages* potentially inexperienced users to run things that way is so problematic.