Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12 May 2014 15:39, Donald Stufft don...@stufft.io wrote: On May 12, 2014, at 12:50 AM, Nick Coghlan ncogh...@gmail.com wrote: There are some more notable names in the unsafe lists, but a few spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, django-piston, ipaddr, matplotlib, and mayavi showed that a number of them *have* switched to PyPI hosting for recent releases, but have left older releases as externally hosted. (A few notable names, like wxPython and Spyder, *did* show up as genuinely externally hosted. Something that would be nice to be able to do, but isn't really practical without a server side dependency graph, is to be able to figure out how many packages have an externally hosted dependency *somewhere in their dependency chain*, and *how many* other projects are depending on particular externally hosted projects transitively). I could maybe do it with a mirror and a throw away VM but I think it’d be a decent chunk of effort. It's one of the things metadata 2.0 should eventually enable, but I think the numbers you already have are indicative enough to justify find a way to kill off the feature. Regardless, even with those caveats, the numbers are already solid enough to back up the notion that the only possible reasons to support enabling verified external hosting support independently of unverified external hosting are policy and relationship management ones. Relationship management would just mean providing a deprecation period before removing the capability, but I want to spend some time exploring a possible concrete *policy* related rationale for keeping it. The main legitimate reason I am aware of for wanting to avoid PyPI hosting is for non-US based individuals and organisations to avoid having to sign up to the Any uploads of packages must comply with United States export controls under the Export Administration Regulations. requirement that the PSF is obliged to place on uploads to the PSF controlled US hosted PyPI servers. That rationale certainly applies in MAL's case, since eGenix is a German company, and I believe they mostly do business outside the US (for example, their case study in the Python brochure is for a government project in Ghana). Yes that is the main reason I can distill from the various threads that have occurred over time. So can we agree that's the use case we need to have a solid answer for before completely dropping external hosting support from pip? I’m not sure the distinction makes much sense for PyPI/pip. You basically have to trust the authors of the packages you’re installing. If a package author is willing to hijack another package with a custom index they could just as easily do something malicious in a setup.py. Even if we get rid of the setup.py there are still endless ways of attacking someone who is installing your package and they are basically impossible to prevent and are just as bad or worse than that. Yeah, that's a good point - there's little or nothing a malicious index can do that a malicious setup.py couldn't already do. My reasons are: * It's only somewhat nicer up front than providing a custom index however it represents an additional command line flag that users have to learn. We also have the option of some day providing a general access European hosted index server that omits the US export restriction requirement from its upload terms. That's a mechanism pip could enable by default without introducing the multiple single points of failure in series problem for complex dependency stacks. * I hate the idea of a long term --allow-all-verified-external (or any variant of it). It feels way too much to me like a unbreak my pip please flag and I think that it is how users who need to use it will perceive it. This will create more animosity and hostility towards the packaging toolchain. I went into this on the pip PR, but essentially I see this becoming a turd that people chuck into their ~/.pip/pip.conf, requirements.txt, environment, or build scripts. They'll run into a problem where they need it, shove it into their config and then forget about it until they try to deploy to a new machine, or service, or whatever and run into that problem again. Agreed - it would be better to have a solution that points a way towards an eventual enabled by default solution, and the multiple index server support does indeed seem to better fit that bill. * I don't agree it says to non-US users that they must agree to the US export rules in order to participate in PyPI at all. They'll still be able to register their projects with PyPI, provide docs there. They just won't get as streamlined install experience. They'll have to provide some installation instructions. There is possibly even something we can do to make this more streamlined. Like perhaps they can register their custom index with PyPI and PyPI can advise pip of it and if
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On May 12, 2014, at 2:21 AM, Nick Coghlan ncogh...@gmail.com wrote: On 12 May 2014 15:39, Donald Stufft don...@stufft.io wrote: On May 12, 2014, at 12:50 AM, Nick Coghlan ncogh...@gmail.com wrote: There are some more notable names in the unsafe lists, but a few spot checks on projects like PyGObject, PyGTK, biopython, dbus-python, django-piston, ipaddr, matplotlib, and mayavi showed that a number of them *have* switched to PyPI hosting for recent releases, but have left older releases as externally hosted. (A few notable names, like wxPython and Spyder, *did* show up as genuinely externally hosted. Something that would be nice to be able to do, but isn't really practical without a server side dependency graph, is to be able to figure out how many packages have an externally hosted dependency *somewhere in their dependency chain*, and *how many* other projects are depending on particular externally hosted projects transitively). I could maybe do it with a mirror and a throw away VM but I think it’d be a decent chunk of effort. It's one of the things metadata 2.0 should eventually enable, but I think the numbers you already have are indicative enough to justify find a way to kill off the feature. Regardless, even with those caveats, the numbers are already solid enough to back up the notion that the only possible reasons to support enabling verified external hosting support independently of unverified external hosting are policy and relationship management ones. Relationship management would just mean providing a deprecation period before removing the capability, but I want to spend some time exploring a possible concrete *policy* related rationale for keeping it. The main legitimate reason I am aware of for wanting to avoid PyPI hosting is for non-US based individuals and organisations to avoid having to sign up to the Any uploads of packages must comply with United States export controls under the Export Administration Regulations. requirement that the PSF is obliged to place on uploads to the PSF controlled US hosted PyPI servers. That rationale certainly applies in MAL's case, since eGenix is a German company, and I believe they mostly do business outside the US (for example, their case study in the Python brochure is for a government project in Ghana). Yes that is the main reason I can distill from the various threads that have occurred over time. So can we agree that's the use case we need to have a solid answer for before completely dropping external hosting support from pip? Yes. I'm less worried about the specific timeline (as long as it's reasonable) than I am worried about having **a** timeline and an answer in the intrim that I can point people towards. I'm sorry but it's getting fixed and here's the plan is so much better to tell people than I'm sorry. I’m not sure the distinction makes much sense for PyPI/pip. You basically have to trust the authors of the packages you’re installing. If a package author is willing to hijack another package with a custom index they could just as easily do something malicious in a setup.py. Even if we get rid of the setup.py there are still endless ways of attacking someone who is installing your package and they are basically impossible to prevent and are just as bad or worse than that. Yeah, that's a good point - there's little or nothing a malicious index can do that a malicious setup.py couldn't already do. My reasons are: * It's only somewhat nicer up front than providing a custom index however it represents an additional command line flag that users have to learn. We also have the option of some day providing a general access European hosted index server that omits the US export restriction requirement from its upload terms. That's a mechanism pip could enable by default without introducing the multiple single points of failure in series problem for complex dependency stacks. We'd have to figure this out, but I'm not against trying to sort it out. Rackspace has European DCs and I think Fastly has the ability to select only European POPs (if that matters?) so it wouldn't even really require a degraded performance. There are logistics and other considerations of course, but it's not in and of itself something that I think would be completely off the table. We'd of course want to make sure there was demand for it because it'd adding more work on the Python Infrastructure team (and we'd need buy in there too) but I don't think it's an outlandish thing. * I hate the idea of a long term --allow-all-verified-external (or any variant of it). It feels way too much to me like a unbreak my pip please flag and I think that it is how users who need to use it will perceive it. This will create more animosity and hostility towards the packaging toolchain. I went into this on the pip PR, but essentially I see this becoming a turd that people chuck into their
[Distutils] Problem with latest buildout bootstrap on Windows with Python 3.3
Hi all, I'm facing a problem trying to bootstrap a buildout with its latest bootstrap script under Windows, using Python 3.3. I'm looking for some hint to decide whether the issue is with buildout, Python 3, or with my own installation ... The problem is that executing the bootstrap script I obtain an ImportError on the standard `getopt` module. Digging a bit, I see that ``site.getsitepackages()`` returns a list composed by two elements, ``c:\\Python33`` and ``c:\\Python33\\lib\\site-packages``: effectively even the 3.4 code at http://hg.python.org/cpython/file/bc160f985b7b/Lib/site.py#l312 explicitly puts the plain prefix into the list. Then the buildout bootstrap script does something like:: for sitepackage_path in site.getsitepackages(): sys.path[:] = [x for x in sys.path if sitepackage_path not in x] and obviously after that ``sys.path`` does *not* contain any standard library path (which lives under c:\\Python33\\Lib), just the current working directory and `c:\\Windows\\system32\\python33.zip`. The code in the bootstrap script seems correct (even if it should probably use something like ``not x.startswith(sitepackage_path)``), what seems strange is that, under Windows, the ``getsitepackages()`` result contains a bare ``c:\\Python33``... for comparison, on my Debian sid I get:: Python 3.3.5 (default, Mar 22 2014, 13:24:53) [GCC 4.8.2] on linux Type help, copyright, credits or license for more information. import site site.getsitepackages() ['/usr/local/lib/python3.3/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.3/dist-packages', '/usr/lib/dist-python'] What is going wrong here? Thanks a lot for any suggestion, ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 09.05.2014 12:16, Paul Moore wrote: So there's an ongoing debate over pip's behaviour around disallowing external hosting by default (see thread pip: cdecimal an externally hosted file and may be unreliable over on python-dev for the latest round). It appears that the reason for disallowing external hosting (as opposed to unverifiable downloads) is purely about reliability - we can't be sure that an external host provides the same level of uptime as PyPI[1]. Given that, it seems to me that the situation is, for an externally hosted package foo: `pip install foo` - fails immediately, 100% of the time `pip install --allow-external foo foo` - works in all but a few cases where foo's host is down[2] I cannot understand how guaranteed failure is ever better than occasional but rare failure. For situations where it is critical to minimise the risk of an external host outage causing a deployment to fail, the only answer is to not use foo, or to host foo on your own private index. In both cases, all you need is to know that foo is externally hosted to do that - you certainly don't need pip to fail. As a concrete proposal: 1. Remove --allow-external/--allow-all-external and make it the default behaviour. +1 2. Add a new command to pip, maybe something like `pip check-external` which checks a set of reuirements, and reports the requirements that are externally hosted and which hosts they rely on. That gives users who need 100% reliability the information they need to implement the appropriate solution. Without causing pain for users who don't. +1 Note that the above is based on the fact[3] that *there are no security risks to --allow-external*. I am not arguing for a reduction in security, or a change to any sort of threat model. Comments? I think that's a good proposal. The main argument we addressed in the PEP 438 discussion was security, which was addressed by having tools check the checksums of downloaded packages. This also clears up the current effect of making externally hosted packages second class citizens in Python land. Paul [1] Donald explicitly stated that this is the case in the earlier thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html). I think Nick confirmed this (although I don't have a reference to hand). If it's not true then we need to be a lot clearer and more explicit about *why* ignoring external hosting by default is needed, because it seems nobody knows :-( [2] BTW, the syntax of `--allow-external` is hideous, but that's a side issue I don't want to debate right now. [3] See the first note. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PyPI down?
On 10 May 2014, at 21:10, Donald Stufft don...@stufft.io wrote: Hm, it’s working here. It works correctly now, but it was down for me at the time I send that mail about 13 hours ago. Can you tell me where about in the world you are? The Netherlands. For routing purposes: AS 3265. Wichert. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)
Given the thread on python-dev and comments I have read elsewhere, I would like to remind everyone in this discussion to come back to a respectful attitude towards the issues being discussed and the people involved. I am writing this as Python core developer and as PSF board member. PyPI is run by the PSF and both the PSF as well as the core developers have a responsibility to keep the Python eco system a friendly, open and inviting place to be. This includes accepting that package authors and companies do have reasons for not using PyPI to host packages, which some developers may not follow or find reasonable. PyPI as community resource still has to make sure that these package authors are not singled out and can publish their packages just like other authors who can host their packages on PyPI. What we can do is mandate security requirements, to make sure that the users downloading packages via the PyPI index get the packages that the package author registered, and I'm sure many authors will understand and respect such added requirements, but we may not marginalize these authors for not wanting to host on the PSF infrastructure. Think about it: PyPI has become a great hosting platform in the last year, it's attractive to host packages on the platform and this also shows in the number of package authors that have decided to switch over to PyPI for hosting. The ones that don't, will have good reasons not to host on PyPI. We need to respect those reasons, not question them, and, if possible, improve the infrastructure to make it more inviting for them to join in. We should also reach out to the package authors that currently do not host on PyPI to find out what their reasons are and whether we can do something to help convince them. Note: I haven't yet read the whole thread on the distutils list, but do want everyone to keep the above in mind when discussing the topic. Thank you, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 11.05.2014 16:48, Paul Moore wrote: On 11 May 2014 13:47, Donald Stufft don...@stufft.io wrote: https://pypi.python.org/simple/egenix-mx-base/ has verifiable external links. I'm pretty surprised that Donald hasn't heard of mx-base. egenix-mx-base does not have verifiable external links.Verifiable external links must be both directly linked to from the /simple/ index page and must include a hash. egenix-mx-base does not do this. OK, that clarifies that, and also makes it clear that what constitutes safe is not immediately obvious (something you've been saying a lot, but which never eally hit home to me before). So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Note that we also provide MD5, SHA1 hashes and GPG signature for all packages, so users get more security, not less :-) We had wanted to register links to the download files directly using the PyPI API and may still implement this (even though it gets difficult to admin with so many links per release), but have since shifted focus to working on a web installer which solves multiple problems at once: * solving the problem of choosing the right file to download * making sure downloads are verified for all Python versions we support * adding other features like automatically requesting and installing evaluation licenses which we would like to have for our commercial products * making all of the above possible with multiple installers such as pip, easy_install, conda, etc. including older versions of those installers With the web installer, we'd just have to upload one file per release. PS: Thanks for pointing the broken link on the download page. This is caused by copying the index page from our normal PyPI-style simple index to a fixed URL at release, which is done to make sure that the registered page content hash doesn't change when we recreate our index. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)
On 12 May 2014 21:34, M.-A. Lemburg m...@egenix.com wrote: Think about it: PyPI has become a great hosting platform in the last year, it's attractive to host packages on the platform and this also shows in the number of package authors that have decided to switch over to PyPI for hosting. The ones that don't, will have good reasons not to host on PyPI. We need to respect those reasons, not question them, and, if possible, improve the infrastructure to make it more inviting for them to join in. We should also reach out to the package authors that currently do not host on PyPI to find out what their reasons are and whether we can do something to help convince them. Note: I haven't yet read the whole thread on the distutils list, but do want everyone to keep the above in mind when discussing the topic. When you get to the end of that thread, you'll find that we reached two main conclusions: 1. Wanting to avoid US export laws is still an excellent reason for not wanting to host files directly on PyPI (and we really only need one such reason to accept external hosting as a use case that needs to continue to be handled)* 2. The user experience of the current external hosting arrangements really is incredibly poor, especially since there are *two* competing external hosting mechanisms (the verifiable link spidering and specifying additional index pages) and the implementation requirements for the link spidering are also limiting pip's ability to provide decent error messages for some failure modes. To reduce user confusion (and simplify the pip implementation so it can start providing better error messages), we'd like to consolidate that down to just one external hosting mechanism over the next couple of pip releases: additional custom index pages, as that's the more powerful and flexible of the two systems, as well as the one that's easiest to explain. That will mean a few different ideas need to be explored/defined: 1. A deprecation process and timeline for the link spidering mechanism (including a grandfather clause for existing packages) 2. An improved user experience for discovery of custom index server requirements via PyPI 3. A simple process for publishing custom index pages (perhaps these could even be PyPI hosted?) 4. A simple client configuration process for custom index servers (perhaps with the ability to get custom servers pre-approved by the pip developers and included in the default pip configuration) 5. Exploring the possibility of the PSF (or, if necessary, another trusted entity incorporated outside the US) providing a secondary hosting service located in Europe which would allow non-US users to publish packages without needing to agree to be subject to US export restrictions Donald is planning to start working on a PEP this week to propose this transition from implicit link spidering to an explicitly federated model that is closer to the way Linux distros have historically worked (one where you can register multiple trusted package sources locally, and the installer is preconfigured with a set of trusted sources, but, aside from mirroring networks, there's no way for one source to implicitly delegate trust to another location). Regards, Nick. P.S. *Regarding the reasons for external hosting, I also acknowledge that there are currently some understandable concerns with the current wording of the distribution license terms for PyPI, but I believe that particular issue is best addressed by clarifying the wording of the terms to make it clear they don't override the packaging licensing terms in general, but are instead a supplemental license ensuring that PyPI mirroring, along with automated installation and use of packages is always permitted for all uploaded packages. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
If this was apt-get or yum, there would be no concept of hosting apart from an index and you would have to run a command like apt-add-repository http://xyz.com; or place a file in /etc/... Then the extra repository + packages would become available. On Mon, May 12, 2014 at 8:28 AM, M.-A. Lemburg m...@egenix.com wrote: On 11.05.2014 16:48, Paul Moore wrote: On 11 May 2014 13:47, Donald Stufft don...@stufft.io wrote: https://pypi.python.org/simple/egenix-mx-base/ has verifiable external links. I'm pretty surprised that Donald hasn't heard of mx-base. egenix-mx-base does not have verifiable external links.Verifiable external links must be both directly linked to from the /simple/ index page and must include a hash. egenix-mx-base does not do this. OK, that clarifies that, and also makes it clear that what constitutes safe is not immediately obvious (something you've been saying a lot, but which never eally hit home to me before). So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Note that we also provide MD5, SHA1 hashes and GPG signature for all packages, so users get more security, not less :-) We had wanted to register links to the download files directly using the PyPI API and may still implement this (even though it gets difficult to admin with so many links per release), but have since shifted focus to working on a web installer which solves multiple problems at once: * solving the problem of choosing the right file to download * making sure downloads are verified for all Python versions we support * adding other features like automatically requesting and installing evaluation licenses which we would like to have for our commercial products * making all of the above possible with multiple installers such as pip, easy_install, conda, etc. including older versions of those installers With the web installer, we'd just have to upload one file per release. PS: Thanks for pointing the broken link on the download page. This is caused by copying the index page from our normal PyPI-style simple index to a fixed URL at release, which is done to make sure that the registered page content hash doesn't change when we recreate our index. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote: So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for safe external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. since shifted focus to working on a web installer which solves multiple problems at once: [...] With the web installer, we'd just have to upload one file per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a wheel cache, where the normal install route will be pip wheel; pip install the generated wheel, bypassing setup.py install totally). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
Paul Moore p.f.mo...@gmail.com wrote: I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, To me that sounds like a lot of work for many package authors. Is it possible to reconsider the deprecation? Stefan Krah ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12.05.2014 15:58, Paul Moore wrote: On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote: So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for safe external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. since shifted focus to working on a web installer which solves multiple problems at once: [...] With the web installer, we'd just have to upload one file per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a wheel cache, where the normal install route will be pip wheel; pip install the generated wheel, bypassing setup.py install totally). Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 12 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On May 12, 2014, at 3:58 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 15:58, Paul Moore wrote: On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote: So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for safe external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. since shifted focus to working on a web installer which solves multiple problems at once: [...] With the web installer, we'd just have to upload one file per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a wheel cache, where the normal install route will be pip wheel; pip install the generated wheel, bypassing setup.py install totally). Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___
Re: [Distutils] PEP 438, pip and --allow-external
On 12.05.2014 22:31, Donald Stufft wrote: On May 12, 2014, at 3:58 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 15:58, Paul Moore wrote: On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote: So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for safe external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. since shifted focus to working on a web installer which solves multiple problems at once: [...] With the web installer, we'd just have to upload one file per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a wheel cache, where the normal install route will be pip wheel; pip install the generated wheel, bypassing setup.py install totally). Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py will go away. Thanks for the clarification, -- Marc-Andre Lemburg eGenix.com Professional
Re: [Distutils] PEP 438, pip and --allow-external
On May 12, 2014, at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 22:31, Donald Stufft wrote: On May 12, 2014, at 3:58 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 15:58, Paul Moore wrote: On 12 May 2014 13:28, M.-A. Lemburg m...@egenix.com wrote: So, some questions: 1. Is MAL aware that egenix-mx-base is not verifiably externally hosted[1], and if so, what is he asking for? Automatic download with no need for opt-in of unverifiable external downloads? That seems pretty much in conflict with the whole intent of PEP 438. What we are implementing is a proposal that I brought up before PEP 438 was put in place: Instead of linking directly to all packages, we put up a verifiable link to an index page with verifiable links, with the net effect being that tools can verify the whole chain. Thanks for clarifying. My main concern is that people do actually understand the requirements for safe external hosting (I discovered that I certainly didn't - it's quite complex!). From what you say, you do understand the requirements but have chosen not to follow them at this time. That's fine, I'm not trying to debate the validity of your choice. But in the context of PEP 438, that means that users of the egenix downloads will have to opt into unverifiable downloads in order to install using pip. We're working towards improving the user experience for end users who need to install unverifiable downloads - I'd expect that one consequence of this will be that we'll be better able to communicate the situation to those users, which will reduce the confusion. If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. I don't know if you're suggesting that your proposal is still something that we should be looking at implementing. I'm happy to discuss that, but it should probably be a separate thread. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. since shifted focus to working on a web installer which solves multiple problems at once: [...] With the web installer, we'd just have to upload one file per release. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, we will likely miss your use case unless you get involved in those discussions (they are on the back burner at the moment, but will likely resurface at some point - there's currently a work-in-progress PR for pip to use a wheel cache, where the normal install route will be pip wheel; pip install the generated wheel, bypassing setup.py install totally). Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py
Re: [Distutils] PEP 438, pip and --allow-external
On 12.05.2014 22:37, Donald Stufft wrote: On May 12, 2014, at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py will go away. Thanks for the clarification, I should expand on that answer, that sdist 2.0 will ideally include static metadata but it won't be a static build system. Things like name, version, dependencies etc will be static, but building will be done by executing some code. Now, you've got me really confused. Is this something I can read up somewhere ? sdists already includes static package information in the PKG-INFO file (format 1.0, but that could be changed to e.g. 2.0). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external
On May 12, 2014, at 4:50 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 22:37, Donald Stufft wrote: On May 12, 2014, at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py will go away. Thanks for the clarification, I should expand on that answer, that sdist 2.0 will ideally include static metadata but it won't be a static build system. Things like name, version, dependencies etc will be static, but building will be done by executing some code. Now, you've got me really confused. Is this something I can read up somewhere ? sdists already includes static package information in the PKG-INFO file (format 1.0, but that could be changed to e.g. 2.0). It's really not written up anywhere yet because nobody has done any work on it yet. Most the work in that area has been focused on Metadata 2.0. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external
On May 12, 2014, at 4:51 PM, Donald Stufft don...@stufft.io wrote: On May 12, 2014, at 4:50 PM, M.-A. Lemburg m...@egenix.com wrote: On 12.05.2014 22:37, Donald Stufft wrote: On May 12, 2014, at 4:33 PM, M.-A. Lemburg m...@egenix.com wrote: Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I don't think that's a good idea. It still is a very good idea to make more meta data available in static non-executable form in order to simplify finding packages, determining dependencies, features, enhancing the PyPI UI, and for deciding which distribution file to download and install. This can be generated by setup.py as part of the build process and made available to PyPI during package release registration (much like it is now, only in extended form). (*) This does work if you are only targeting a few select systems and system versions, but the Python user base out just has too many diverse setups to be able to address them all to be able to completely drop setup.py. This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. Ah, so this is just a misunderstanding on my part then. I thought Paul was saying that having pip run setup.py will go away. Thanks for the clarification, I should expand on that answer, that sdist 2.0 will ideally include static metadata but it won't be a static build system. Things like name, version, dependencies etc will be static, but building will be done by executing some code. Now, you've got me really confused. Is this something I can read up somewhere ? sdists already includes static package information in the PKG-INFO file (format 1.0, but that could be changed to e.g. 2.0). It's really not written up anywhere yet because nobody has done any work on it yet. Most the work in that area has been focused on Metadata 2.0. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig I’ll try to write up a more coherent thing sometime this week. I’m working on a PEP atm and I’m a little out of it from a root canal I had earlier today. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Need for respect (was: PEP 438, pip and --allow-external)
On 12 May 2014 16:57, Stefan Krah stefan-use...@bytereef.org wrote: Thank you for your measured responses, and I agree with you that pip should follow PEP 438. The main argument on python-dev was about *editorializing* the contents of the PEP in both pip warning messages and posts to the mailing lists (and by that I certainly do *not* mean your posts). My feeling is that there was not as much deliberate editorializing going on as may have seemed the case at first. More that people interpreted the situation and the intent of the PEP differently - there is a *lot* of confusion and misunderstanding in this whole area. Many people, notably myself, have a fairly shallow understanding of security issues, and find the more security-oriented explanations pretty confusing (and, unfortunately, somewhat paranoid - I say unfortunately because it's all too easy to misunderstand or dismiss a key point simply because of a difference in mindset). Also, I'd have to say that packaging PEPs have traditionally been pretty divergent from reality, so people could be excused for thinking it couldn't possibly mean X. I've definitely been guilty of arguing based on what I *think* is going on rather than checking the PEP and the code. Come to that, I honestly don't know if anyone has checked the pip implementation against the PEP (I believe they match, that's the best I can say). And from what I recall of the PEP discussion, I'm not sure it reflected the sort of issues being raised now, so I suspect more than one person either didn't get involved in the discussion, or didn't spot issues that affected them (from what I recall, the PEP discussion felt more like a technical PyPI internals issue than a significant user interface debate). The initial implementation in pip didn't help the situation at all, as it described the situation very badly (one example of this is the warning that you mention). We're improving that right now, but I doubt the next iteration will be perfect, just (hopefully) better. The PEP appears to say: Installers should provide a blanket option to allow installing any verifiable external link. Perhaps something like --allow-verifiable-external would do? I would not be unhappy if link-spidering were to be removed, I find it reasonable to provide the explicit link. I believe that option has been there for a while as --allow-[all]-external. Again, naming and discoverability may be an issue, but the functionality is available. One issue *may* be that it's not clear to everyone what verifiable involves. In another thread I'm trying to clarify with MAL over his understanding of how the egenix packages are linked. There is a chain of links with hashes, but the PEP doesn't allow for a chain to be verifiable, just a direct link to a downloadable file. Is that what you mean by link-spidering? If so, then as it stands link-spidering is allowed, but will never be verifiable. I could easily imagine some package maintainers feeling that characterising a 2-link chain as not verifiable is inaccurate and/or scaremongering. Suggestions for better terminology would be useful here. (Note that direct links vs link chains is an important distinction for pip, as it has performance implications as well as security ones - link spidering was a huge performance issue at one point, and a lot of the non-controversial benefits in PEP 438 are in terms of performance. It would be a shame to lose those.) But don't take that too seriously, some important looking packages rely on link spidering: https://pypi.python.org/pypi/bzr/2.6.0 https://pypi.python.org/pypi/meliae/0.4.0.final.0 https://pypi.python.org/pypi/CobraWinLDTP/4.0.0 https://pypi.python.org/pypi/PloneIISApp/4.2 https://pypi.python.org/pypi/which/1.1.0 https://pypi.python.org/pypi/python-cjson-custom/1.0.3x7 https://pypi.python.org/pypi/CAGE/1.1.4 https://pypi.python.org/pypi/libavg/1.8.0 https://pypi.python.org/pypi/BlogBackup/1.4 https://pypi.python.org/pypi/Polygon3/3.0.6 https://pypi.python.org/pypi/Pygame/1.8.0 https://pypi.python.org/pypi/DOLFIN/1.3.0 https://pypi.python.org/pypi/snippet/2.3.6.1 https://pypi.python.org/pypi/savReaderWriter/ https://pypi.python.org/pypi/PyCY/1.0.0 https://pypi.python.org/pypi/WikklyText/1.6.0 https://pypi.python.org/pypi/python-lzo/1.08 https://pypi.python.org/pypi/blockcanvas/4.0.3 https://pypi.python.org/pypi/boolopt/1.0 https://pypi.python.org/pypi/egenix-mx-base/3.1.0 https://pypi.python.org/pypi/egenix-mx-commercial/2.0.7 https://pypi.python.org/pypi/python-musicbrainz2/0.7.2 https://pypi.python.org/pypi/OpenAlea/0.9 Many of those I've never heard of, which shows how hard it is to spot important stuff. Right now, all of these will need an --allow-unverifiable flag - and nobody (yet...) has suggested that this rule is weakened. The only thing that will change the user experience is if the projects add
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12 May 2014 17:15, Stefan Krah stefan-use...@bytereef.org wrote: Paul Moore p.f.mo...@gmail.com wrote: I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. As the general trend is towards binary installs using wheels, with pip eventually deprecating sdist installs and running setup.py directly, To me that sounds like a lot of work for many package authors. Is it possible to reconsider the deprecation? MAL also commented on this. Taking this point any further is going to derail the current discussion so I propose that we don't go into any more details just now. But I note your concern and will make sure you and MAL are brought into any discussion well before anything gets implemented. One thing I will say is that as far as I am aware, a significant majority of packages already support pip wheel with no action needed from the authors. The proposal is little more than automating the current 2-step manual process that's needed. Outstanding points: 1. Projects need to either use setuptools, or at least not conflict with setuptools if pip injects it into setup.py. That's not a battle I want to fight over pip - whether setuptools is the de facto standard distutils extension has been debated endlessly and I'm pretty sure it's a done deal by now. I'm pretty sure that Nick, for example, has been involved in discussions where he's supported that stance, and while I don't know about others, I took his view as being (at least to some extent) on behalf of python-dev. (Apologies Nick if I'm misrepresenting you). Ultimately, pip's going to have a hard time working with a package that won't tolerate setuptools injection, and if that's a problem then so be it. 2. For a minority of projects, pre and post install scripts might be needed when installing wheels. That's coming in Metadata 2.0. Let's table any further discussion on this for now. I will make sure that doing so doesn't mean your views (and MAL's) get missed. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12 May 2014 20:58, M.-A. Lemburg m...@egenix.com wrote: If it helps convince you that allowing verifiable external links per default is a good thing for user experience, we will register the distribution file download URLs with the PyPI web API. Personally, I'm on the fence over that one. There are so few projects with verifiable external links that it's hard to be sure where the costs and gains are. Certainly if more projects use the deature that will affect the decision, but I'm not going to be the one to say how much weight individual projects carry... To be clear, --allow-external and --allow-all-external exist right now. That matches PEP 438. Either of making --allow-all-external the default or treating both verifiable and unverifiable external links as opt-in under the same option would be changes to both the PEP and pip, and neither has happened yet. You can think of it as per-package PyPI-style simple index that's registered with PyPI in a secure way (via the check sum hash). There's most certainly room for improvement and I'm open for discussions. That sounds like an interesting proposal, and worth discussing. I presume it would need a PEP to implement, as there would be impacts on pip, PyPI and warehouse at a minimum, and possibly easy_install and at some point distil/distlib. I'm not quite sure how you expect this will work, but it's probably important that you get involved with the various packaging PEPs. The only way I can see such a solution working with pip would be if you have a customised setup.py. Yep, since that's the way installers (not only pip) work when they see a source distribution. OK. The move away from having executable code run at install time is well outside the scope of this debate or any discussion around pip at the current point. You might want to look for threads mentioning the meta build system for current thinking, but it's very much vapourware at the moment. Binary installs are nice, but they are not the answer to everything and no matter how much meta data you put into static files, there will always be cases where that meta data description just doesn't include those bits you would need to complete the setup, esp. for packages with C extensions and more complex external dependencies or setup requirements. (*) The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. See my response to Stefan, and those threads about the meta build system I mentioned earlier. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. Nothing is being designed in private, but any discussions that do happen will be on distutils-sig, so I'd recommend monitoring that list so you don't get taken by surprise. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
On 12 May 2014 21:31, Donald Stufft don...@stufft.io wrote: This is slightly confusing but pip will always be able to go from an sdist to an installed system. It'll just build a Wheel first and then install the Wheel (at least that's the idea). This is a sort of vague idea right now but it's the direction we want to go in. MAL's concern (if I read his comments correctly) is that people who use complex setup.py arrangements to do things like extend commands in setuptools-incompatible ways may not support build to wheel then install as a viable approach, whether it's done manually or as an internal step in pip. This is one of the fundamental issues with distutils, and is the source of much of the old corrosive distutils must die rhetoric that hurt previous distutils-sig debates. The current approach is to solve 90% of the problem by noting that nearly all projects don't take advantage of any of the (usually undocumented) flexibility that distutils allows. This has thus far been a great success, in terms of getting people on board with the new tools and technologies. The problem is that it does nothing for that remaining 10%[1] We're now at a point where people like MAL, who are in that 10%, are getting excluded and we don't have a good answer for them. That's a big problem - particularly as the people in question are quite often those to whom we owe a lot of the progress that distutils gave us (I remember the pre-distutils world, and it *SUCKED*). I wish we had an answer here. I'm particularly worried that some proportion of the 10% may be complex in-house environments that we'll never hear about till we break them horribly. But I don't know what we can do. We need to be able to move forwards, and the lack of encapsulation in distutils means we will break things when we do. Now I'm depressed... Paul [1] 95% of all statistics quoted on the internet were made up on the spot. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metabuild hooks
On 13 May 2014 08:03, Paul Moore p.f.mo...@gmail.com wrote: The current approach is to solve 90% of the problem by noting that nearly all projects don't take advantage of any of the (usually undocumented) flexibility that distutils allows. This has thus far been a great success, in terms of getting people on board with the new tools and technologies. The problem is that it does nothing for that remaining 10%[1] We're now at a point where people like MAL, who are in that 10%, are getting excluded and we don't have a good answer for them. That's a big problem - particularly as the people in question are quite often those to whom we owe a lot of the progress that distutils gave us (I remember the pre-distutils world, and it *SUCKED*). I wish we had an answer here. I'm particularly worried that some proportion of the 10% may be complex in-house environments that we'll never hear about till we break them horribly. But I don't know what we can do. We need to be able to move forwards, and the lack of encapsulation in distutils means we will break things when we do. Now I'm depressed... Note that this problem is specifically why the metadata 2.0 extension design now has the concept of mandatory extensions - so we can later define an installation hook system (along the lines of RPMs triggers), and have installers that don't understand the relevant extension fall back to installing directly from source. We should make that fall back to running 'setup.py install' directly if you don't understand a metadata extension in the wheel file aspect explicit, but either a metabuild system spec or the revision of the wheel spec that adds metadata 2.0 support would likely be the right place for that, rather than having it directly in the metadata 2.0 details (on the other hand, it may also make sense to include as an example use case for the attribute in the PEP 426 explanatory notes). Cheers, Nick. Paul [1] 95% of all statistics quoted on the internet were made up on the spot. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438, pip and --allow-external (was: pip: cdecimal an externally hosted file and may be unreliable from python-dev)
The setup.py interface makes all this possible, which is why so many Python packages use it to configure themselves automatically. Deprecating this interface would make some distributions impossible to install without manual user intervention and we'd be back to the Makefile.pre.in days. I thought there was a general consensus to move *away* from the need to run setup.py when it's practicable to do so. There was no choice about this with distutils and setuptools, but going forwards, I'm not sure about impossible to install as long as *some* mechanism is there for package publisher-defined code to run at installation time. However, setup.py as it is now is a complete free-for-all where anything goes, which I don't think is a good thing. Many PyPI packages are installable with distil (which doesn't use setup.py at installation time), and that includes packages with C extensions, Cython, and even Fortran. The packages distil has problems with are those that do significant things in setup.py, such as moving files around in the source tree, generating new source files, subclassing distutils so you can't see what the actual operations being carried out are, etc. I'm fairly sure that all of these things can be accommodated using installation time-hooks with sensible APIs rather than the ad hoc-ery of setup.py. Of course I'm not suggesting that publishers have to rework existing releases - it's just that the setup.py scheme is a bit archaic and not entirely problem-free. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
