Re: [Distutils] Immutable Files on PyPI
Thanks for the confirmation that my interpretation was wrong. This makes things a lot better :-) More below... On 29.09.2014 11:39, Nick Coghlan wrote: On 29 Sep 2014 18:49, M.-A. Lemburg m...@egenix.com wrote: You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. Such changes often do not affect the software itself, and so are not detected by software tests. Fixing such packaging errors is the primary intended use case of the post field in PEP 440. Alternatively, many projects will just spin a new release that addresses those issues, just as they would for any other bug. Both of those approaches have the advantage of letting users (especially those operating mirrors, or downloading tarballs and feeding them into a separate redistribution system) easily tell whether or not they have the fixed version. I don't see how that would help. AFAIU, PyPI would regard a 3.1.4.post1 as new release and so invalidate all other already uploaded distribution files rather than just regard the fixed upload as update to the 3.1.4 release. If we could get a widely adopted notion of build numbers into the tools that would help a lot. Installers and PyPI would then regard 3.1.4-1 as belonging to release 3.1.4, but being a more current build as a distribution file carrying 3.1.4 in its file name. This would also have to work for all files uploaded for a release, not only for some distribution formats, of course, including source release files, Windows installers, egg files, etc. I'd have to run some experiments, but don't believe that setuptools and associated tools support this at the moment. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. No, deletion remains supported. The only capability being removed is silent substitution of hosted files with different ones bearing the same name. So if, for example, release 3.1.4 had a packaging error, then deleting it would still be possible, but the replacement would need to be something like 3.1.4.post1 or 3.1.5, rather than being permitted to reuse the 3.1.4 name. So just to summarize: the proposal is to turn PyPI into a WORM, but at least it's still possible to remove distribution files. The external hosting support is then the mechanism by which authors can retain complete and total control over their release process. That approach avoids all licensing concerns (including those related to US export controls), as well as ensuring they have the ability to silently change the contents of previously released files if they choose to do so (although, as noted above, actually doing so may break installation for anyone installing with peep, which checks file hashes based on the first version downloaded). You're regularly bringing up this argument. Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment. pip is unfortunately using the same kind of --no-one-will-want-to-use-this-option-because-its-too-long approach as setuptools/easy_install has done in the past to force people into installing packages as eggs rather than installing the packages in the standard write to site-packages dir way. The end result is the same: users will not want to go through those extra hoops and thus packages not hosted on PyPI itself will be regarded as broken (because they don't install using the standard method; not because they are really broken). This is what I'm trying to address in discussions like these all along. PyPI has a responsibility not only for the part consuming part of the Python community, but also for the creating part of it. While PyPI is great for indexing packages, it's not necessarily the best answer for hosting the distribution files and I believe we should open up some more to allow for making it possible to offer the same kind of user experience while not making pypi.python.org the only source of distribution files. In the Linux world, this already works great by having multiple repos which you can switch on/off easily. For eGenix, we've found a way to work around the PyPI constraints: http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/ which addresses our user's problems. Still, we'd much rather use standard ways of working *with* PyPI rather than work around it. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 30 2014) Python Projects, Consulting and Support ...
Re: [Distutils] Immutable Files on PyPI
On 30 Sep 2014 19:06, M.-A. Lemburg m...@egenix.com wrote: You're regularly bringing up this argument. Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment. pip is unfortunately using the same kind of --no-one-will-want-to-use-this-option-because-its-too-long approach as setuptools/easy_install has done in the past to force people into installing packages as eggs rather than installing the packages in the standard write to site-packages dir way. The end result is the same: users will not want to go through those extra hoops and thus packages not hosted on PyPI itself will be regarded as broken (because they don't install using the standard method; not because they are really broken). I personally think pip needs a virtualenv friendly equivalent to yum.repos.d or conda channels to improve the multi-index experience. This is a solved problem from my perspective, we just need folks willing to spend the time making the case for the proven solution, and adapting it to the PyPI ecosystem. This is what I'm trying to address in discussions like these all along. PyPI has a responsibility not only for the part consuming part of the Python community, but also for the creating part of it. While PyPI is great for indexing packages, it's not necessarily the best answer for hosting the distribution files and I believe we should open up some more to allow for making it possible to offer the same kind of user experience while not making pypi.python.org the only source of distribution files. In the Linux world, this already works great by having multiple repos which you can switch on/off easily. And I've said all along that's the experience I would like to see in upstream Python as well. But the code isn't going to write itself, and there's also the political work needed to persuade folks that it's the right path to go down. For eGenix, we've found a way to work around the PyPI constraints: http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/ which addresses our user's problems. Still, we'd much rather use standard ways of working *with* PyPI rather than work around it. Sure, I'd like to see that to. It's just only one problem amongst a great many of them, and people's upstream work is driven by their own balancing of a complex set of priorities. Cheers, Nick. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 30 2014) Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-09-30: Python Meeting Duesseldorf ... today : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 30 September 2014 10:06, M.-A. Lemburg m...@egenix.com wrote: Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment. This is a fair point. But it is an acknowledged problem in pip and we're working on resolving it. See http://legacy.python.org/dev/peps/pep-0470/. The problem (as with anything else) is that someone needs to write the code, and that hasn't happened yet. We'll get to it, but the usual PRs welcome comment applies here :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On September 30, 2014 at 5:07:06 AM, M.-A. Lemburg (m...@egenix.com) wrote: Thanks for the confirmation that my interpretation was wrong. This makes things a lot better :-) More below... On 29.09.2014 11:39, Nick Coghlan wrote: On 29 Sep 2014 18:49, M.-A. Lemburg wrote: You are missing out on cases, where the release process causes files to be omitted, human errors where packagers forget to apply changes to e.g. documentation files, version files, change logs, etc., where packagers want to add information that doesn't affect the software itself, but meta information included in the distribution files. Such changes often do not affect the software itself, and so are not detected by software tests. Fixing such packaging errors is the primary intended use case of the post field in PEP 440. Alternatively, many projects will just spin a new release that addresses those issues, just as they would for any other bug. Both of those approaches have the advantage of letting users (especially those operating mirrors, or downloading tarballs and feeding them into a separate redistribution system) easily tell whether or not they have the fixed version. I don't see how that would help. AFAIU, PyPI would regard a 3.1.4.post1 as new release and so invalidate all other already uploaded distribution files rather than just regard the fixed upload as update to the 3.1.4 release. If we could get a widely adopted notion of build numbers into the tools that would help a lot. Installers and PyPI would then regard 3.1.4-1 as belonging to release 3.1.4, but being a more current build as a distribution file carrying 3.1.4 in its file name. This would also have to work for all files uploaded for a release, not only for some distribution formats, of course, including source release files, Windows installers, egg files, etc. I'd have to run some experiments, but don't believe that setuptools and associated tools support this at the moment. I don’t personally believe it makes sense for a source distribution to have a build number. Yes it would invalidate all other uploads because it should, They take the source distribution as an input, if you change the input then in all likelihood you change the output. This won’t be true in every case but to be able to determine when and where it won’t be true requires intimate knowledge of the package formats as well as the project in question. On the other hand Wheel files already support the concept of build numbers, I don’t see any technical reason why Windows installers can’t support them given that nothing automated pulls them down. Getting Egg files to support them may be a lost cause though since I’m not sure there’s a way to add that information to the Egg spec without mandating a new setuptools. If I understand you correctly, you are essentially suggesting that it becomes impossible to ever delete anything uploaded to PyPI, i.e. turning PyPI into a WORM. No, deletion remains supported. The only capability being removed is silent substitution of hosted files with different ones bearing the same name. So if, for example, release 3.1.4 had a packaging error, then deleting it would still be possible, but the replacement would need to be something like 3.1.4.post1 or 3.1.5, rather than being permitted to reuse the 3.1.4 name. So just to summarize: the proposal is to turn PyPI into a WORM, but at least it's still possible to remove distribution files. The external hosting support is then the mechanism by which authors can retain complete and total control over their release process. That approach avoids all licensing concerns (including those related to US export controls), as well as ensuring they have the ability to silently change the contents of previously released files if they choose to do so (although, as noted above, actually doing so may break installation for anyone installing with peep, which checks file hashes based on the first version downloaded). You're regularly bringing up this argument. Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment. pip is unfortunately using the same kind of --no-one-will-want-to-use-this-option-because-its-too-long approach as setuptools/easy_install has done in the past to force people into installing packages as eggs rather than installing the packages in the standard write to site-packages dir way. The end result is the same: users will not want to go through those extra hoops and thus packages not hosted on PyPI itself will be regarded as broken (because they don't install using the standard method; not because they are really broken). This is what I'm trying to address in discussions like these all along. PyPI
Re: [Distutils] Immutable Files on PyPI
On 30 September 2014 21:30, Donald Stufft don...@stufft.io wrote: In pip 1.5.x (current latest release) people can add additional indexes (or replace PyPI all together) on a per user basis. In pip 6.0 (next release) that still exists, and in addition it also includes a global configuration file for the entire machine and a per virtual env configuration file. About the only thing it doesn't support is a directory of files ala apt.sources.d/*.list. Donald I chatted about this off-list, and the base URL is currently the only configuration setting pip retains per index. While we may still want to explore named indices at some point in the future, at this point in time, they wouldn't add a lot over just using the index URL directly. So I think this is basically already covered on the pip side (once 6.0 goes out), and then PEP 470 aims to improve the user experience for external index discovery. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Wheels and dependent third party dlls on windows
Hi, I was wondering what is the recommended approach to bundling runtime dll dependencies when using wheels. We are migrating from egg to wheels for environment installation and of various python dependencies. Some of those have extension modules, and some have extension modules that depend on the presence of a third party dll (in our situation, libzmq-v100-mt-4_0_3.dll). Up to now, these dlls have been installed by the use of the scripts parameter in the setup command of setup.py, but https://mail.python.org/pipermail/distutils-sig/2014-July/024554.html points to it as not being a good idea. But the only way to get a dependent dll found on windows is to have it on PATH, and the scripts directory on windows is on path when a virtualenv is activated. I have observed two situations: 1) If we use pip wheel to build the wheel, the scripts parameter is ignored and the dlls do not even get to the archive. 2) If we use setup.py bdist_wheel, the dll gets into the archive, but this relies on the non-documented feature of packaging scripts-as-data of dlls. What is the correct approach at this time ? Thanks, David ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 30 September 2014 14:32, David Genest david.gen...@ubisoft.com wrote: But the only way to get a dependent dll found on windows is to have it on PATH, and the scripts directory on windows is on path when a virtualenv is activated. This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd file. You should therefore be able to ship the dependent dll in the package directory (which wheels support fine). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling Case/Normalization Differences
On 28 August 2014 19:58, Donald Stufft don...@stufft.io wrote: To fix this I'm going to modify PyPI so that it uses the normalized name in the /simple/ URL and redirects everything else to the non-normalized name. I'm also going to submit a PR to bandersnatch so that it will use normalized names for it's directories and such as well. These two changes will make it so that the client side will know ahead of time exactly what form the server expects any given name to be in. This will allow a change in pip to happen which will pre-normalize all names which will make the interaction with mirrors better and will reduce the number of HTTP requests that a single ``pip install`` needs to make. Just to clarify, this means that if I want to find the simple index page for a distribution, without hitting redirects, I should first normalise the project name (so Django becomes django) and then request https://pypi.python.org/simple/normalised_name/ (with a slash on the end). Is that correct? It seems to match what I see in practice (in particular, the version without a terminating slash redirects to the version with a terminating slash). The JSON API has the opposite behaviour - https://pypi.python.org/pypi/Django/json redirects to https://pypi.python.org/pypi/django/json. Should that not be changed to match? Will it be? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd file. You should therefore be able to ship the dependent dll in the package directory (which wheels support fine). Paul Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? Thanks for your response, D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 2014-09-30 07:26:32 -0400 (-0400), Donald Stufft wrote: [...] I don’t personally believe it makes sense for a source distribution to have a build number. [...] I'm becoming less and less convinced it actually *is* a source distribution any more. My constant interaction with downstream Linux distro packagers shows a growing disinterest in consuming release tarballs of software, that they would generally prefer to pull releases directly from tags in the project's revision control systems instead. Couple this with the fact that setup.py sdist can (and often does) include autogenerated content in its output which the packagers would rather strip or regenerate themselves, and I'm of the opinion that the tarballs I create are only for PyPI/pip consumption any longer. This effectively makes them a channel-specific packaging format rather than a generally reusable release source artifact. -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 30 September 2014 15:31, David Genest david.gen...@ubisoft.com wrote: Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? I honestly don't know in that case, sorry. You might get a better answer on python-list for that, if no-one here can help. Presumably the usage is all within one distribution, otherwise the question would have to be, which distribution ships the DLL? But that question ends up leading onto the sort of discussion that starts well, I wouldn't design your system the way you have, which isn't likely to be of much help to you :-( Sorry I can't offer any more help. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 2014-09-30 11:06:40 +0200 (+0200), M.-A. Lemburg wrote: [...] You're regularly bringing up this argument. Let's just be fair here: external hosting of packages has been made so user unfriendly in recent pip releases, that this has pretty much become a non-option for anyone who wants to create a user friendly package installation environment. [...] And I'm seeing this argument regularly brought up as well. As a heavy user of Python packages and someone who maintains very large systems depending on them, I had a hard time trusting pip back when it still would spontaneously grab software from wherever the upstream author had decided to stick it that day (with whatever hosting stability issues that implied). Our projects would regularly audit our hundreds of requirements just to make sure nobody *accidentally* added one which was hosted off PyPI, and that our dependencies hadn't suddenly decided to start sticking new releases off PyPI. The suggestion that some developers want to control their release process *so* tightly that they host their software in random places without uploading them to the community package system or quietly replace broken releases with new packages using the same version numbers is a non-argument as far as I'm concerned. The software authors I've talked to in these cases are pretty much always easily convinced that those are troublesome behaviors (once it's pointed out) and readily adjust their release processes to a more user-friendly end result. If there are a few who are so completely insistent on continuing in this manner the projects I work on will not use them (for our own sanity), and if pip and PyPI implement assurances against these which have a side effect of driving *those particular* development teams off of the community packaging channel then that can only be a positive net effect in my opinion. -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 1 October 2014 00:37, Paul Moore p.f.mo...@gmail.com wrote: On 30 September 2014 15:31, David Genest david.gen...@ubisoft.com wrote: Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? I honestly don't know in that case, sorry. You might get a better answer on python-list for that, if no-one here can help. Presumably the usage is all within one distribution, otherwise the question would have to be, which distribution ships the DLL? But that question ends up leading onto the sort of discussion that starts well, I wouldn't design your system the way you have, which isn't likely to be of much help to you :-( Sorry I can't offer any more help. Note that this is the external binary dependency problem that the scientific folks are currently using conda to address. It's basically the point where you cross the line from language specific packaging system to multi-language cross-platform platform. That said, pip/wheel *may* get some capabilities along these lines in the future, it just isn't a high priority at this point. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
Or you could just create a Python package that only contains the dll, and depend on it from your others. On Tue, Sep 30, 2014 at 10:44 AM, Nick Coghlan ncogh...@gmail.com wrote: On 1 October 2014 00:37, Paul Moore p.f.mo...@gmail.com wrote: On 30 September 2014 15:31, David Genest david.gen...@ubisoft.com wrote: Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? I honestly don't know in that case, sorry. You might get a better answer on python-list for that, if no-one here can help. Presumably the usage is all within one distribution, otherwise the question would have to be, which distribution ships the DLL? But that question ends up leading onto the sort of discussion that starts well, I wouldn't design your system the way you have, which isn't likely to be of much help to you :-( Sorry I can't offer any more help. Note that this is the external binary dependency problem that the scientific folks are currently using conda to address. It's basically the point where you cross the line from language specific packaging system to multi-language cross-platform platform. That said, pip/wheel *may* get some capabilities along these lines in the future, it just isn't a high priority at this point. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On 30 September 2014 15:45, Daniel Holth dho...@gmail.com wrote: Or you could just create a Python package that only contains the dll, and depend on it from your others. The problem is getting the DLL on PATH. What you could do is distribute a package containing: 1. The dll 2. An __init__.py that adds the package directory (where the DLL is) to os.environ['PATH']. If you import this package before any of the ones that depend on the DLL (or even in the __init__ of those packages) then you should have PATH set up correctly, and things will work as you need. I think this works, although I will admit it feels like a bit of a hack to me. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 30 September 2014 11:35, Robin Becker ro...@reportlab.com wrote: What would be the objection to removing or nulling a release package that had actual malware embedded in it some how. It seems reasonable to have some last resort take down mechanism. None at all. Removal is specifically still allowed. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 29/09/2014 10:50, Nick Coghlan wrote: On 29 Sep 2014 19:04, M.-A. Lemburg m...@egenix.com wrote: Do you seriously want to force package authors to cut a new release just because a single uploaded distribution file is broken for some reason and then ask all users who have already installed one of the non-broken ones to upgrade again, even though they are not affected ? Yes, I do. Silently changing released artefacts is actively user hostile. It breaks mirroring, it breaks redistribution, it breaks security audits, and it can even break installation for security conscious users that are using peep rather than pip. ... What would be the objection to removing or nulling a release package that had actual malware embedded in it some how. It seems reasonable to have some last resort take down mechanism. -- Robin Becker ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling Case/Normalization Differences
On 30 September 2014 15:25, Paul Moore p.f.mo...@gmail.com wrote: On 28 August 2014 19:58, Donald Stufft don...@stufft.io wrote: To fix this I'm going to modify PyPI so that it uses the normalized name in the /simple/ URL and redirects everything else to the non-normalized name. I'm also going to submit a PR to bandersnatch so that it will use normalized names for it's directories and such as well. These two changes will make it so that the client side will know ahead of time exactly what form the server expects any given name to be in. This will allow a change in pip to happen which will pre-normalize all names which will make the interaction with mirrors better and will reduce the number of HTTP requests that a single ``pip install`` needs to make. Just to clarify, this means that if I want to find the simple index page for a distribution, without hitting redirects, I should first normalise the project name (so Django becomes django) and then request https://pypi.python.org/simple/normalised_name/ (with a slash on the end). Is that correct? It seems to match what I see in practice (in particular, the version without a terminating slash redirects to the version with a terminating slash). The JSON API has the opposite behaviour - https://pypi.python.org/pypi/Django/json redirects to https://pypi.python.org/pypi/django/json. Should that not be changed to match? Will it be? One further thought. Where is the definition of how to normalise a name? I could probably dig through the pip sources and find it, but it would be nice if it were documented somewhere. From experiment, it seems like lowercase, and with hyphens rather than underscores, is the definition. Does PyPI allow names not allowed by http://legacy.python.org/dev/peps/pep-0426/#name and if it does, how are they normalised? In case it's not obvious, I'm writing a client for the PyPI API, and these questions are coming out of that process. Paul. PS The Python wiki has pages for the XMLRPC and JSON API. Any objections to me adding a page for the simple API? (The obvious objection being that it's documented somewhere else, and I should just put a pointer to the real documentation...) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 09/30/2014 08:41 AM, Nick Coghlan wrote: Why is your setup.py sdist including autogenerated content? It shouldn't be doing that. Don't almost all sdists? At the very least egg-info is autogenerated, MANIFEST usually is too. Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote: Installers and PyPI would then regard 3.1.4-1 as belonging to release 3.1.4, but being a more current build as a distribution file carrying 3.1.4 in its file name. Please don't literally use 3.1.4-1. That will cause all kinds of havoc with the Debian ecosystem. There we use a dash to separate upstream version numbers from Debian version numbers. Thus 3.1.4-1 means the first Debian upload of upstream's 3.1.4. 3.1.4-2 is the second, etc. Cheers, -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 30 Sep 2014, at 17:35, Barry Warsaw ba...@python.org wrote: On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote: Installers and PyPI would then regard 3.1.4-1 as belonging to release 3.1.4, but being a more current build as a distribution file carrying 3.1.4 in its file name. Please don't literally use 3.1.4-1. That will cause all kinds of havoc with the Debian ecosystem. There we use a dash to separate upstream version numbers from Debian version numbers. Thus 3.1.4-1 means the first Debian upload of upstream's 3.1.4. 3.1.4-2 is the second, etc. Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember the package version is considered to start after the last dash. Debian will also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging” might be necessary in other situations as well. Wichert. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 2014-09-30 09:22:29 -0600 (-0600), Carl Meyer wrote: On 09/30/2014 08:41 AM, Nick Coghlan wrote: Why is your setup.py sdist including autogenerated content? It shouldn't be doing that. Don't almost all sdists? At the very least egg-info is autogenerated, MANIFEST usually is too. Bingo. Also in some cases I see files autogenerated from VCS metadata to avoid double-entry... things like change histories, authors lists, documentation indices, and even version numbers. -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
Thank you very Steve for pushing that installer out, this is very appreciated. What is the story for project maintainers who want to also support Python 3.3+ (for 32 bit and 64 bit python) for their project with binary wheels for windows? At the moment it's possible to use the Windows SDK as documented here: http://scikit-learn.org/dev/install.html#building-on-windows However getting VC Express + Windows SDK is hard and slow to setup and cannot be scripted in a CI environment. In the mean time, it's possible to use CI environments that already feature all the necessary versions of the VC compilers and libraries such as appveyor.com, see this demo project: https://github.com/ogrisel/python-appveyor-demo https://ci.appveyor.com/project/ogrisel/python-appveyor-demo -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
On 30 September 2014 16:56, Olivier Grisel olivier.gri...@ensta.org wrote: What is the story for project maintainers who want to also support Python 3.3+ (for 32 bit and 64 bit python) for their project with binary wheels for windows? It would be so easy at this point to ask What's the chance of a similarly packaged version of VS2010 for Python 3.2/3.3/3.4? But I really don't want Steve to get into any trouble with people saying now look what you've started :-) Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
Olivier Grisel wrote: Thank you very Steve for pushing that installer out, this is very appreciated. What is the story for project maintainers who want to also support Python 3.3+ (for 32 bit and 64 bit python) for their project with binary wheels for windows? At the moment it's possible to use the Windows SDK as documented here: http://scikit-learn.org/dev/install.html#building-on-windows However getting VC Express + Windows SDK is hard and slow to setup and cannot be scripted in a CI environment. It can be, but there are a few tricks involved... In the mean time, it's possible to use CI environments that already feature all the necessary versions of the VC compilers and libraries such as appveyor.com, see this demo project: https://github.com/ogrisel/python-appveyor-demo https://ci.appveyor.com/project/ogrisel/python-appveyor-demo This is the best way to have it set up - create a base VM image for your CI environment manually and clone it. I believe all the major cloud providers support this, though using a CI specialist like Appveyor makes it even easier. As far as the future story, it will probably be move to 3.5 on VC14 as soon as possible. Internally, I'll be pushing for a CI-compatible installer for our build tools, which I expect will actually get quite a bit of traction right now. Unfortunately, going back in time to do it for both VC9 and VC10 was not an option. We chose VC9 because 2.7 is where people are stuck, while migrating from 3.3-3.5 should not be as big an issue. Cheers, Steve -- Olivier ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 30, 2014, at 02:34 PM, Jeremy Stanley wrote: I'm becoming less and less convinced it actually *is* a source distribution any more. My constant interaction with downstream Linux distro packagers shows a growing disinterest in consuming release tarballs of software, that they would generally prefer to pull releases directly from tags in the project's revision control systems instead. This is not a universally held consensus. We had a discussion about this at the recently concluded Debian conference. There are folks who only want to use git tags as the consumption point for Debian packages, but this opinion was not the majority opinion. There's no guarantee that what you get from a tagged upstream source revision will match what comes in the sdist tarball. Plus, the greater Debian ecosystem is firmly camped in the tarball world, so even if you do checkout from a tag, you still have to build a tarball for uploads, *and* you have to do it in a binary exact copy reproducible way. Thus, in the Debian Python team our policy is that if upstream produces tarballs (as is the case for the vast majority of our packages, which are sourced from PyPI), then we want the Debian package to use tarballs. There can be exceptions to the rule, but still today they are exceptions. I don't think the tarball format is dead yet. Cheers, -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 30, 2014, at 05:40 PM, Wichert Akkerman wrote: Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember the package version is considered to start after the last dash. Debian will also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging” might be necessary in other situations as well. Yeah, but it's maddeningly confusing. The havoc I mention is human based[*] if not tools based. ;) -Barry [*] at least to *this* human. signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 2014-09-30 12:07:23 -0400 (-0400), Barry Warsaw wrote: We had a discussion about this at the recently concluded Debian conference. There are folks who only want to use git tags as the consumption point for Debian packages, but this opinion was not the majority opinion. Good to know. The Debian Developer packaging the majority of the projects I work on must be in that minority. There's no guarantee that what you get from a tagged upstream source revision will match what comes in the sdist tarball. [...] Indeed, we've implemented quite a few workarounds specifically requested by distro packagers who want to be able to ignore our tarballs and use their own tools/workflow to generate them without ever even running sdist. It seems backwards to me, but I'm not the one doing their packaging work. Thus, in the Debian Python team our policy is that if upstream produces tarballs (as is the case for the vast majority of our packages, which are sourced from PyPI), then we want the Debian package to use tarballs. [...] Refreshing to hear! -- Jeremy Stanley ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7
On 30 September 2014 17:07, Steve Dower steve.do...@microsoft.com wrote: The answer is basically no chance - the slippery slope was considered and shut down. Fair enough. Actually, it's good to know that this sort of thing was thought through. If VC14 slips significantly and we have to stick with VC10 for Python 3.5, then I'll make the case again and see what we get, but for now the future story is to upgrade. Luckily, 3.3-3.5 is not going to be as hard as 2.7-3.5. Agreed - and given that VC14 Express will include 32- and 64-bit compilers, the whole SDK rigmarole is avoided which was the key pain point (well, that and the fact that things went out of support, but the forward compatibility of VC14 addresses that one too). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On Sep 30, 2014, at 04:25 PM, Jeremy Stanley wrote: Good to know. The Debian Developer packaging the majority of the projects I work on must be in that minority. IIRC, the OpenStack packagers were probably the most prominent proponent of release-from-tag. Cheers, -Barry signature.asc Description: PGP signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Immutable Files on PyPI
On 1 October 2014 04:40, Wichert Akkerman wich...@wiggy.net wrote: On 30 Sep 2014, at 17:35, Barry Warsaw ba...@python.org wrote: On Sep 30, 2014, at 11:06 AM, M.-A. Lemburg wrote: Installers and PyPI would then regard 3.1.4-1 as belonging to release 3.1.4, but being a more current build as a distribution file carrying 3.1.4 in its file name. Please don't literally use 3.1.4-1. That will cause all kinds of havoc with the Debian ecosystem. There we use a dash to separate upstream version numbers from Debian version numbers. Thus 3.1.4-1 means the first Debian upload of upstream's 3.1.4. 3.1.4-2 is the second, etc. Debian does allow 3.1.4-1-1. I forgot the exact rules, but I seem to remember the package version is considered to start after the last dash. Debian will also sort 3.1.4a after 3.1.4 unlike Python rules, so version “massaging” might be necessary in other situations as well. It's all in policy :) PyPI 3.1.4a should be 3.1.4~a in Debian. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Handling Case/Normalization Differences
On Sep 30, 2014, at 11:14 AM, Paul Moore p.f.mo...@gmail.com wrote: On 30 September 2014 15:25, Paul Moore p.f.mo...@gmail.com wrote: On 28 August 2014 19:58, Donald Stufft don...@stufft.io wrote: To fix this I'm going to modify PyPI so that it uses the normalized name in the /simple/ URL and redirects everything else to the non-normalized name. I'm also going to submit a PR to bandersnatch so that it will use normalized names for it's directories and such as well. These two changes will make it so that the client side will know ahead of time exactly what form the server expects any given name to be in. This will allow a change in pip to happen which will pre-normalize all names which will make the interaction with mirrors better and will reduce the number of HTTP requests that a single ``pip install`` needs to make. Just to clarify, this means that if I want to find the simple index page for a distribution, without hitting redirects, I should first normalise the project name (so Django becomes django) and then request https://pypi.python.org/simple/normalised_name/ (with a slash on the end). Is that correct? It seems to match what I see in practice (in particular, the version without a terminating slash redirects to the version with a terminating slash). The JSON API has the opposite behaviour - https://pypi.python.org/pypi/Django/json redirects to https://pypi.python.org/pypi/django/json. Should that not be changed to match? Will it be? One further thought. Where is the definition of how to normalise a name? I could probably dig through the pip sources and find it, but it would be nice if it were documented somewhere. From experiment, it seems like lowercase, and with hyphens rather than underscores, is the definition. Does PyPI allow names not allowed by http://legacy.python.org/dev/peps/pep-0426/#name and if it does, how are they normalised? In case it's not obvious, I'm writing a client for the PyPI API, and these questions are coming out of that process. Paul. PS The Python wiki has pages for the XMLRPC and JSON API. Any objections to me adding a page for the simple API? (The obvious objection being that it's documented somewhere else, and I should just put a pointer to the real documentation...) Paul PyPI follows PEP 426, I think we even include the confusables support. Generally the normalization is done with pkg_resources.safe_name(…).lower(). I don’t think there’s any reason not to document it, setuptools has it’s routine documented but that does’t have everything that the /simple/ API supports documented since it’s really documentation for what setuptools does. The URL redirect for the json endpoint was made to match what happens with /pypi/django/. Lately I’ve been thinking that maybe we should just use the normalized form in URLs always and use the author provided name for display purposes. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
David Genest wrote: Subject: Re: [Distutils] Wheels and dependent third party dlls on windows This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd file. You should therefore be able to ship the dependent dll in the package directory (which wheels support fine). Paul Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? A DLL can only be loaded once per process (python.exe, in this case) and it will be loaded based on its file name (not the full path). Whoever loads first will win every future load for the same filename. If you're loading it directly, it's fairly easy to rename a DLL to something likely to be unique to your project (or at least to put a version number in it) so that other packages won't use it. There are more complicated approaches using manifests and activation contexts (this is how different .pyd files with the same name can be correctly loaded), but ensuring a unique name is much easier. If the DLL is loaded implicitly by a .pyd, then as Paul says it should be loaded correctly if it is alongside the .pyd. Dependency Walker from www.dependencywalker.com is a great tool for checking what DLLs will be loaded by an executable or DLL. I recommend enabling profiling of your python.exe process when you try and import your packages to see where it is looking for its dependencies. Hope that helps, Steve Thanks for your response, D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On Tue, Sep 30, 2014 at 3:31 PM, David Genest david.gen...@ubisoft.com wrote: This is not true. Python loads DLLs with LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd file. You should therefore be able to ship the dependent dll in the package directory (which wheels support fine). Paul Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? Yes it will, and it is indeed problematic. There are no great solutions: - bundle it in your package - have a separate wheel and then put it in PATH - have a separate wheel but use preload tricks (e.g. using ctypes) to avoid using PATH There are better solutions if one can patch python itself, though that's obviously not practical in most cases. David Thanks for your response, D. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Wheels and dependent third party dlls on windows
On Tue, Sep 30, 2014 at 3:44 PM, Nick Coghlan ncogh...@gmail.com wrote: On 1 October 2014 00:37, Paul Moore p.f.mo...@gmail.com wrote: On 30 September 2014 15:31, David Genest david.gen...@ubisoft.com wrote: Ok, so what if the dll is shared in a given environment (multiple extensions use it)?, the shared dll should be copied to every package? Won't that cause multiple loads by the system? I honestly don't know in that case, sorry. You might get a better answer on python-list for that, if no-one here can help. Presumably the usage is all within one distribution, otherwise the question would have to be, which distribution ships the DLL? But that question ends up leading onto the sort of discussion that starts well, I wouldn't design your system the way you have, which isn't likely to be of much help to you :-( Sorry I can't offer any more help. Note that this is the external binary dependency problem that the scientific folks are currently using conda to address. It's basically the point where you cross the line from language specific packaging system to multi-language cross-platform platform. Conda is one such solution, not the solution ;) I don't know any sumo distribution which solves this problem correctly ATM, and windows makes this rather difficult to solve. David That said, pip/wheel *may* get some capabilities along these lines in the future, it just isn't a high priority at this point. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] advice re: packaging tasks
Hi, I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run python setup.py sdist. Are there better solutions for this out there so I don't have to rely on my own hack? 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? Thanks, --Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] advice re: packaging tasks
On Sep 30, 2014, at 7:44 PM, Chris Jerdonek chris.jerdo...@gmail.com wrote: Hi, I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run python setup.py sdist. Are there better solutions for this out there so I don't have to rely on my own hack? https://warehouse.python.org/project/check-manifest/ 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Not that I’m aware of. Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? I’d probably do it in the setup.py, or as an invoke task, see https://warehouse.python.org/project/invoke/. Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? This is especially the case when I start using invoke. Thanks, --Chris ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
