Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
Heads up to anyone running pypiserver, this breaks the default redirection from pypiserver to PyPI. There's an open issue[1] to fix it. In the meantime, you can use the pypiserver option `--fallback-url https://pypi.python.org/simple/` to work around it. --nate [1] https://github.com/pypiserver/pypiserver/issues/179 On Thu, Oct 26, 2017 at 11:11 AM, Donald Stufft wrote: > Historically PyPI was only available over either HTTP or unvalidated > HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. > In Warehouse the decision was made to *not* redirect “API” URLs from HTTP > to HTTPS, but to rather return an error accessing them from HTTP. This is > because while logged in views have HSTS to ensure HTTPS in the browser (and > with humans manually entering them into the URL bar regularly they are more > error prone) APIs which are typically accessed by automated clients with an > URL configured or hardcoded typically do not respect HSTS, so if you had a > script that did ``curl http://pypi.python.org/simple/``, it would > silently get redirects to https and appear to “work”, but you wouldn’t get > any of the security properties of TLS because an attacker would just > intercept the request prior to the redirect happening. > > Today I’ve backported this changed to the current production deployment of > PyPI, which means that you can no longer access /simple/ and /packages/ > over HTTP and you will have to directly go to HTTPS. For most people this > should have no effect, because most tooling should be defaulting to HTTPS > anyways, however if you’re using a significantly old version of tooling, it > may still be defaulting to the HTTP url and will now stop functioning. > > The recommended remediation is to upgrade your tooling to versions that > support verified TLS connections and which default to the proper HTTPS URLs. > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] wheel 0.28.0 released
Hi Xavier, Thanks for the fix! I added a test to catch this in the future and uploaded 0.29.0 to PyPI. --nate On Sat, Feb 6, 2016 at 11:47 AM, Xavier Fernandez wrote: > Hello Nate, > > I think there is another regression with version 0.27: wheel files are not > zipped anymore. > > Cf https://bitbucket.org/pypa/wheel/issues/155 and > https://bitbucket.org/pypa/wheel/pull-requests/62/ for a possible fix. > > So watch the size of your wheels with latest version :) > > Regards, > Xavier > > On Fri, Feb 5, 2016 at 10:41 PM, Nate Coraor wrote: > >> Hi all, >> >> There was a bug introduced in 0.27.0 where scripts in the wheel archive >> were created with the wrong permissions. This has been fixed and released >> in 0.28.0. >> >> --nate >> >> ___ >> Distutils-SIG maillist - Distutils-SIG@python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] wheel 0.28.0 released
Hi all, There was a bug introduced in 0.27.0 where scripts in the wheel archive were created with the wrong permissions. This has been fixed and released in 0.28.0. --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On Fri, Feb 5, 2016 at 12:52 PM, Nathaniel Smith wrote: > On Feb 5, 2016 9:35 AM, "Nate Coraor" wrote: > > > > On Sat, Jan 30, 2016 at 3:37 AM, Nathaniel Smith wrote: > >> > >> On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan > wrote: > >> > On 30 January 2016 at 09:29, Nathaniel Smith wrote: > >> >> Hi all, > >> >> > >> >> I think this is ready for pronouncement now -- thanks to everyone for > >> >> all their feedback over the last few weeks! > >> >> > >> >> The only change relative to the last posting is that we rewrote the > >> >> section on "Platform detection for installers", to switch to letting > >> >> distributors explicitly control manylinux1 compatibility by means of > a > >> >> _manylinux module. > >> > > >> > In terms of the proposal itself, I think this version is excellent :) > >> > > >> > However, I realised that there's an implicit assumption we've been > >> > making that really should be spelled out explicitly: manylinux1 wheels > >> > targeting CPython 3.2 and earlier need to be compiled against a > >> > CPython built in wide Unicode mode, and in those cases, the detection > >> > of manylinux1 compatibility at the platform level should include > >> > checking for "sys.maxunicode > 0x". > >> > >> Doh, excellent catch! > >> > >> I've just pushed the obvious update to handle this directly to the > >> copy of the PEP in the manylinux repository. > >> > >> Diff: > https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc > >> > >> New text in full: > >> > https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst > >> > >> I haven't sent to the PEP editors, because they already have another > >> diff from me sitting in their inboxes and I'm not sure how to do this > >> in a way that doesn't confuse things :-) > > > > > > Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is > this necessary? The ABI tag should be set correctly on both the build and > installation systems, so is including it as part of the manylinux1 ABI (and > fixing it to wide builds only) overkill? > > Hah, I just asked the same thing :-). Clearly I should finish scrolling > through my inbox before replying... > Heh, me too. Just replied over on the other thread. > Anyway, I would like to know this as well. :-) > > Also, just to confirm, the new releases of pip and wheel enable this for > 2.x; is it also available for all 3.x? > > -n > ABI tags always worked with wheel/pip on CPython 3.2+ (it has the SOABI config var), the new change "backports" this functionality to 2.X. --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOABI for Unicode ABI on 2.x (was: wheel 0.27.0 released)
On Fri, Feb 5, 2016 at 12:46 PM, Nathaniel Smith wrote: > On Feb 5, 2016 8:47 AM, "Nate Coraor" wrote: > > > [...] > > - Add SOABI tags to platform-specific wheels built for Python 2.X (Pull > Request > > #55, Issue #63, Issue #101) > > I can't quite untangle all the documents linked from this PR, so let me > ask here :-). Does this mean that python 2.x extension wheels now can and > should declare whether they're assuming the 16- or 32-bit Unicode ABI > inside the abi field? And if so, should PEP 513 be updated to allow for > both options to be used with manylinux1? (Right not manylinux1 just > implies/requires a UCS4 build, for older pythons where this matters.) > > -n > It isn't declared, wheel determines the ABI of the interpreter upon which the wheel is being built and tags it accordingly. So yes, I think a PEP 513 update is appropriate. As to whether the manylinux1 Docker images should include UCS-2 Pythons is a separate question, though. If there's interest, I can provide statistics of how many of Galaxy's UCS-2 Linux eggs were downloaded over time. --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [final version?] PEP 513 - A Platform Tag for Portable Linux Built Distributions
On Sat, Jan 30, 2016 at 3:37 AM, Nathaniel Smith wrote: > On Fri, Jan 29, 2016 at 11:52 PM, Nick Coghlan wrote: > > On 30 January 2016 at 09:29, Nathaniel Smith wrote: > >> Hi all, > >> > >> I think this is ready for pronouncement now -- thanks to everyone for > >> all their feedback over the last few weeks! > >> > >> The only change relative to the last posting is that we rewrote the > >> section on "Platform detection for installers", to switch to letting > >> distributors explicitly control manylinux1 compatibility by means of a > >> _manylinux module. > > > > In terms of the proposal itself, I think this version is excellent :) > > > > However, I realised that there's an implicit assumption we've been > > making that really should be spelled out explicitly: manylinux1 wheels > > targeting CPython 3.2 and earlier need to be compiled against a > > CPython built in wide Unicode mode, and in those cases, the detection > > of manylinux1 compatibility at the platform level should include > > checking for "sys.maxunicode > 0x". > > Doh, excellent catch! > > I've just pushed the obvious update to handle this directly to the > copy of the PEP in the manylinux repository. > > Diff: > https://github.com/manylinux/manylinux/commit/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc > > New text in full: > > https://raw.githubusercontent.com/manylinux/manylinux/2e49cd16b89e0d6e84a5dc98ddb1a916968b73bc/pep-513.rst > > I haven't sent to the PEP editors, because they already have another > diff from me sitting in their inboxes and I'm not sure how to do this > in a way that doesn't confuse things :-) > Now that pip and wheel both support the Python 3 SOABI tags on 2.X, is this necessary? The ABI tag should be set correctly on both the build and installation systems, so is including it as part of the manylinux1 ABI (and fixing it to wide builds only) overkill? --nate > > > The main reason we need to spell this out explicitly is that while > > distros (and I believe other redistributors) build CPython-for-Linux > > in wide mode as a matter of course, a Linux checkout of CPython 2.7 > > will build in narrow mode by default. > > I can confirm that Debian and Anaconda builds of CPython 2.7 both have > sys.maxunicode == 0x10, but Enthought Canopy has sys.maxunicode == > 0x. Hmm. I guess they should fix that. > > Also the manylinux docker image currently has sys.maxunicode == > 0x, so we should definitely fix that :-). > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] wheel 0.27.0 released
Hi all, I have just released wheel 0.27.0. This version includes a few new features and fixes for long standing minor issues, many from outside contributions. >From the changelog: 0.27.0 == - Support forcing a platform tag using `--plat-name` on pure-Python wheels, as well as nonstandard platform tags on non-pure wheels (Pull Request #60, Issue #144, thanks Andrés Díaz) - Add SOABI tags to platform-specific wheels built for Python 2.X (Pull Request #55, Issue #63, Issue #101) - Support reproducible wheel files, wheels that can be rebuilt and will hash to the same values as previous builds (Pull Request #52, Issue #143, thanks Barry Warsaw) - Support for changes in keyring >= 8.0 (Pull Request #61, thanks Jason R. Coombs) - Use the file context manager when checking if dependency_links.txt is empty, fixes problems building wheels under PyPy on Windows (Issue #150, thanks Cosimo Lupo) - Don't attempt to (recursively) create a build directory ending with `..` (invalid on all platforms, but code was only executed on Windows) (Issue #91) - Added the PyPA Code of Conduct (Pull Request #56) --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Fri, Jan 29, 2016 at 9:14 PM, Nick Coghlan wrote: > On 30 January 2016 at 05:30, Nate Coraor wrote: > > I wonder if, in relation to this, it may be best to have two separate > tags: > > one to indicate that the wheel includes external libraries rolled in to > it, > > and one to indicate that it doesn't. That way, a user can make a > conscious > > decision as to whether they want to install any wheels that could include > > libraries that won't be maintained by the distribution package manager. > That > > way if we end up in a future world where manylinux wheels and > > distro-specific wheels (that may depend on non-default distro packages) > live > > in PyPI together, there'd be a way to indicate a preference. > > I don't think we want to go into that level of detail in the platform > tag, but metadata for bundled pre-built binaries in wheels and > vendored dependencies in sdists is worth considering as an enhancement > in its own right. > I thought the same thing - the only reason I proposed tags that it was my understanding that such metadata is not available to installation tool(s) until the distribution is fetched and inspected. If my limited understanding is incorrect then I agree that having this in the tags is too much. --nate > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Fri, Jan 29, 2016 at 11:44 PM, Donald Stufft wrote: > > On Jan 29, 2016, at 2:35 PM, Nate Coraor wrote: > > Is there a distro-specific wheel tagging PEP in development somewhere that > I missed? If not, I will get the ball rolling on it. > > > > I think this a great idea, and I think it actually pairs nicely with the > manylinux proposal. It should be pretty easy to cover the vast bulk of > users with a handful of platform specific wheels (1-3ish) and then a > manylinux wheel to cover the rest. It would let a project use newer > toolchains/libraries in the common case, but still fall back to the older > ones on more unusual platforms. > Fantastic, this is exactly the sort of usage I was hoping to see. I'll move forward with it, then. --nate > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Fri, Jan 22, 2016 at 10:26 PM, Nick Coghlan wrote:
> On 22 January 2016 at 22:07, M.-A. Lemburg wrote:
> > However, system vendors will often be a lot faster with updates
> > than package authors, simply because it's their business model,
> > so as user you will want to benefit from those updates and
> > not have to rely on the package author to ship new wheel files.
>
> This is true for the subset of packages monitored by distro security
> response teams, but there's a *lot* of software not currently packaged
> for Linux distros that never will be as more attention is given to the
> "rebuild the world on demand" model that elastic cloud computing and
> fast internet connections enable.
>
> My fundamental concern is that if a package author publishes a distro
> dependent wheel file, pip attempts to install it, and it doesn't work,
> the reaction for many users is going to be "Python packaging is
> broken", not "the packaging of this particular package is broken".
>
This is already broken with source dists if you don't have the appropriate
-dev packages (or a compiler) installed. Some package authors provide more
useful feedback explaining what the problem is and how one might resolve
it, rather than dying on a compiler error due to a missing header, but many
do not.
One solution to this for both source and binary distributions is package
manager awareness in the build/install tools, and to have packages declare
their dependencies in structured metadata. A translational layer would make
this easier on package authors: If they only had to say they depend on
"OpenSSL headers" and that was translated to the correct package for the
OS, this could be relayed to the user at build time ("install these
packages using this command") or the package manager could be directly
invoked, if the user has chosen to allow the build/install tool to do that.
--nate
> However, moving the "generic linux wheels are ignored by default"
> behaviour to pip-the-client, rather than enforcing it as a restriction
> on PyPI uploads could definitely be a reasonable alternative way of
> addressing that concern.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Fri, Jan 22, 2016 at 5:42 AM, Nick Coghlan wrote: > On 22 January 2016 at 19:33, M.-A. Lemburg wrote: > > For example, if a package needs a specific version of libpng, > > the package author can document this and the user can then make > > sure to install that particular version. > > The assumption that any given Python user will know how to do this is > not a reasonable assumption in 2016. > > If a publisher wants to bundle a particular version of libpng, they > can. If (as is also entirely reasonable) they don't want to assume the > associated responsibilities for responding to CVEs, then they can > stick with source distributions, or target more specific Linux > versions (as previously discussed in the context of Nate Coraor's > Starforge work) > I wonder if, in relation to this, it may be best to have two separate tags: one to indicate that the wheel includes external libraries rolled in to it, and one to indicate that it doesn't. That way, a user can make a conscious decision as to whether they want to install any wheels that could include libraries that won't be maintained by the distribution package manager. That way if we end up in a future world where manylinux wheels and distro-specific wheels (that may depend on non-default distro packages) live in PyPI together, there'd be a way to indicate a preference. --nate > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Thu, Jan 21, 2016 at 6:31 PM, Nathaniel Smith wrote: > On Thu, Jan 21, 2016 at 2:22 PM, Nate Coraor wrote: > > Could this instead use the more powerful json-based syntax proposed by > Nick > > here: > > > > > https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html > > > > I have already implemented support for this in pip and wheel. > > Totally happy to change the compatibility.cfg stuff -- the version in > the PEP was written in about 5 minutes in hopes of sparking discussion > :-). > > Some other questions: > 1) is this going to work for multi-arch (binaries for multiple cpu > architectures sharing a single /etc)? Multiple interpreters? I guess > the advantage of Nick's design is that it's scoped by the value of > distutils.util.get_platform(), so multi-arch installs could have > different values -- a distro could declare that their x86-64 python > builds are manylinux1 compatible but their i386 python builds aren't. > Maybe it would be even better if the files were > /etc/python/binary-compatibility/linux_x86_64.cfg etc., so that the > different .cfg files could be shipped in each architecture's package > without colliding. OTOH I don't know if any of this is very useful in > practice. > I don't think the proposed syntax would have any trouble with multiarch other than that it's contained in one file and so would need to live in a single package, or be dynamically generated based on which arches you installed. If that was a problem we could support a /etc/python/binary-compatibility.d type of thing. > 2) in theory one could imaging overriding this on a per-venv, > per-user, or per-system level; which of these are useful to support? > (Per-system seems like the most obvious, since an important use case > will be distros setting this flag on behalf of their users.) > Per-venv overriding was part of the original proposal and my implementation, per-user could be useful too. > There is one feature that I do think is important in the PEP 513 > draft, and that Nick's suggestion from July doesn't have: in the PEP > 513 design, the manylinux1 flag can be true, false, or unspecified, > and this is independent of other compatibility settings; in Nick's > proposal there's an exhaustive list of all compatible tags, and > everything not on that list is assumed to be incompatible. Where this > matters is when we want to release manylinux2. At this point we'll > want pip to use some autodetection logic on old distros that were > released before manylinux2, while respecting the compatibility flag on > newer distros that do know about manylinux2. This requires a tri-state > setting with "not specified" as a valid value. > One solution would be `compatible` and `incompatible` keys rather than `install`? --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
On Fri, Jan 22, 2016 at 6:29 AM, Nick Coghlan wrote: > On 22 January 2016 at 20:48, M.-A. Lemburg wrote: > > People who rely on Linux distributions want to continue > > to do so and get regular updates for system packages from > > their system vendor. Having wheel files override these > > system packages by including libs directly in the wheel > > silently breaks this expectation, potentially opening > > up the installations for security holes, difficult to > > track bugs and possible version conflicts with already > > loaded versions of the shared libs. > > For the time being, these users should either pass the "--no-binary" > option to pip, ask their distro to provide an index of pre-built wheel > files for that distro (once we have the distro-specific wheel tagging > PEP sorted out), or else ask their distro to update system Python > packages in a more timely fashion (or all of the above). > Is there a distro-specific wheel tagging PEP in development somewhere that I missed? If not, I will get the ball rolling on it. --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux1
Nathaniel, Robert, I'm really excited to see how quickly you're making progress. A few comments below as I haven't had a chance to catch up on the day's discussion: On Wed, Jan 20, 2016 at 10:55 PM, Nathaniel Smith wrote: > Building on the compability lessons learned from these companies, we thus > define a baseline ``manylinux1`` platform tag for use by binary Python > wheels, and introduce the implementation of preliminary tools to aid in the > construction of these ``manylinux1`` wheels. > Just a standards question: does this still require an update to PEP 425, or would the definition of the manylinux1 platform here supersede that section of 425? * Eventually, in the future, there may exist distributions that break > compatibility with this profile > > To handle the third case, we propose the creation of a file > ``/etc/python/compatibility.cfg`` in ConfigParser format, with sample > contents: :: > >[manylinux1] >compatible = true > > where the supported values for the ``manylinux1.compatible`` entry are the > same as those supported by the ConfigParser ``getboolean`` method. > Could this instead use the more powerful json-based syntax proposed by Nick here: https://mail.python.org/pipermail/distutils-sig/2015-July/026617.html I have already implemented support for this in pip and wheel. > Security Implications > = > > One of the advantages of dependencies on centralized libraries in Linux is > that bugfixes and security updates can be deployed system-wide, and > applications which depend on on these libraries will automatically feel the > effects of these patches when the underlying libraries are updated. This > can > be particularly important for security updates in packages communication > across the network or cryptography. > > ``manylinux1`` wheels distributed through PyPI that bundle > security-critical > libraries like OpenSSL will thus assume responsibility for prompt updates > in > response disclosed vulnerabilities and patches. This closely parallels the > security implications of the distribution of binary wheels on Windows that, > because the platform lacks a system package manager, generally bundle their > dependencies. In particular, because its lacks a stable ABI, OpenSSL > cannot be > included in the ``manylinux1`` profile. > I appreciate that this was addressed. I don't want to be responsible for keeping the versions of these things up to date. So instead, I made a docker-based build system that builds a ton of wheels on different distros/versions using a common definition. But I appreciate that it's a bit heavy and not everyone will prefer this. One piece that is not yet complete in the work I've done so far is actually ensuring that the external dependencies are installed, and providing some feedback on what's missing. But that can be done. > Rejected Alternatives > = > > One alternative would be to provide separate platform tags for each Linux > distribution (and each version thereof), e.g. ``RHEL6``, ``ubuntu14_10``, > ``debian_jessie``, etc. Nothing in this proposal rules out the possibility > of > adding such platform tags in the future, or of further extensions to wheel > metadata that would allow wheels to declare dependencies on external > system-installed packages. However, such extensions would require > substantially > more work than this proposal, and still might not be appreciated by package > developers who would prefer not to have to maintain multiple build > environments > and build multiple wheels in order to cover all the common Linux > distributions. > Therefore we consider such proposals to be out-of-scope for this PEP. > ;) For anyone who's interested, the next release of Galaxy (a popular bioinformatics framework for running tools, workflows, etc.), due next week, will ship with our modified pip that includes support for distro/version-specific platform tags in wheels. All but one of our dependent package's wheels are built with the generic `linux_x86_64` tag on Debian Squeeze and will work with most distros, though, so we're basically doing a "loose" version of manylinux1. Only our psycopg2 wheels are built per-distro/version. I'm happy to see a more rigid definition for what we're doing with the "generic" ones, this is certainly necessary should support for generic Linux wheels ever be allowed into PyPI. manylinux1 and this PEP seem to me like the right idea to do this. We build these distro/version wheels using a modified wheel and the aforementioned docker-based build system, called Starforge. Here's where all of it lives: Galaxy: https://github.com/galaxyproject/galaxy Starforge: https://github.com/galaxyproject/starforge "Galaxy pip": https://github.com/natefoo/pip/tree/linux-wheels "Galaxy wheel": https://bitbucket.org/natefoo/wheel I say all of this (again) because I think there's still a place for the work we've done even if manylinux1 is accepted. Both manylinux1 *and* more speci
Re: [Distutils] heads-up on a plot to bring linux wheels to pypi
On Thu, Jan 14, 2016 at 12:55 AM, Nathaniel Smith wrote:
> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
> https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.
>
Hi Nathaniel,
This is exciting news. As you probably know, I've been working in this
space as well. In fact, the January 2016 release of Galaxy[1] will be the
first to use wheels to distribute or framework dependencies, and they're
installed via pip. We also have a Docker-based build system that automates
the process of building these wheels[2].
Unfortunately, this necessitates our own modified versions of pip[3] and
wheel[4] for messing with the platform tags. My previous discussion here on
distutils-sig on the subject stalled on the discussion of these
environments, but it sounds like you're making progress where that stopped.
If there is momentum to manylinux and it will gain the favor of pip/PyPI
maintainers, I would be happy to transfer my effort there. I've written a
linux distro detection library that may be of some use if it's not possible
to base all environments off Centos 5. I, and the Galaxy project as a
whole, are very motivated to make official Linux wheels a reality.
> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.
>
> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)
>
FWIW, as an illumos proponent, this same work as is being done for Linux
will transfer over to other "distro" OSes like illumos.
> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
Not directly related, but Galaxy is moving toward Conda as its tool
dependency distribution system, so we are likely to converge efforts in
other places as well.
--nate
[1] https://github.com/galaxyproject/galaxy/
[2] https://github.com/galaxyproject/starforge/
[3] https://github.com/natefoo/pip/tree/linux-wheels
[4] https://bitbucket.org/natefoo/wheel
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
Hi all, I think Nathaniel raised a lot of important points below and I do see the case for a "base environment" meta tag. The implementation of sniffing out those environments on a wide array of systems may be complicated, but perhaps we can, er, borrow from conda here. I do think the glibc tag is useful as well, although it may be unnecessary if there's a way to deal with the glibc version in a base environment. However, I don't think I'm qualified to make a decision on what direction to go, and I'd like to work on updating PEP 425 for improved platform tags. So, I'm hoping to kickstart the discussion again and see if we can get a consensus on what to do. One proposal - if PEP 425 were updated to indicate that the platform tag can be more than simply `distutils.util.get_platform()`, and some language as to its intent, without specifying exactly what it must be, we could separate out the exact details into the packaging documentation as Nick has suggested. --nate On Wed, Sep 9, 2015 at 7:49 PM, Nathaniel Smith wrote: > On Wed, Sep 9, 2015 at 8:06 AM, Nate Coraor wrote: > > On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith wrote: > >> > >> On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft wrote: > >> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) > wrote: > >> >> >>> > >> >> >>> I'll create PRs for this against wheel and pip shortly. I can > also > >> >> >>> work > >> >> >>> on a PEP for the platform tag - I don't think it's going to need > to > >> >> >>> be a > >> >> >>> big one. Are there any preferences as to whether this should be a > >> >> >>> new PEP > >> >> >>> or an update to 425? > >> >> >>> > >> > > >> > Coming back to this, I'm wondering if we should include the libc > >> > implementation/version in a less generic, but still generic linux > wheel. > >> > Right > >> > now if you staticly link I think the only platform ABIs you need to > >> > worry about > >> > are libc and Python itself. Python itself is handled already but libc > is > >> > not. > >> > The only thing I've seen so far is "build on an old enough version of > >> > glibc > >> > that it handles anything sane", but not all versions of Linux even use > >> > glibc at > >> > all. > >> > >> This feels kinda half-baked to me? > >> > >> "linux" is a useful tag because it has a clear meaning: "there exists > >> a linux system somewhere that can run this, but no guarantees about > >> which one, good luck". When building a wheel it's easy to tell whether > >> this tag can be correctly applied. > > > > > > I'm not sure how it'd be possible to tell. The same meaning for a generic > > tag would be true of any wheel built, regardless of whether the wheel has > > dependencies in addition to libc. > > Sure... my point is just that "linux" is unambiguous and fills a > niche: it unambiguously says "you're on your own", and sometimes > that's the best we can hope to say. > > >> Distro-specific tags are useful because they also have a fairly clear > >> meaning: "here's a specific class of systems that can run this, so > >> long as you install enough packages to fulfill the external > >> dependencies". Again, when building a wheel it's pretty easy to tell > >> whether this tag can be correctly applied. (Of course someone could > >> screw this up, e.g. by building on a system is technically distro X > >> but has some incompatible hand-compiled libraries installed, but 99% > >> of the time we can guess correctly.) > >> > >> If we define a LSB-style base system and give it a tag, like I don't > >> know, the "Python base environment", call it "linux_pybe1_core" or > >> something, that describes what libraries are and aren't available and > >> their ABIs, and provide docs/tooling to help people explicitly create > >> such wheels and check whether they're compatible with their system, > >> then this is also useful -- we have proof that this is sufficient to > >> actually distribute arbitrary software usefully, given that multiple > >> distributors have converged on this strategy. (I've been talking to > >> some people off-list about maybe actually putting toget
Re: [Distutils] Working toward Linux wheel support
On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith wrote: > On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft wrote: > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote: > >> >>> > >> >>> I'll create PRs for this against wheel and pip shortly. I can also > work > >> >>> on a PEP for the platform tag - I don't think it's going to need to > be a > >> >>> big one. Are there any preferences as to whether this should be a > new PEP > >> >>> or an update to 425? > >> >>> > > > > Coming back to this, I'm wondering if we should include the libc > > implementation/version in a less generic, but still generic linux wheel. > Right > > now if you staticly link I think the only platform ABIs you need to > worry about > > are libc and Python itself. Python itself is handled already but libc is > not. > > The only thing I've seen so far is "build on an old enough version of > glibc > > that it handles anything sane", but not all versions of Linux even use > glibc at > > all. > > This feels kinda half-baked to me? > > "linux" is a useful tag because it has a clear meaning: "there exists > a linux system somewhere that can run this, but no guarantees about > which one, good luck". When building a wheel it's easy to tell whether > this tag can be correctly applied. > I'm not sure how it'd be possible to tell. The same meaning for a generic tag would be true of any wheel built, regardless of whether the wheel has dependencies in addition to libc. > Distro-specific tags are useful because they also have a fairly clear > meaning: "here's a specific class of systems that can run this, so > long as you install enough packages to fulfill the external > dependencies". Again, when building a wheel it's pretty easy to tell > whether this tag can be correctly applied. (Of course someone could > screw this up, e.g. by building on a system is technically distro X > but has some incompatible hand-compiled libraries installed, but 99% > of the time we can guess correctly.) > > If we define a LSB-style base system and give it a tag, like I don't > know, the "Python base environment", call it "linux_pybe1_core" or > something, that describes what libraries are and aren't available and > their ABIs, and provide docs/tooling to help people explicitly create > such wheels and check whether they're compatible with their system, > then this is also useful -- we have proof that this is sufficient to > actually distribute arbitrary software usefully, given that multiple > distributors have converged on this strategy. (I've been talking to > some people off-list about maybe actually putting together a proposal > like this...) > > To me "linux_glibc_2.18" falls between the cracks though. If this > starts being what you get by default when you build a wheel, then > people will use it for wheels that are *not* statically linked, and > what that tag will mean is "there exists some system that can run > this, and that has glibc 2.18 on it, and also some other unspecified > stuff, good luck". Which is pretty useless -- we might as well just > stick with "linux" in this case. OTOH if it's something that builders > have to opt into, then we could document that it's only to be used for > wheels that are statically linked except for glibc, and make it mean > "*any* system which has glibc 2.18 or later on it can run this". Which > would be useful in some cases. This is a tooling issue. If wheel (the package) inspects the built .so files and finds they are not dynamically linked to anything not included with glibc, it can apply the glibc tag. Otherwise, it'd apply the distro tag. There's no possibility for human error here, unless they explicitly override the platform tag. > But at this point it's basically a > version of the "defined base environment" approach, and once you've > gone that far you might as well take advantage of the various > distributors' experience about what should actually be in that > environment -- glibc isn't enough. > While I agree that glibc isn't always enough, defining a base environment that may not be met by the "standard" install of popular distributions makes unprivileged wheel installation much more difficult. It's also not going to work out of the box on older distributions that wouldn't provide whatever standardized mechanism is defined for a list of "base environments currently provided by this system" (unless pip does the work itself at runtime to determine whether a base environment is met). Maybe an important question: how many popular packages with C Extensions have dependencies in addition to glibc? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote: > On September 3, 2015 at 1:23:03 PM, Nate Coraor (n...@bx.psu.edu) wrote: > > >>> > > >>> I'll create PRs for this against wheel and pip shortly. I can also > work > > >>> on a PEP for the platform tag - I don't think it's going to need to > be a > > >>> big one. Are there any preferences as to whether this should be a > new PEP > > >>> or an update to 425? > > >>> > > Coming back to this, I'm wondering if we should include the libc > implementation/version in a less generic, but still generic linux wheel. > Right > now if you staticly link I think the only platform ABIs you need to worry > about > are libc and Python itself. Python itself is handled already but libc is > not. The only thing I've seen so far is "build on an old enough version of glibc > that it handles anything sane", but not all versions of Linux even use > glibc at > all. This proposal makes a lot of sense to me. pip will need an update to do the backwards compatibility, and it may be a little ugly to do this all on the platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be installed on systems that identify as linux_x86_64_ubuntu_14_04, but linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag prefixes or patterns should be considered backward compatible, and which should not. Granted, new libcs do not pop up overnight, so it's not exactly a nightmare scenario. Wheel should be updated to generate the "libc-generic" wheels by default when nothing other than libc is dynamically linked. It'll need libc vendor/version detection. Alternatively, the platform tag could be split in two, in which case you have a "generic" portion (which would probably be what it currently is, distutils.util.get_platform()) and a "specific" portion (the distro or libc), possibly prefixed with something to avoid having to maintain a list of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs. 'c_glibc_2_19')? I don't think there is a strong case to include the libc version in the specific portion when a distro version will also be specified, because the distro is supposed to define the ABI (at least in the case of distros with stable ABIs), and that includes the libc compatibility. So for psycopg2 wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19). It's then up to PyPI project owners to build on whatever platforms they wish to support. --nate > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Thu, Sep 3, 2015 at 10:04 AM, Nate Coraor wrote: > On Thu, Sep 3, 2015 at 9:56 AM, Daniel Holth wrote: > >> IIRC there's also a bug where we use pypy's version "2.6.2" and not the >> version of Python it implements "2.7" for the first tag. >> > > It's the other way around: > > https://github.com/pypa/pip/issues/2882 > > My changes set the Python tag to the version of PyPy. > > >> On Thu, Sep 3, 2015 at 9:53 AM Nate Coraor wrote: >> >>> On Thu, Sep 3, 2015 at 8:16 AM, Donald Stufft wrote: >>> >>>> On September 3, 2015 at 8:15:53 AM, Daniel Holth (dho...@gmail.com) >>>> wrote: >>>> > We could at least merge the implementation of the SOABI tag for >>>> Python 2.7 >>>> > (cp27m, cp27mu, ...), which has been in the PEP from the beginning >>>> but was >>>> > never implemented for Python 2. This lets you distinguish between >>>> wheels >>>> > built for CPython with debug, pymalloc, unicode builds. >>>> > >>>> > For pypy which does not have SOABI, the current 'none' should suffice. >>>> >>> >>> The ABI tag code as written will actually set it for PyPy (e.g. >>> 'pp222mu') since the SOABI config var is unset on it (and probably any >>> other non-Python-3 implementation). This was intentional since PyPy does >>> actually build some C Extensions, but I can limit SOABI detection to >>> CPython if it doesn't make sense to do it on PyPy. >>> >>> However, I see now it will also be set for Jython, which it definitely >>> should not do, so I'll fix that regardless. >>> >> >>> >>>> > >>>> >>>> Merging the SOABI tag sounds like a win to me. >>>> >>> >>> I'll create PRs for this against wheel and pip shortly. I can also work >>> on a PEP for the platform tag - I don't think it's going to need to be a >>> big one. Are there any preferences as to whether this should be a new PEP >>> or an update to 425? >>> >> Here are the PRs for SOABI support and PyPy version tag correction: https://bitbucket.org/pypa/wheel/pull-requests/55/soabi-support-for-python-2x-and-pypy/diff https://github.com/pypa/pip/pull/3075 --nate >>> >>>> >>>> - >>>> Donald Stufft >>>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >>>> DCFA >>>> >>>> >>>> > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Thu, Sep 3, 2015 at 9:56 AM, Daniel Holth wrote: > IIRC there's also a bug where we use pypy's version "2.6.2" and not the > version of Python it implements "2.7" for the first tag. > It's the other way around: https://github.com/pypa/pip/issues/2882 My changes set the Python tag to the version of PyPy. > On Thu, Sep 3, 2015 at 9:53 AM Nate Coraor wrote: > >> On Thu, Sep 3, 2015 at 8:16 AM, Donald Stufft wrote: >> >>> On September 3, 2015 at 8:15:53 AM, Daniel Holth (dho...@gmail.com) >>> wrote: >>> > We could at least merge the implementation of the SOABI tag for Python >>> 2.7 >>> > (cp27m, cp27mu, ...), which has been in the PEP from the beginning but >>> was >>> > never implemented for Python 2. This lets you distinguish between >>> wheels >>> > built for CPython with debug, pymalloc, unicode builds. >>> > >>> > For pypy which does not have SOABI, the current 'none' should suffice. >>> >> >> The ABI tag code as written will actually set it for PyPy (e.g. >> 'pp222mu') since the SOABI config var is unset on it (and probably any >> other non-Python-3 implementation). This was intentional since PyPy does >> actually build some C Extensions, but I can limit SOABI detection to >> CPython if it doesn't make sense to do it on PyPy. >> >> However, I see now it will also be set for Jython, which it definitely >> should not do, so I'll fix that regardless. >> > >> >>> > >>> >>> Merging the SOABI tag sounds like a win to me. >>> >> >> I'll create PRs for this against wheel and pip shortly. I can also work >> on a PEP for the platform tag - I don't think it's going to need to be a >> big one. Are there any preferences as to whether this should be a new PEP >> or an update to 425? >> >> >>> >>> - >>> Donald Stufft >>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >>> DCFA >>> >>> >>> ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Thu, Sep 3, 2015 at 8:16 AM, Donald Stufft wrote: > On September 3, 2015 at 8:15:53 AM, Daniel Holth (dho...@gmail.com) wrote: > > We could at least merge the implementation of the SOABI tag for Python > 2.7 > > (cp27m, cp27mu, ...), which has been in the PEP from the beginning but > was > > never implemented for Python 2. This lets you distinguish between wheels > > built for CPython with debug, pymalloc, unicode builds. > > > > For pypy which does not have SOABI, the current 'none' should suffice. > The ABI tag code as written will actually set it for PyPy (e.g. 'pp222mu') since the SOABI config var is unset on it (and probably any other non-Python-3 implementation). This was intentional since PyPy does actually build some C Extensions, but I can limit SOABI detection to CPython if it doesn't make sense to do it on PyPy. However, I see now it will also be set for Jython, which it definitely should not do, so I'll fix that regardless. > > > > Merging the SOABI tag sounds like a win to me. > I'll create PRs for this against wheel and pip shortly. I can also work on a PEP for the platform tag - I don't think it's going to need to be a big one. Are there any preferences as to whether this should be a new PEP or an update to 425? > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Tue, Aug 25, 2015 at 1:54 PM, Nate Coraor wrote:
> I've started down this road of Linux platform detection, here's the work
> so far:
>
> https://bitbucket.org/natefoo/wheel/src/tip/wheel/platform/linux.py
>
> I'm collecting distribution details here:
>
> https://gist.github.com/natefoo/814c5bf936922dad97ff
>
> One thing to note, although it's not used, I'm attempting to label a
> particular ABI as stable or unstable, so for example, Debian testing is
> unstable, whereas full releases are stable. Arch and Gentoo are always
> unstable, Ubuntu is always stable, etc. Hopefully this would be useful in
> making a decision about what wheels to allow into PyPI.
>
> --nate
>
>
Hi all,
Platform detection and binary-compatibility.cfg support is now available in
my branch of pip[1]. I've also built a large number of psycopg2 wheels for
testing[2]. Here's what happens when you try to install one of them on
CentOS 7 using my pip:
# pip install --index https://wheels.galaxyproject.org/ --no-cache-dir
psycopg2
Collecting psycopg2
Could not find a version that satisfies the requirement psycopg2 (from
versions: )
No matching distribution found for psycopg2
Then create /etc/python/binary-compatibility.cfg:
# cat /etc/python/binary-compatibility.cfg
{
"linux_x86_64_centos_7": {
"install": ["linux_x86_64_rhel_6"]
}
}
# pip install --index https://wheels.galaxyproject.org/ --no-cache-dir
psycopg2
Collecting psycopg2
Downloading
https://wheels.galaxyproject.org/packages/psycopg2-2.6.1-cp27-cp27mu-linux_x86_64_rhel_6.whl
(307kB)
100% || 307kB 75.7MB/s
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1
Of course, I have not attempted to solve the external dependency problem:
# python -c 'import psycopg2; print psycopg2'
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/psycopg2/__init__.py", line 50, in
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
ImportError: libpq.so.5: cannot open shared object file: No such file or
directory
But after installing postgresql-libs, everything works as expected:
# python -c 'import psycopg2; print psycopg2'
This is an improvement over the current situation of an sdist in PyPI,
however, since only one non-default package (postgresql-libs) needs to be
installed as opposed to postgresql-devel and the build tools (gcc, make,
etc.). In addition, a user installing psycopg2 is likely to already have
postgresql-libs installed.
I'd really appreciate if this work could be given a look, and some
discussion could take place on where to go from here.
Thanks,
--nate
[1]: https://github.com/natefoo/pip/tree/linux-wheels
[2]: https://wheels.galaxyproject.org/simple/psycopg2
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
I've started down this road of Linux platform detection, here's the work so far: https://bitbucket.org/natefoo/wheel/src/tip/wheel/platform/linux.py I'm collecting distribution details here: https://gist.github.com/natefoo/814c5bf936922dad97ff One thing to note, although it's not used, I'm attempting to label a particular ABI as stable or unstable, so for example, Debian testing is unstable, whereas full releases are stable. Arch and Gentoo are always unstable, Ubuntu is always stable, etc. Hopefully this would be useful in making a decision about what wheels to allow into PyPI. --nate On Mon, Aug 24, 2015 at 2:17 PM, Nate Coraor wrote: > On Mon, Aug 24, 2015 at 1:51 PM, Wes Turner wrote: > >> >> >> On Mon, Aug 24, 2015 at 10:03 AM, Nate Coraor wrote: >> >>> On Fri, Aug 21, 2015 at 2:51 AM, Nick Coghlan >>> wrote: >>> >>>> On 21 August 2015 at 05:58, Robert Collins >>>> wrote: >>>> > On 21 August 2015 at 07:25, Donald Stufft wrote: >>>> >> >>>> >> On August 20, 2015 at 3:23:09 PM, Daniel Holth (dho...@gmail.com) >>>> wrote: >>>> >>> If you need that for some reason just put the longer information in >>>> the >>>> >>> metadata, inside the WHEEL file for example. Surely "does it work >>>> on my >>>> >>> system" dominates, as opposed to "I have a wheel with this mnemonic >>>> tag, >>>> >>> now let me install debian 5 so I can get it to run". >>>> >>> >>>> >>> >>>> >> >>>> >> It’s less about “now let me install Debian 5” and more like tooling >>>> that doesn’t run *on* the platform but which needs to make decisions based >>>> on what platform a wheel is built for. >>>> > >>>> > Cramming that into the file name is a mistake IMO. >>>> > >>>> > Make it declarative data, make it indexable, and index it. We can do >>>> > that locally as much as via the REST API. >>>> > >>>> > That btw is why the draft for referencing external dependencies >>>> > specifies file names (because file names give an ABI in the context of >>>> > a platform) - but we do need to identify the platform, and >>>> > platform.distribution should be good enough for that (or perhaps we >>>> > start depending on lsb-release for detection >>>> >>>> LSB has too much stuff in it, so most distros aren't LSB compliant out >>>> of the box - you have to install extra packages. >>>> >>>> /etc/os-release is a better option: >>>> http://www.freedesktop.org/software/systemd/man/os-release.html >>> >>> >>> As per this discussion, and because I've discovered that the entire >>> platform module is deprecated in 3.5 (and other amusements, like a >>> Ubuntu-modified version of platform that ships on Ubuntu - platform as >>> shipped with CPython detects Ubuntu as debian), I'm switching to >>> os-release, but even that is unreliable - the file does not exist in >>> CentOS/RHEL 6, for example. On Debian testing/sid installs, VERSION and >>> VERSION_ID are unset (which is not wrong - there is no release of testing, >>> but it does make identifying the platform more complicated since even the >>> codename is not provided other than at the end of PRETTY_NAME). Regardless >>> of whether a hash or a human-identifiable string is used to identify the >>> platform, there still needs to be a way to reliably detect it. >>> >>> Unless someone tells me not to, I'm going to default to using os-release >>> and then fall back to other methods in the event that os-release isn't >>> available, and this will be in some sort of library alongside pep425tags in >>> wheel/pip. >>> >>> FWIW, os-release's `ID_LIKE` gives us some ability to make assumptions >>> without explicit need for a binary-compatibility.cfg (although not blindly >>> - for example, CentOS sets this to "rhel fedora", but of course RHEL/CentOS >>> and Fedora versions are not congruent). >>> >> >> IIUC, then the value of os-release >> will be used to generalize >> the compatible versions of *.so deps >> of a given distribution at a point in time? >> >> This works for distros that don't change [libc] much during a release, >> but for rolling release models (e.g.
Re: [Distutils] Working toward Linux wheel support
On Mon, Aug 24, 2015 at 1:51 PM, Wes Turner wrote: > > > On Mon, Aug 24, 2015 at 10:03 AM, Nate Coraor wrote: > >> On Fri, Aug 21, 2015 at 2:51 AM, Nick Coghlan wrote: >> >>> On 21 August 2015 at 05:58, Robert Collins >>> wrote: >>> > On 21 August 2015 at 07:25, Donald Stufft wrote: >>> >> >>> >> On August 20, 2015 at 3:23:09 PM, Daniel Holth (dho...@gmail.com) >>> wrote: >>> >>> If you need that for some reason just put the longer information in >>> the >>> >>> metadata, inside the WHEEL file for example. Surely "does it work on >>> my >>> >>> system" dominates, as opposed to "I have a wheel with this mnemonic >>> tag, >>> >>> now let me install debian 5 so I can get it to run". >>> >>> >>> >>> >>> >> >>> >> It’s less about “now let me install Debian 5” and more like tooling >>> that doesn’t run *on* the platform but which needs to make decisions based >>> on what platform a wheel is built for. >>> > >>> > Cramming that into the file name is a mistake IMO. >>> > >>> > Make it declarative data, make it indexable, and index it. We can do >>> > that locally as much as via the REST API. >>> > >>> > That btw is why the draft for referencing external dependencies >>> > specifies file names (because file names give an ABI in the context of >>> > a platform) - but we do need to identify the platform, and >>> > platform.distribution should be good enough for that (or perhaps we >>> > start depending on lsb-release for detection >>> >>> LSB has too much stuff in it, so most distros aren't LSB compliant out >>> of the box - you have to install extra packages. >>> >>> /etc/os-release is a better option: >>> http://www.freedesktop.org/software/systemd/man/os-release.html >> >> >> As per this discussion, and because I've discovered that the entire >> platform module is deprecated in 3.5 (and other amusements, like a >> Ubuntu-modified version of platform that ships on Ubuntu - platform as >> shipped with CPython detects Ubuntu as debian), I'm switching to >> os-release, but even that is unreliable - the file does not exist in >> CentOS/RHEL 6, for example. On Debian testing/sid installs, VERSION and >> VERSION_ID are unset (which is not wrong - there is no release of testing, >> but it does make identifying the platform more complicated since even the >> codename is not provided other than at the end of PRETTY_NAME). Regardless >> of whether a hash or a human-identifiable string is used to identify the >> platform, there still needs to be a way to reliably detect it. >> >> Unless someone tells me not to, I'm going to default to using os-release >> and then fall back to other methods in the event that os-release isn't >> available, and this will be in some sort of library alongside pep425tags in >> wheel/pip. >> >> FWIW, os-release's `ID_LIKE` gives us some ability to make assumptions >> without explicit need for a binary-compatibility.cfg (although not blindly >> - for example, CentOS sets this to "rhel fedora", but of course RHEL/CentOS >> and Fedora versions are not congruent). >> > > IIUC, then the value of os-release > will be used to generalize > the compatible versions of *.so deps > of a given distribution at a point in time? > > This works for distros that don't change [libc] much during a release, > but for rolling release models (e.g. arch, gentoo), > IDK how this simplification will work. > (This is a graph with nodes and edges (with attributes), and rules). > Arch, Gentoo, and other rolling release distributions don't have a stable ABI, so by definition I don't think we can support redistributable wheels on them. I'm adding platform detection support for them regardless, but I don't think there's any way to allow wheels built for these platforms in PyPI. > * Keying/namespacing is a simplification which may work. > * *conda preprocessing selectors* (and ~LSB-Python-Conda) > ~'prune' large parts of the graph > > * Someone mentioned LSB[-Python-Base] (again as a simplification) > * [[package, [version<=>verstr]]] > > Salt > * __salt__['grains']['os'] = "Fedora" || "Ubuntu" > * __salt__['grains']['os_family'] = "RedHat" || "Debian" > * __salt__[
Re: [Distutils] Working toward Linux wheel support
On Fri, Aug 21, 2015 at 2:51 AM, Nick Coghlan wrote: > On 21 August 2015 at 05:58, Robert Collins > wrote: > > On 21 August 2015 at 07:25, Donald Stufft wrote: > >> > >> On August 20, 2015 at 3:23:09 PM, Daniel Holth (dho...@gmail.com) > wrote: > >>> If you need that for some reason just put the longer information in the > >>> metadata, inside the WHEEL file for example. Surely "does it work on my > >>> system" dominates, as opposed to "I have a wheel with this mnemonic > tag, > >>> now let me install debian 5 so I can get it to run". > >>> > >>> > >> > >> It’s less about “now let me install Debian 5” and more like tooling > that doesn’t run *on* the platform but which needs to make decisions based > on what platform a wheel is built for. > > > > Cramming that into the file name is a mistake IMO. > > > > Make it declarative data, make it indexable, and index it. We can do > > that locally as much as via the REST API. > > > > That btw is why the draft for referencing external dependencies > > specifies file names (because file names give an ABI in the context of > > a platform) - but we do need to identify the platform, and > > platform.distribution should be good enough for that (or perhaps we > > start depending on lsb-release for detection > > LSB has too much stuff in it, so most distros aren't LSB compliant out > of the box - you have to install extra packages. > > /etc/os-release is a better option: > http://www.freedesktop.org/software/systemd/man/os-release.html As per this discussion, and because I've discovered that the entire platform module is deprecated in 3.5 (and other amusements, like a Ubuntu-modified version of platform that ships on Ubuntu - platform as shipped with CPython detects Ubuntu as debian), I'm switching to os-release, but even that is unreliable - the file does not exist in CentOS/RHEL 6, for example. On Debian testing/sid installs, VERSION and VERSION_ID are unset (which is not wrong - there is no release of testing, but it does make identifying the platform more complicated since even the codename is not provided other than at the end of PRETTY_NAME). Regardless of whether a hash or a human-identifiable string is used to identify the platform, there still needs to be a way to reliably detect it. Unless someone tells me not to, I'm going to default to using os-release and then fall back to other methods in the event that os-release isn't available, and this will be in some sort of library alongside pep425tags in wheel/pip. FWIW, os-release's `ID_LIKE` gives us some ability to make assumptions without explicit need for a binary-compatibility.cfg (although not blindly - for example, CentOS sets this to "rhel fedora", but of course RHEL/CentOS and Fedora versions are not congruent). --nate > > > My original concern with using that was that it *over*specifies the > distro (e.g. not only do CentOS and RHEL releases show up as different > platforms, but so do X.Y releases within a series), but the > binary-compatibility.txt idea resolves that issue, since a derived > distro can explicitly identify itself as binary compatible with its > upstream and be able to use the corresponding wheel files. > > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Thu, Aug 20, 2015 at 3:14 PM, Antoine Pitrou wrote:
> On Thu, 20 Aug 2015 14:26:44 -0400
> Nate Coraor wrote:
> >
> > So I need a bit of guidance here. I've arbitrarily chosen some tags -
> > `rhel` for example - and wonder if, like PEP 425's mapping of Python
> > implementations to tags, a defined mapping of Linux distributions to
> > shorthand tags is necessary (of course this would be difficult to keep up
> > to date, but binary-compatibility.cfg would make it less relevant in the
> > long run).
> >
> > Alternatively, I could simply trust and normalize
> > platform.linux_distribution()[0],
>
> In practice, the `platform` module does not really keep up to date with
> evolution in the universe of Linux distributions.
>
Understandable, although so far it's doing a pretty good job:
('Red Hat Enterprise Linux Server', '6.5', 'Santiago')
('CentOS', '6.7', 'Final')
('CentOS Linux', '7.1.1503', 'Core')
('Scientific Linux', '6.2', 'Carbon')
('debian', '6.0.10', '')
('debian', '7.8', '')
('debian', '8.1', '')
('debian', 'stretch/sid', '')
('Ubuntu', '12.04', 'precise')
('Ubuntu', '14.04', 'trusty')
('Fedora', '21', 'Twenty One')
('SUSE Linux Enterprise Server ', '11', 'x86_64')
('Gentoo Base System', '2.2', '')
platform.linux_distribution(full_distribution_name=False) might be nice but
it made some bad assumptions, e.g. on Scientific Linux it returned the
platform as 'redhat'.
--nate
>
> Regards
>
> Antoine.
>
>
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Thu, Aug 20, 2015 at 3:25 PM, Donald Stufft wrote: > > On August 20, 2015 at 3:23:09 PM, Daniel Holth (dho...@gmail.com) wrote: > > If you need that for some reason just put the longer information in the > > metadata, inside the WHEEL file for example. Surely "does it work on my > > system" dominates, as opposed to "I have a wheel with this mnemonic tag, > > now let me install debian 5 so I can get it to run". > > > > > > It’s less about “now let me install Debian 5” and more like tooling that > doesn’t run *on* the platform but which needs to make decisions based on > what platform a wheel is built for. > This makes binary-compatibility.cfg much more difficult, however. There'd still have to be a maintained list of "platform" to hash. --nate > > - > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
On Fri, Aug 14, 2015 at 3:38 AM, Nathaniel Smith wrote: > On Thu, Aug 13, 2015 at 7:27 PM, Robert Collins > wrote: > > On 14 August 2015 at 14:14, Nathaniel Smith wrote: > > ...> > >> Of course if you have an alternative proposal than I'm all ears :-). > > > > Yeah :) > > > > So, I want to dedicate some time to contributing to this discussion > > meaningfully, but I can't for the next few weeks - Jury duty, Kiwi > > PyCon and polishing up the PEP's I'm already committed to... > > Totally hear that... it's not super urgent anyway. We should make it > clear to Nate -- hi Nate! -- that there's no reason that solving this > problem should block putting together the basic > binary-compatibility.cfg infrastructure. > Hi! I've been working on bits of this as I've also been working on, as a test case, building out psycopg2 wheels for lots of different popular distros on i386 and x86_64, UCS2 and UCS4, under Docker. As a result, it's clear that my Linux distro tagging work in wheel's pep425tags has some issues. I've been adding to this list of distributions but it's going to need a lot more work: https://bitbucket.org/pypa/wheel/pull-requests/54/soabi-2x-platform-os-distro-support-for/diff#Lwheel/pep425tags.pyT61 So I need a bit of guidance here. I've arbitrarily chosen some tags - `rhel` for example - and wonder if, like PEP 425's mapping of Python implementations to tags, a defined mapping of Linux distributions to shorthand tags is necessary (of course this would be difficult to keep up to date, but binary-compatibility.cfg would make it less relevant in the long run). Alternatively, I could simply trust and normalize platform.linux_distribution()[0], but this means that the platform tag on RHEL would be something like `linux_x86_64_red_hat_enterprise_linux_server_6_5` Finally, by *default*, the built platform tag will include whatever version information is provided in platform.linux_distribution()[1], but the "major-only" version is also included in the list of platforms, so a default debian tag might look like `linux_x86_64_debian_7_8`, but it would be possible to build (and install) `linux_x86_64_debian_7`. However, it may be the case that the default (at least for building, maybe not for installing) ought to be the major-only tag since it should really be ABI compatible with any minor release of that distro. --nate > > I think the approach of being able to ask the *platform* for things > > needed to build-or-use known artifacts is going to enable a bunch of > > different answers in this space. I'm much more enthusiastic about that > > than doing anything that ends up putting PyPI in competition with the > > distribution space. > > > > My criteria for success are: > > > > - there's *a* migration path from what we have today to what we > > propose. Doesn't have to be good, just exist. > > > > - authors of scipy, numpy, cryptography etc can upload binary wheels > > for *linux, Mac OSX and Windows 32/64 in a safe and sane way > > So the problem is that, IMO, "sane" here means "not building a > separate wheel for every version of distro on distrowatch". So I can > see two ways to do that: > - my suggestion that we just pick a particular highly-compatible > distro like centos 5 to build against, and make a standard list of > which libraries can be assumed to be provided > - the PEP-497-or-number-to-be-determined approach, in which we still > have to pick a highly-compatible distro like centos 5 to build > against, but each wheel has a list of which libraries from that distro > it is counting on being provided > > I can see the appeal of the latter approach, since if you want to do > the former approach right you need to be careful about exactly which > libraries you're assuming are present, etc. They both could work. But > in practice, you still have to pick which distro you are going to use > to build, and you still have to say "when I say I need libblas.so.1, > what I mean is that I need a file that is ABI-compatible with the > version of libblas.so.1 that existed in centos 5 exactly, not any > other libblas.so.1". And then in practice not every distro will have > such a thing, so for a project like numpy that wants to make things > easy for a wide variety of users, we'll still only be able to take > advantage of external dependencies for libraries that are effectively > universally available and compatible anyway and end up vendoring the > rest... so in the end basically we'd be distributing exactly the same > wheels under either of these proposals, just the latter requires a > much much more complicated scheme for metadata and installation. > > And in practice I think the main alternative possibility if we don't > come up with some solid guidance for how packages can build > works-everywhere-wheels is that we'll see wheels for > latest-version-of-Ubuntu-only, plus the occasional smattering of other > distros, varying randomly on a project-by-project basis. Which would > suck. > > > - we don't
Re: [Distutils] Working toward Linux wheel support
On Wed, Aug 12, 2015 at 9:05 PM, Nathaniel Smith wrote: > On Aug 12, 2015 13:57, "Nate Coraor" wrote: > > > > Hello all, > > > > I've implemented the wheel side of Nick's suggestion from very early in > this thread to support a vendor-providable binary-compatibility.cfg. > > > > https://bitbucket.org/pypa/wheel/pull-request/54/ > > > > If this is acceptable, I'll add support for it to the pip side. What > else should be implemented at this stage to get the PR accepted? > > From my reading of what the Enthought and Continuum folks were saying > about how they are successfully distributing binaries across different > distributions, it sounds like the additional piece that would take this > from a interesting experiment to basically-immediately-usable would be to > teach pip that if no binary-compatibility.cfg is provided, then it should > assume by default that the compatible systems whose wheels should be > installed are: (1) the current system's exact tag, > This should already be the case - the default tag will no longer be -linux_x86_64, it'd be linux_x86_64_distro_version. > (2) the special hard-coded tag "centos5". (That's what everyone actually > uses in practice, right?) > The idea here is that we should attempt to install centos5 wheels if more specific wheels for the platform aren't available? --nate > To make this *really* slick, it would be cool if, say, David C. could make > a formal list of exactly which system libraries are important to depend on > (xlib, etc.), and we could hard-code two compatibility profiles > "centos5-minimal" (= just glibc and the C++ runtime) and "centos5" (= that > plus the core too-hard-to-ship libraries), and possibly teach pip how to > check whether that hard-coded core set is available. > > Compare with osx, where there are actually a ton of different ABIs but in > practice everyone distributing wheels basically sat down and picked one and > wrote some ad hoc tools to make it work, and it does: > https://github.com/MacPython/wiki/wiki/Spinning-wheels > > -n > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
Hello all, I've implemented the wheel side of Nick's suggestion from very early in this thread to support a vendor-providable binary-compatibility.cfg. https://bitbucket.org/pypa/wheel/pull-request/54/ If this is acceptable, I'll add support for it to the pip side. What else should be implemented at this stage to get the PR accepted? Thanks, --nate On Tue, Jul 28, 2015 at 12:21 PM, Wes Turner wrote: > > On Jul 28, 2015 10:02 AM, "Oscar Benjamin" > wrote: > > > > On Fri, 24 Jul 2015 at 19:53 Chris Barker wrote: > >> > >> On Tue, Jul 21, 2015 at 9:38 AM, Oscar Benjamin < > oscar.j.benja...@gmail.com> wrote: > >>> > >>> > >>> I think it would be great to just package these up as wheels and put > them on PyPI. > >> > >> > >> that's the point -- there is no way with the current spec to specify a > wheel dependency as opposed to a package dependency. i.e this particular > binary numpy wheel depends on this other wheel, whereas the numpy source > pacakge does not have that dependency -- and, indeed, a wheel for one > platform may have different dependencies that\n other platforms. > > > > > > I thought it was possible to do this with wheels. It's already possible > to have wheels or sdists whose dependencies vary by platform I thought. > > > > The BLAS dependency is different. In particular the sdist is compatible > with more cases than a wheel would be so the built wheel would have a more > precise requirement than the sdist. Is that not possible with > pip/wheels/PyPI or is that a limitation of using setuptools to build the > wheel? > > > >>> > >>> So numpy could depend on "blas" and there could be a few different > distributions on PyPI that provide "blas" representing the different > underlying libraries. If I want to install numpy with a particular one I > can just do: > >>> > >>> pip install gotoblas # Installs the BLAS library within Python > dirs > >>> pip install numpy > >> > >> > >> well,different implementations of BLAS are theoretically ABI > compatible, but as I understand it, it's not actually that simple, so this > is particularly challenging. > >> > >> > >> But if it were, this would be a particular trick, because then that > numpy wheel would depend on _some_ BLAS wheel, but there may be more than > one option -- how would you express that > > > > > > I imagined having numpy Require "blas OR openblas". Then openblas > package Provides "blas". Any other BLAS library also provides "blas". If > you do "pip install numpy" and "blas" is already provided then the numpy > wheel installs fine. Otherwise it falls back to installing openblas. > > > > Potentially "blas" is not specific enough so the label could be > "blas-gfortran" to express the ABI. > > BLAS may not be the best example, but should we expect such linked > interfaces to change over time? (And e.g. be versioned dependencies with > shim packages that have check functions)? > > ... How is an ABI constraint different from a package dependency? > > iiuc, ABI tags are thus combinatorial with package/wheel dependency > strings? > > Conda/pycosat solve this with "preprocessing selectors" : > http://conda.pydata.org/docs/building/meta-yaml.html#preprocessing-selectors > : > > ``` > linux True if the platform is Linux > linux32 True if the platform is Linux and the Python architecture is 32-bit > linux64 True if the platform is Linux and the Python architecture is 64-bit > armv6 True if the platform is Linux and the Python architecture is armv6l > osx True if the platform is OS X > unix True if the platform is Unix (OS X or Linux) > win True if the platform is Windows > win32 True if the platform is Windows and the Python architecture is 32-bit > win64 True if the platform is Windows and the Python architecture is 64-bit > py The Python version as a two digit string (like '27'). See also the > CONDA_PY environment variable below. > py3k True if the Python major version is 3 > py2k True if the Python major version is 2 > py26 True if the Python version is 2.6 > py27 True if the Python version is 2.7 > py33 True if the Python version is 3.3 > py34 True if the Python version is 3.4 > np The NumPy version as a two digit string (like '17'). See also the > CONDA_NPY environment variable below. > Because the selector is any valid Python expression, complicated logic is > possible. > ``` > > > > > -- > > Oscar > > > > ___ > > Distutils-SIG maillist - Distutils-SIG@python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Working toward Linux wheel support
Hi all, Thanks for the lively debate - I sent the message to start the thread and then had a week's vacation - and I appreciate the discussion that took place in the interim. I've encountered all of the problems discussed here, especially the dependencies both with Python and other attempts at package management in distributed heterogeneous systems. For Galaxy's controlled ecosystem we deal with this using static linking (e.g. our psycopg2 egg is statically linked to a version of libpq5 built for the egg), but this is not an ideal solution for a variety of reasons that I doubt I need to explain. As for the Python side of things, I certainly agree with the point raised by Leonardo that an ENOENT is probably easier for most to debug than a missing Python.h. For what it's worth, some libraries like PyYAML have a partial solution for this: If libyaml.so.X is not found at runtime, it defaults to a pure Python implementation. This is not ideal, for sure, nor will it be possible for all packages, and it depends on the package author to implement a pure Python version, but it does avoid an outright runtime failure. I hope - and I think that Nick is advocating for this - that incremental improvements can be made, rather than what's been the case so far: identifying the myriad of problems and the shortcomings of the packaging format(s), only to stall on making progress towards a solution. As to the comments regarding our needs being met today with a wheelhouse, while this is partially true (e.g. we've got our own PyPI up at https://wheels.galaxyproject.org), we still need to settle on an overspecified tag standard and fix SOABI support in Python 2.x in order to avoid having to ship a modified wheel/pip with Galaxy. Is there any specific direction the Distutils-SIG would like me to take to continue this work? Thanks, --nate On Mon, Jul 27, 2015 at 10:19 AM, Nick Coghlan wrote: > On 21 July 2015 at 04:37, Paul Moore wrote: > > On 20 July 2015 at 18:37, Chris Barker wrote: > >> sure -- but isn't that use-case already supported by wheel -- define > your > >> own wheelhouse that has the ABI you know you need, and point pip to it. > > > > I presume the issue is wanting to have a single shared wheelhouse for > > a (presumably limited) number of platforms. So being able to specify a > > (completely arbitrary) local platform name at build and install time > > sounds like a viable option. > > While supporting multiple distros in a single repo is indeed one use > case (and the one that needs to be solved to allow distribution via > PyPI), the problem I'm interested in isn't the "success case" where a > precompiled Linux wheel stays nicely confined to the specific > environment it was built to target, but rather the failure mode where > a file "escapes". > > Currently, there's nothing in a built Linux wheel file to indicate its > *intended* target environment, which makes debugging ABI mismatches > incredibly difficult. By contrast, if the wheel filename says "Fedora > 22" and you're trying to run it on "Ubuntu 14.04" and getting a > segfault, you have a pretty good hint as to the likely cause of your > problem. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Working toward Linux wheel support
Hi all, I've recently been working on adding SOABI support for Python 2.x and other pieces needed to get wheels w/ C extensions for Linux working. Here's the work for wheels: https://bitbucket.org/pypa/wheel/pull-request/54/ Based on that, I've added support for those wheels to pip here: https://github.com/natefoo/pip/tree/linux-wheels As mentioned in the wheels PR, there are some questions and decisions made that I need guidance on: - On Linux, the distro name/version (as determined by platform.linux_distribution()) will be appended to the platform string, e.g. linux_x86_64_ubuntu_14_04. This is going to be necessary to make a reasonable attempt at wheel compatibility in PyPI. But this may violate PEP 425. - By default, wheels will be built using the most specific platform information. In practice, I build our wheels[1] using Debian Squeeze in Docker and therefore they should work on most currently "supported" Linuxes, but allowing such wheels to PyPI could still be dangerous because forward compatibility is not always guaranteed (e.g. if a SO version/name changes, or a C lib API method changes in a non-backward compatible way but the SO version/name does not change). That said, I'd be happy to make a much more generalized version of our docker-build[2] system that'd allow package authors to easily/rapidly build distro/version-specific wheels for many of the popular Linux distros. We can assume that a wheel built on a vanilla install of e.g. Ubuntu 14.04 will work on any other installation of 14.04 (this is what the distro vendors promise, anyway). - I attempt to set the SOABI if the SOABI config var is unset, this is for Python 2, but will also be done even on Python 3. Maybe that is the wrong decision (or maybe SOABI is guaranteed to be set on Python 3). - Do any other implementations define SOABI? PyPy does not, I did not test others. What should we do with these? Because the project I work for[3] relies heavily on large number of packages, some of which have complicated build-time dependencies, we have always provided them as eggs and monkeypatched platform support back in to pkg_resources. Now that the PyPA has settled on wheels as the preferred binary packaging format, I am pretty heavily motivated to do the work to work out all the issues with this implementation. Thanks, --nate [1] https://wheels.galaxyproject.org/ [2] https://github.com/galaxyproject/docker-build/ [3] https://galaxyproject.org/ ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform naming standardization
David Lyon wrote: On Thu, 14 Jan 2010 10:36:46 +0100, "M.-A. Lemburg" wrote: Instead, the version string should include the details of all included builds, ie. 'x86', 'x64', 'ppc', 'ppc64'. Hashing out the details on what combinations of architectures are valid during installation will be fun though ;-). That is, if my python says its machine is "i386,x86_64" is it then acceptable to install an "i386" binary, an "i386,x86_64" binary, and "i386,ppc, x86_64" binary? The point is that even though your Python binary may say it's "i386,x86_64", the version you run your application with will either be "i386" or "x86_64" (depending on the OS environment settings). That's right. Now let's say you're running the "i386" version. As long as all installed components provide the "i386" part you should be fine. In your example all components provide the "i386" part, so all of them can be installed. The main point here though is being able to trigger the correct build for non-standard configuration, no matter what the platform. We're having the same issue on Windows as it is running the same processors. In a few years, we'll probably only see "x86_64" binaries for Mac OS, but until then, package installers will have to be able to work out the problem of finding installable distribution files among the available ones. We'll see what if anything gets changed/fixed. I'm concerned about the excessive use of code-freeze-spray being used in 2010. That's where you take 5-10 year old code and just freeze it into place. I'd really like to see distutils create packages and run in a browser window. That's my wish to santa. Having a 64-bit processor and a 10 year old command line interface on distutils, seems somewhat wrong in this millenia. There really is so much to do.. we could make a really fresh start with Python 3 and make it really different and better than Python 2.x I'd love to see a plan or a roadmap for packaging for 2010.. I'd hate to see this only end up in 3. We'll probably be supporting 2.x for a long time. Are we any closer to a plan? =) I just found another: (g)libc version. --nate David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Platform naming standardization
Tarek Ziadé wrote: It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided. Nothing has been decided yet on this. If you provide a patch for get_platform() with a rational, we can start the discussion here and work on a standard, we can then bring it up at python-dev once its ready. I was holding off on creating such a patch until I had some input on what how the platform string should be constructed. And indeed, from other messages in this thread it sounds like hacking up the platform string should not even be the way to do this. I would be happy to provide something, though, if I should go ahead and come up with something that seems reasonable to me. --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Platform naming standardization
Hi, I'm involved with a software project that makes extensive use of third party modules with C extensions. To ease installation, we pre-build them for some popular platforms and provide a distribution system for the app to fetch them. setuptools uses distutils.util.get_platform() to decide whether an egg found on the path is compatible, with some special extra magic added in for Mac OS X. Unfortunately, get_platform() as implemented does not provide enough information to ensure binary compatibility. Among the platforms we use, these are the issues: 1) On all platforms, interpreters compiled with UCS2 and UCS4 are binary incompatible, so modules compiled on one will fail to import on the other. We work around this by appending -ucsX to the platform string, based on the value of sys.maxunicode. 2) On OS X, the modification to the value returned by pkg_resources.get_platform() isn't correct for fat version of Python 2.5, as detailed in setuptools issue 19. To solve that, we're using the patch I submitted to the issue (with a couple recent changes). 3) On Solaris (and likely other UNIXes where 32 and 64 bit user spaces coexist), no distinction is made between a 32 bit and 64 bit interpreter, and they are not binary compatible. We work around this by checking sys.maxint, and right now in testing I'm appending -32 or -64 to the platform string (but before -ucsX). I haven't settled on this, though, since I have a feeling maybe it should be part of the arch (sun4u_32, i86pc_64, etc.) or something like 'solaris32' and 'solaris64' instead of 'solaris'. 4) Also on Solaris, the OS is binary compatible with older releases, so Solaris binary distributors will often build on the oldest reasonable release. This is not possible with setuptools now, although extending pkg_resources.compatible_platforms() in pretty much the same manner as is done for OS X solves this (and I can provide a patch). It's not even crucial to me that these be fixed, but before I continue to hack up the platform string, I wanted to ask the SIG to address these issues and hopefully decide on a standard. That way, I can at least implement patches in my app that will be compatible with whatever (if anything) is decided. Thanks, --nate ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
