Re: [Distutils] draft PEP: manylinux2
On 13 February 2018 at 20:07, Chris Jerdonek wrote: > On Tue, Feb 13, 2018 at 1:59 AM, Robin Becker wrote: >> I am a bit confused about the meaning of 'backfilling'. Does it mean that a >> particular manylinux will evolve in time so an early manylinux2010 wheel >> will differ from a later one? > > I think it just means that, say, manylinux2008 could be released after > manylinux2010. So the version numbers wouldn't need to increase with > each release as it would if the numbering scheme were, say, > manylinux1, manylinux2, manylinux3, etc. Yep, exactly this (the idea originally came from the fact we're going to need a manylinux variant with a baseline year around 2014 or 2015, or potentially even later, if we want to support aarch64 and/or ppc64le). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 13/02/2018 10:07, Chris Jerdonek wrote: On Tue, Feb 13, 2018 at 1:59 AM, Robin Becker wrote: I am a bit confused about the meaning of 'backfilling'. Does it mean that a particular manylinux will evolve in time so an early manylinux2010 wheel will differ from a later one? I think it just means that, say, manylinux2008 could be released after manylinux2010. So the version numbers wouldn't need to increase with each release as it would if the numbering scheme were, say, manylinux1, manylinux2, manylinux3, etc. --Chris thanks-- Robin Becker ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Tue, Feb 13, 2018 at 1:59 AM, Robin Becker wrote: > I am a bit confused about the meaning of 'backfilling'. Does it mean that a > particular manylinux will evolve in time so an early manylinux2010 wheel > will differ from a later one? I think it just means that, say, manylinux2008 could be released after manylinux2010. So the version numbers wouldn't need to increase with each release as it would if the numbering scheme were, say, manylinux1, manylinux2, manylinux3, etc. --Chris > -- > Robin Becker > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
I am a bit confused about the meaning of 'backfilling'. Does it mean that a particular manylinux will evolve in time so an early manylinux2010 wheel will differ from a later one? -- Robin Becker ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 12 February 2018 at 07:34, Matthew Brett wrote: > On Sun, Feb 11, 2018 at 7:53 AM, Alex Walters wrote: >> "There is a tool that can make software run on a lot of different computers, >> but only if you build it for an ancient computer. The tool is a little >> complicated - you have to learn how to get it and to use it with any >> success. The people who make it are considering changing the way they name >> it. The new naming scheme is the bare minimum year the computer running the >> code can be from. > > I think the problem is that the whole discussion turns on whether we > should care about the fact that it's more complicated than the last > sentence would suggest. It isn't really - what "manylinux2010" tells you is that distros released before that year will almost certainly *not* comply with that variant of the spec, since at least some of the relevant library versions weren't available yet. While it also conveys some fuzzier signals (such as "distros released in 2012 or later will *probably* be compatible, unless they make some unconventional library choices"), I agree with Mark's last email explaining that we shouldn't view that as the primary benefit of switching to CalVer based numbering: the primary practical benefit is the fact that CalVer based numbering will let us backfill specs for arbitrary years whenever it suits us to do so (e.g. a manylinux2008 as the oldest base platform that more recent compilers are able to target) We're also not assessing the CalVer numbering scheme in a vacuum, we're assessing it relative to: * numbering in order of definition (which would make backfilling intermediate years confusing) * numbering with arbitrary gaps (which allows backfilling up to a point, but means having to explain the gaps [1]) * numbering based on RHEL/CentOS version (which makes it difficult to ever choose a different baseline distro and still doesn't solve the backfilling problem) And from that point of view, we can see that if we assume CalVer as the recommended path forward, then we wouldn't have a compelling argument for switching away from it to any of the other plausible schemes. Cheers, Nick. [1] Ah, that would bring back memories of BASIC line numbering :) -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Sun, Feb 11, 2018 at 7:53 AM, Alex Walters wrote: > Just out of curiosity, I did a little experiment. I explained this thread > to my mother. My mother is a wonderful woman, but she wouldn't know a byte > from a bite. I explained it as follows: > > "There is a tool that can make software run on a lot of different computers, > but only if you build it for an ancient computer. The tool is a little > complicated - you have to learn how to get it and to use it with any > success. The people who make it are considering changing the way they name > it. The new naming scheme is the bare minimum year the computer running the > code can be from. I think the problem is that the whole discussion turns on whether we should care about the fact that it's more complicated than the last sentence would suggest. Cheers, Matthew ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
Just out of curiosity, I did a little experiment. I explained this thread to my mother. My mother is a wonderful woman, but she wouldn't know a byte from a bite. I explained it as follows: "There is a tool that can make software run on a lot of different computers, but only if you build it for an ancient computer. The tool is a little complicated - you have to learn how to get it and to use it with any success. The people who make it are considering changing the way they name it. The new naming scheme is the bare minimum year the computer running the code can be from. The old naming scheme is just a sequence - 1, 2, 3. Would you be confused by the new naming scheme? Do you think people using it would be confused?" Her response, which I will paraphrase because my lovely mother likes four letter words, "If it's complicated to use already, then changing the name isn't any more confusing." Not exactly scientific, but based on that, I don't think CalVer will be that confusing to library developers. > -Original Message- > From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- > list=sdamon@python.org] On Behalf Of Nick Coghlan > Sent: Sunday, February 11, 2018 7:15 AM > To: Mark Williams > Cc: Geoffrey Thomas ; DistUtils mailing list > ; Mark Williams > Subject: Re: [Distutils] draft PEP: manylinux2 > > On 10 February 2018 at 16:03, Mark Williams > wrote: > > On Tue, Feb 06, 2018 at 05:55:36PM +1000, Nick Coghlan wrote: > >> By contrast, year-based CalVer maintains distro-neutrality, while also > >> giving a good sense of the maximum age of compatible target platforms. > >> (e.g. given "manylinux2010", it's a pretty safe guess that Ubuntu > >> 12.04, 14.04 and 16.04 are all expected to be compatible, while that > >> isn't as clear given "manylinux2" or "manylinux6") > > > > I'm convinced we should use CalVer. > > > > I'm still skeptical of the utility of CalVer here. Debian 6.0 > > (squeeze), for example, was released in 2011 but is incompatible with > > `manylinux2010` wheels because it uses glibc 2.11. I'm concerned that > > the sooner `manylinux2015` is defined, the more likely it is to > > describe too fuzzy an ABI era for CalVer to convey meaningful > > information to the LTS audience. > > Yeah, I'd agree with that - there's a fuzzy multi-year period from > when libraries are available to when they become ubiquitous, so given > a "manylinux2010", it would be surprising if a 2012 release like > Ubuntu 12.04 didn't support it, but for distros released in 2010 or > 2011 you'd still need to check the details. And even after that > adoption period, there are always going to be distros that make other > choices (like Alpine deciding glibc was too large). > > > What makes it worth it is the ability to skip and backfill versions. > > As you you pointed out, it would be a strange version scheme that had > > an architecture that gained wide support in 2015 become `manylinux3` > > and one that gained wide support in 2014 `manylinux4`. > > > > In particular, Geoffrey Thomas pointed out that it should be possible > > to produce nearly-`manylinux1` compliant wheels with a much newer > > toolchain: > > > > https://mail.python.org/pipermail/wheel-builders/2017-July/000283.html > > > > We may decide that an update to `manylinux1` is worthwhile, and by > > switching to CalVer, backfilling that version as `manylinux2008` would > > be straight forward. > > Indeed, that concrete pragmatic benefit provides a more compelling > rationale for switching the numbering scheme. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 10 February 2018 at 16:03, Mark Williams wrote: > On Tue, Feb 06, 2018 at 05:55:36PM +1000, Nick Coghlan wrote: >> By contrast, year-based CalVer maintains distro-neutrality, while also >> giving a good sense of the maximum age of compatible target platforms. >> (e.g. given "manylinux2010", it's a pretty safe guess that Ubuntu >> 12.04, 14.04 and 16.04 are all expected to be compatible, while that >> isn't as clear given "manylinux2" or "manylinux6") > > I'm convinced we should use CalVer. > > I'm still skeptical of the utility of CalVer here. Debian 6.0 > (squeeze), for example, was released in 2011 but is incompatible with > `manylinux2010` wheels because it uses glibc 2.11. I'm concerned that > the sooner `manylinux2015` is defined, the more likely it is to > describe too fuzzy an ABI era for CalVer to convey meaningful > information to the LTS audience. Yeah, I'd agree with that - there's a fuzzy multi-year period from when libraries are available to when they become ubiquitous, so given a "manylinux2010", it would be surprising if a 2012 release like Ubuntu 12.04 didn't support it, but for distros released in 2010 or 2011 you'd still need to check the details. And even after that adoption period, there are always going to be distros that make other choices (like Alpine deciding glibc was too large). > What makes it worth it is the ability to skip and backfill versions. > As you you pointed out, it would be a strange version scheme that had > an architecture that gained wide support in 2015 become `manylinux3` > and one that gained wide support in 2014 `manylinux4`. > > In particular, Geoffrey Thomas pointed out that it should be possible > to produce nearly-`manylinux1` compliant wheels with a much newer > toolchain: > > https://mail.python.org/pipermail/wheel-builders/2017-July/000283.html > > We may decide that an update to `manylinux1` is worthwhile, and by > switching to CalVer, backfilling that version as `manylinux2008` would > be straight forward. Indeed, that concrete pragmatic benefit provides a more compelling rationale for switching the numbering scheme. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Tue, Feb 06, 2018 at 05:55:36PM +1000, Nick Coghlan wrote: > The CalVer idea first came up in the context of skipping ahead in the > numbering sequence to go straight to a baseline that supported ppc64le > and/or aarch64. Even 2014 would likely be too old for that, since > CentOS 7 didn't support those at launch, and neither did Ubuntu 14.04. > While such a PEP hasn't actually been written yet, the kinds of > numbers we were looking at for a suitable baseline year were around > 2015 or 2016, as that's when support for them started showing up in > mainline Linux distros. > > > Given that `manylinux` PEP numbers determine their sequence number, I > > don't see how CalVer would change the situation. > > It lets us deterministically skip numbers if we decide we want to > enable access to things that older platforms just straight up don't > support (like new instruction set architectures). > > > A bigger issue is that `manylinux` isn't really one dimensional. Lots > > of things happened in 2014; for example, IBM shipped the first POWER8 > > systems and glibc 2.19 and 2.20 were released. But RHEL 7 and thus > > CentOS ship glibc 2.17. Why should `manylinux2014` support ppc64le > > but not glibc 2.19? > > Mainly because we aim for "oldest version still used in new releases > that year", but it's also why each version still needs a PEP that maps > out the actual platform ABI as specific library versions. > > > Since the definition of `manylinux` depends on the state of RHEL and > > CentOS, maybe we should change the sequence number to match the > > underlying major release of RHEL/CentOS. That would have `manylinux2` > > become `manylinux6`, and its successor `manylinux7`. If we require > > that each `manylinux` support all the platforms its RHEL/CentOS > > supports, implementers and users could simply refer to that release to > > know what they're in for. > > We discussed that too, and one key reason for not doing it is that we > only build off Red Hat's platform definitions as a matter of > convenience, and because they currently have the longest support > lifecycles. > > In the future, we could instead decide that a particular version of > Ubuntu LTS or Debian stable (or even some other LTS distro) was a more > suitable baseline for a given manylinux version, depending on how the > relative timing works out. > > For non-RHEL/CentOS users, the RHEL/CentOS version is also just as > arbitrary a sequence number as 1-based indexing. > > By contrast, year-based CalVer maintains distro-neutrality, while also > giving a good sense of the maximum age of compatible target platforms. > (e.g. given "manylinux2010", it's a pretty safe guess that Ubuntu > 12.04, 14.04 and 16.04 are all expected to be compatible, while that > isn't as clear given "manylinux2" or "manylinux6") I'm convinced we should use CalVer. I'm still skeptical of the utility of CalVer here. Debian 6.0 (squeeze), for example, was released in 2011 but is incompatible with `manylinux2010` wheels because it uses glibc 2.11. I'm concerned that the sooner `manylinux2015` is defined, the more likely it is to describe too fuzzy an ABI era for CalVer to convey meaningful information to the LTS audience. What makes it worth it is the ability to skip and backfill versions. As you you pointed out, it would be a strange version scheme that had an architecture that gained wide support in 2015 become `manylinux3` and one that gained wide support in 2014 `manylinux4`. In particular, Geoffrey Thomas pointed out that it should be possible to produce nearly-`manylinux1` compliant wheels with a much newer toolchain: https://mail.python.org/pipermail/wheel-builders/2017-July/000283.html We may decide that an update to `manylinux1` is worthwhile, and by switching to CalVer, backfilling that version as `manylinux2008` would be straight forward. -- Mark Williams m...@twistedmatrix.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 05, 2018 at 03:17:55PM -0600, Jonathan Helmus wrote: > On 02/03/2018 02:11 AM, Nathaniel Smith wrote: > > > Docker Images > > > - > > > > > > ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are > > > provided for building binary ``linux`` wheels that can reliably be > > > converted to ``manylinux2`` wheels. [8]_ These images come with a > > > full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` > > > 4.8.2) as well as the latest releases of Python and ``pip``. > > We can and should use newer compiler versions than that, and probably > > upgrade them again over the course of the image's lifespan, so let's > > just drop the version numbers from the PEP entirely. (Maybe s/6.9/6/ > > as well for the same reason.) > > > > Moving to GCC 5 and above will introduced the new libstd++ ABI. [1] The > manylinux2 standard need to define which ABI compiled libraries should be > compiled against as older version of libstdc++ will not support the new > ABI. From what I recall the devtoolset packages for CentOS can only target > the older, _GLIBCXX_USE_CXX11_ABI=0, ABI. Geoffrey Thomas helped to confirm that devtoolset-7 does the Right Thing here, as much as that's possible. Its libstdc++.so is actually the following linker script: # cat /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/libstdc++.so /* GNU ld script Use the shared library, but some functions are only in the static library, so try that secondarily. */ OUTPUT_FORMAT(elf64-x86-64) INPUT ( /usr/lib64/libstdc++.so.6 -lstdc++_nonshared ) The INPUT command instructs ld to search for GCC symbols in the system's libstdc++_s.so first; if they're present there, the resulting binary will load them from it at runtime, and the binary will match the ABI policy described in the PEP. If the binary requires newer symbols that aren't present in the system's libstdc++.so, ld will statically link them in from its interal libstdc++_nonshared.a. This result will also match the ABI policy, in that it will either depend only on the subset of symbols available in CentOS 6's default libstdc++.so or none at all. See the ld documentation for an explanation of INPUT: https://sourceware.org/binutils/docs/ld/File-Commands.html Geoffrey did point out that there's a potential issue if two C++ libraries end up with their own statically-linked implementations std::string or std::list, whose ABIs changed with GCC 5. If instances of these classes are allocated in a library with pre-GCC 5 versions and freed in another with post-GCC 5 versions something bad might happen (at least, that's my understanding). I'm unsure how serious this issue will be in practice but it's worth discussion! -- Mark Williams m...@twistedmatrix.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 05, 2018 at 08:51:03PM -0800, Mark Williams wrote: > On Sat, Feb 03, 2018 at 12:11:51AM -0800, Nathaniel Smith wrote: > > > > We can and should use newer compiler versions than that, and probably > > upgrade them again over the course of the image's lifespan, so let's > > just drop the version numbers from the PEP entirely. (Maybe s/6.9/6/ > > as well for the same reason.) > > Wouldn't upgrading compiler versions potentially imply a change in > libgcc symbol versions? If so, that would either require the PEP be > updated for each new compiler, or the removal of libgcc from the > library whitelist. > > I may be overly paranoid about this. I was overly paranoid about it :) Geoffrey Thomas helped to confirm that devtoolset-7 does the Right Thing. Its libgcc_s.so is actually the following linker script: # cat /opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/libgcc_s.so /* GNU ld script Use the shared library, but some functions are only in the static library, so try that secondarily. */ OUTPUT_FORMAT(elf64-x86-64) GROUP ( /lib64/libgcc_s.so.1 libgcc.a ) The GROUP command instructs ld to search for GCC symbols in the system's libgcc_s.so first; if they're present there, the resulting binary will load them from it at runtime, and the binary will match the ABI policy described in the PEP. If the binary requires newer symbols that aren't present in the system's libgcc_s.so, ld will statically link them in from its interal libgcc.a. This result will also match the ABI policy, in that it will either depend only on the subset of symbols available in CentOS 6's default libgcc_s.so or none at all. See the ld documentation for an explanation of GROUP: https://sourceware.org/binutils/docs/ld/File-Commands.html -- Mark Williams m...@twistedmatrix.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Wed, Feb 07, 2018 at 08:41:06PM +1000, Nick Coghlan wrote: > > Hmm, that question prompted me to notice a flaw in the current wording > of > https://www.python.org/dev/peps/pep-0571/#platform-detection-for-installers. > > The way that's currently worded suggests that > "bool(_manylinux.manylinux2010_compatible)" would be the only way to > identify whether or not a manylinux2010 wheel should be considered for > installation. > > That isn't the case: installers checking for manylinux2010 > compatibility should fall back to "have_compatible_glibc(2, 12)" if > there's no `_manylinux` module, or if that module doesn't include a > "_manylinux.manylinux2010_compatible" attribute. Agh, thank you! Fortunately that's exactly what the draft pip implementation does: https://github.com/pypa/pip/pull/5008/files#diff-542f0dc2284dcb0cb6a0382dfeeb8ed2R160 I've pushed a new branch that includes this change: https://github.com/markrwilliams/peps/commit/4476f9c77b5adb6df4dcc00829303a5613ec7d9d -- Mark Williams m...@twistedmatrix.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 7 February 2018 at 19:58, Robin Becker wrote: > On 07/02/2018 05:21, Alex Walters wrote: >> This is a really good point. Since pip is the main interface to packages >> for end users anyways, we can call it manylinux8675309 and it wouldn't >> really matter to users - the name only really matters to package >> maintainers, not users. And because of that, manylinux2010, >> manylinux2014, >> etc makes more sense. A package maintainer is expected to be more >> educated >> about these matters, and that naming scheme is more useful to them. >> "Whats >> the oldest linux system my code will run on?" is a very likely question a > > > I dispute the fact that package maintainers should be more educated about > these matters. The package maintainer usually knows about one or a few > packages (in my case reportlab etc). I know very little about the > architectures and platforms that people are using with reportlab today. Nor > do I know (or need to know) about multiple linux distributions and what > libraries they supported and in what year. Aye, there will still be guidance on packaging.python.org for folks that just want an opinionated recommendation on what binary platforms are best to target when producing wheel files. For that scenario, whether we call it "manylinux2" or "manylinux2010" shouldn't matter too much (since folks will just be copying it from the recommendation, or using a helper tool like cibuildwheel). > I do agree that the name of the available packages shouldn't really matter. > Provided there is information in the name that allows the requesting pip to > decide on the appropriate package to use (or lack thereof) that should > suffice. Is pip clever enough to decide this or will we have to rely on the > mysterious _manylinux module? Hmm, that question prompted me to notice a flaw in the current wording of https://www.python.org/dev/peps/pep-0571/#platform-detection-for-installers. The way that's currently worded suggests that "bool(_manylinux.manylinux2010_compatible)" would be the only way to identify whether or not a manylinux2010 wheel should be considered for installation. That isn't the case: installers checking for manylinux2010 compatibility should fall back to "have_compatible_glibc(2, 12)" if there's no `_manylinux` module, or if that module doesn't include a "_manylinux.manylinux2010_compatible" attribute. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 07/02/2018 05:21, Alex Walters wrote: ... This is a really good point. Since pip is the main interface to packages for end users anyways, we can call it manylinux8675309 and it wouldn't really matter to users - the name only really matters to package maintainers, not users. And because of that, manylinux2010, manylinux2014, etc makes more sense. A package maintainer is expected to be more educated about these matters, and that naming scheme is more useful to them. "Whats the oldest linux system my code will run on?" is a very likely question a I dispute the fact that package maintainers should be more educated about these matters. The package maintainer usually knows about one or a few packages (in my case reportlab etc). I know very little about the architectures and platforms that people are using with reportlab today. Nor do I know (or need to know) about multiple linux distributions and what libraries they supported and in what year. I do agree that the name of the available packages shouldn't really matter. Provided there is information in the name that allows the requesting pip to decide on the appropriate package to use (or lack thereof) that should suffice. Is pip clever enough to decide this or will we have to rely on the mysterious _manylinux module? -- Robin Becker ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 7 February 2018 at 15:21, Alex Walters wrote: > This is a really good point. Since pip is the main interface to packages > for end users anyways, we can call it manylinux8675309 and it wouldn't > really matter to users - the name only really matters to package > maintainers, not users. And because of that, manylinux2010, manylinux2014, > etc makes more sense. A package maintainer is expected to be more educated > about these matters, and that naming scheme is more useful to them. "Whats > the oldest linux system my code will run on?" is a very likely question a > maintainer would have when building binary packages, and the year-based > naming scheme is the logical answer. Exactly :) Knowing the baseline year gives publishers a clear set of "almost certainly won't work" environments: anything released prior to the baseline year (since the library versions included in the baseline either won't have existed yet, or may not have been broadly adopted). This is mostly likely to become important if we end up defining a newer platform variant for the sake of ppc64le and aarch64: targeting a compatibility baseline like "manylinux2017" would make it clear that it excludes things like Ubuntu 16.04 and RHEL/CentOS 7.0 (even if it ends up including a later RHEL/CentOS 7.x point release, or the mid-LTS opt-in platform upgrades that Canonical publishes) in a way that manylinuxN doesn't. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
-Original Message- From: Alex Walters [mailto:tritium-l...@sdamon.com] Sent: Wednesday, February 7, 2018 12:21 AM To: 'Janzert' Subject: RE: [Distutils] draft PEP: manylinux2 > -Original Message- > From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- > list=sdamon@python.org] On Behalf Of Janzert > Sent: Tuesday, February 6, 2018 3:33 PM > To: Distutils-Sig@Python.Org > Subject: Re: [Distutils] draft PEP: manylinux2 > > On 2/5/2018 16:01, Nick Coghlan wrote: > > Compare: > > > > - manylinux1 vs manylinux2 vs manylinux3 > > - manylinux2007 vs manylinux2010 vs manylinux2014 > > > > I'll leave this just as a data point (anecdote) from someone that isn't > heavily involved with linux sysadmin or python packaging. Feel free to > make of it what you like. I generally run debian stable and occasionally > ubuntu lts on servers and the latest ubuntu for my workstation. > > If I were looking to install a package and one of the binaries available > is manylinux2010 I probably completely pass over that option and don't > even attempt using it. My assumption would be anything that old probably > isn't going to work on my 2016 or newer OS. Whereas manylinux(1, 2, 3) I > would think has a good chance of working on any reasonably modern linux. > > If not for having read the discussion here I would have interpreted a > date, especially a date that's the better part of a decade in the past, > completely the wrong way. > > Having said that, I'm pretty sure that pip should in general be handling > this decision for me and doing the right thing anyway? So it probably > doesn't matter too much. > > Janzert > This is a really good point. Since pip is the main interface to packages for end users anyways, we can call it manylinux8675309 and it wouldn't really matter to users - the name only really matters to package maintainers, not users. And because of that, manylinux2010, manylinux2014, etc makes more sense. A package maintainer is expected to be more educated about these matters, and that naming scheme is more useful to them. "Whats the oldest linux system my code will run on?" is a very likely question a maintainer would have when building binary packages, and the year-based naming scheme is the logical answer. +1 to manylinux2010, -0 manylinux2 > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 2/5/2018 16:01, Nick Coghlan wrote: Compare: - manylinux1 vs manylinux2 vs manylinux3 - manylinux2007 vs manylinux2010 vs manylinux2014 I'll leave this just as a data point (anecdote) from someone that isn't heavily involved with linux sysadmin or python packaging. Feel free to make of it what you like. I generally run debian stable and occasionally ubuntu lts on servers and the latest ubuntu for my workstation. If I were looking to install a package and one of the binaries available is manylinux2010 I probably completely pass over that option and don't even attempt using it. My assumption would be anything that old probably isn't going to work on my 2016 or newer OS. Whereas manylinux(1, 2, 3) I would think has a good chance of working on any reasonably modern linux. If not for having read the discussion here I would have interpreted a date, especially a date that's the better part of a decade in the past, completely the wrong way. Having said that, I'm pretty sure that pip should in general be handling this decision for me and doing the right thing anyway? So it probably doesn't matter too much. Janzert ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 5, 2018 at 10:01 PM, Nick Coghlan wrote: > On 6 February 2018 at 00:35, Joni Orponen wrote: > > On Mon, Feb 5, 2018 at 2:51 PM, Nick Coghlan wrote: > >> > >> As an illustrative example, manylinux1 was essentially manylinux2007, > >> and it's now running into problems precisely because that baseline is > >> more than a decade old. That's not obvious if all you know is the > >> sequential number "1", but it makes intuitive sense once you realise > >> the effective baseline year is back in 2007. > > > > > > The 2007 baseline of a fairly conservative enterprise Linux distribution, > > which relatively liberally backports features in point releases over the > > lifespan. > > Red Hat only backports features that don't break ABI, so the year > still sets the ABI baseline. > I'm not convinced all the dependencies of Python and especially of eggs out there actually fall within compatibility level 2. https://access.redhat.com/articles/rhel-abi-compatibility > As discussed, the year does not ultimately mean all that much. > > It does, it drives the entire process, as we want to maintain > compatibility with a broad range of environments, and the simplest > metric for ensuring that is "How old is the baseline?". Unless the name conveys the tie to RHEL, the easier assumption to make is an Ubuntu LTS release as they brand versions with the year semantics. I'd prefer a sequential number and an associated compatibility table. It doesn't though, since once we have a few versions out there, it > conveys *no* information about which potential users and deployment > environments are being excluded by targeting a particular manylinux > version. > > Compare: > > - manylinux1 vs manylinux2 vs manylinux3 > - manylinux2007 vs manylinux2010 vs manylinux2014 > > In the first one, you have to go look at the PEPs defining each > version to get any sense whatsoever of who you might be excluding by > targeting each variant. > > In the second, it's pretty clear that you'd be excluding users of > pre-2007 distros, pre-2010 distros, and pre-2014 distros respectively. > It's a heuristic, not a precise guideline (you'll still need to > compare PEPs to distro ABIs to get the exact details), but it conveys > a lot more useful information than the plain sequential numbering > does. I'm not sharing the view of packagers being as systems oriented or aware of packaging PEPs to catch this. What's happening within my bubble of associates is "this is how you manylinux" and that'd be followed "this is how you manylinux2 now". The effort of providing the docker images and such make this so opaque convenient people do not have to care. This is a sign of a job very well done. Congratulations. > Is there a particular reason for not picking RHEL 7 as the base for > > manylinux2 at this point? > > Yes, it's too new - it would set the baseline at around 2014, which > cuts out a lot of end user environments that are still in the process > of being upgraded to newer alternatives (most notably RHEL/CentOS 6, > since they're still supported, but other LTS distros still tend to > linger past their nominal end of life dates). These users would still be catered to by manylinux1. I've provided an opinion and there is no need to seek consensus with me beyond this exchange. Ultimately I'm nitpicking on semantics, which is not very meaningful for the larger topic at hand. -- Joni Orponen ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 6 February 2018 at 17:55, Nick Coghlan wrote: > On 6 February 2018 at 16:05, Mark Williams wrote: >> If I may, a quick question about procedure: do I continue to included >> updates to the PEP in my responses here? Or do I link to my branch on >> GitHub? > > Ah, thanks for the reminder - I'll get your intial PR merged, then we > can track any further changes in the main PEPs repo as additional PRs. OK, I've allocated PEP 571 for this, and it will show up at https://www.python.org/dev/peps/pep-0571/ once the 404 page caching expires in Fastly. In the meantime, the initially merged version can be seen at https://github.com/python/peps/blob/master/pep-0571.rst Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 6 February 2018 at 16:05, Mark Williams wrote: > As a counter point: presumably a `manylinux` standard that supports > those architectures will require a PEP, in which case the author(s) > will have read the preceding `manylinx` PEPs, either to actively > borrow as much as possible or to understand arcane but necessary > details at the behest of this list. In that case the PEPs' numbers > will determine the next `manylinux`: `manylinux1` was PEP 513; if it's > accepted, this will be PEP N > 513; and so a `manylinux` that supports > ppc64le or aarch64 will be PEP M > N. This doesn't account for the > simultaneous proposal of independent PEPs for ppc64le and aarch64, but > in that case I imagine they'd be merged into a single PEP. The CalVer idea first came up in the context of skipping ahead in the numbering sequence to go straight to a baseline that supported ppc64le and/or aarch64. Even 2014 would likely be too old for that, since CentOS 7 didn't support those at launch, and neither did Ubuntu 14.04. While such a PEP hasn't actually been written yet, the kinds of numbers we were looking at for a suitable baseline year were around 2015 or 2016, as that's when support for them started showing up in mainline Linux distros. > Given that `manylinux` PEP numbers determine their sequence number, I > don't see how CalVer would change the situation. It lets us deterministically skip numbers if we decide we want to enable access to things that older platforms just straight up don't support (like new instruction set architectures). > A bigger issue is that `manylinux` isn't really one dimensional. Lots > of things happened in 2014; for example, IBM shipped the first POWER8 > systems and glibc 2.19 and 2.20 were released. But RHEL 7 and thus > CentOS ship glibc 2.17. Why should `manylinux2014` support ppc64le > but not glibc 2.19? Mainly because we aim for "oldest version still used in new releases that year", but it's also why each version still needs a PEP that maps out the actual platform ABI as specific library versions. > Since the definition of `manylinux` depends on the state of RHEL and > CentOS, maybe we should change the sequence number to match the > underlying major release of RHEL/CentOS. That would have `manylinux2` > become `manylinux6`, and its successor `manylinux7`. If we require > that each `manylinux` support all the platforms its RHEL/CentOS > supports, implementers and users could simply refer to that release to > know what they're in for. We discussed that too, and one key reason for not doing it is that we only build off Red Hat's platform definitions as a matter of convenience, and because they currently have the longest support lifecycles. In the future, we could instead decide that a particular version of Ubuntu LTS or Debian stable (or even some other LTS distro) was a more suitable baseline for a given manylinux version, depending on how the relative timing works out. For non-RHEL/CentOS users, the RHEL/CentOS version is also just as arbitrary a sequence number as 1-based indexing. By contrast, year-based CalVer maintains distro-neutrality, while also giving a good sense of the maximum age of compatible target platforms. (e.g. given "manylinux2010", it's a pretty safe guess that Ubuntu 12.04, 14.04 and 16.04 are all expected to be compatible, while that isn't as clear given "manylinux2" or "manylinux6") > I think I'm +1 for `manylinux6`, +0 for `manylinux2010`, and -1 for > `manylinux2`, which now seems to be the worst alternative. > >> Would a manylinux2 -> manylinux2010 version numbering switch >> significantly complicate implementation of the PEP? > > Certainly not! It'll take a little bit more time to adjust > `auditwheel` and `pip`, but I don't consider that to be onerous. Once > we settle on the appropriate versioning scheme I'll be happy to update > everything! > > If I may, a quick question about procedure: do I continue to included > updates to the PEP in my responses here? Or do I link to my branch on > GitHub? Ah, thanks for the reminder - I'll get your intial PR merged, then we can track any further changes in the main PEPs repo as additional PRs. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 5, 2018 at 1:17 PM, Jonathan Helmus wrote: > Moving to GCC 5 and above will introduced the new libstd++ ABI. [1] The > manylinux2 standard need to define which ABI compiled libraries should be > compiled against as older version of libstdc++ will not support the new ABI. > From what I recall the devtoolset packages for CentOS can only target the > older, _GLIBCXX_USE_CXX11_ABI=0, ABI. We're stuck on the devtoolset packages, but it doesn't really matter for manylinux purposes. None of the libraries you're allowed to assume exist expose a C++ ABI, and everything else you have to ship yourself. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 05, 2018 at 12:15:50PM +1000, Nick Coghlan wrote: > > Thanks for this! > > Something we've discussed in the past is switching manylinux over to a > variant of CalVer, where the manylinux version number inherently > conveys the era of operating system compatibility that each variant is > targeting. In the case of this PEP, that would be `manylinux2010`, > since the RHEL/CentOS 6 ABI was formally set with the release of RHEL > 6 in November 2010. > > The intended benefit of that is that it would allow folks to go ahead > and propose newer manylinux variants that allow for ppc64le and > aarch64 support as needed, without having to guess where those > definitions should come in a sequential series. That seems reasonable. I'll admit a bias towards CalVer, though :) As a counter point: presumably a `manylinux` standard that supports those architectures will require a PEP, in which case the author(s) will have read the preceding `manylinx` PEPs, either to actively borrow as much as possible or to understand arcane but necessary details at the behest of this list. In that case the PEPs' numbers will determine the next `manylinux`: `manylinux1` was PEP 513; if it's accepted, this will be PEP N > 513; and so a `manylinux` that supports ppc64le or aarch64 will be PEP M > N. This doesn't account for the simultaneous proposal of independent PEPs for ppc64le and aarch64, but in that case I imagine they'd be merged into a single PEP. Given that `manylinux` PEP numbers determine their sequence number, I don't see how CalVer would change the situation. A bigger issue is that `manylinux` isn't really one dimensional. Lots of things happened in 2014; for example, IBM shipped the first POWER8 systems and glibc 2.19 and 2.20 were released. But RHEL 7 and thus CentOS ship glibc 2.17. Why should `manylinux2014` support ppc64le but not glibc 2.19? Since the definition of `manylinux` depends on the state of RHEL and CentOS, maybe we should change the sequence number to match the underlying major release of RHEL/CentOS. That would have `manylinux2` become `manylinux6`, and its successor `manylinux7`. If we require that each `manylinux` support all the platforms its RHEL/CentOS supports, implementers and users could simply refer to that release to know what they're in for. I think I'm +1 for `manylinux6`, +0 for `manylinux2010`, and -1 for `manylinux2`, which now seems to be the worst alternative. > Would a manylinux2 -> manylinux2010 version numbering switch > significantly complicate implementation of the PEP? Certainly not! It'll take a little bit more time to adjust `auditwheel` and `pip`, but I don't consider that to be onerous. Once we settle on the appropriate versioning scheme I'll be happy to update everything! If I may, a quick question about procedure: do I continue to included updates to the PEP in my responses here? Or do I link to my branch on GitHub? -- Mark Williams m...@twistedmatrix.com ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Sat, Feb 03, 2018 at 12:11:51AM -0800, Nathaniel Smith wrote: > > Huzzah! This is an amazing bit of work, and I'm glad you got that > weird email problem sorted out :-). Me too! I'd rather deal with Linux ABI fussiness than email any day of the week. > >ABI. *[Citation for UCS ABI tags?]* > > For the citation: maybe PEP 3149? Or just > https://github.com/pypa/pip/pull/3075 Check. > > > Compilation of Compliant Wheels > > === > > > > Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` > > platform tags to ``linux`` wheels built by ``pip wheel`` or > > ``bdist_wheel`` a ``manylinux2`` Docker container. > > Missing word: "*in* a" Check. > > > Docker Images > > - > > ... [8]_ These images come with a full compiler suite installed > > (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest > > releases of Python and ``pip``. > > We can and should use newer compiler versions than that, and probably > upgrade them again over the course of the image's lifespan, so let's > just drop the version numbers from the PEP entirely. (Maybe s/6.9/6/ > as well for the same reason.) Wouldn't upgrading compiler versions potentially imply a change in libgcc symbol versions? If so, that would either require the PEP be updated for each new compiler, or the removal of libgcc from the library whitelist. I may be overly paranoid about this. Inspecting the manylinux2 image as it stands now reveals that devtoolset-2 installs an archive of libgcc into /opt/rh/devtoolset-2/root and not a shared object. Inasmuch as NumPy's extension modules depend on code in libgcc, they appear to statically link it in from this archive. If that's a reliable consequence of using devtoolset, then I think we should remove libgcc from the whitelist, which would certainly allow the PEP to remove any mention of compiler versions. Can anybody with GCC or even devtoolset expertise weigh in? In the meantime I'll attempt to build an executable with gcc 7 that depends on libgcc_s.so and run it against a gcc 4 installation. I think it's a good idea to change CentOS 6.9 to CentOS 6, though! It looks like CentOS can release Update Sets (where .9 is the current Update Set) even into the Maintenance phase of a release's lifetime: https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Product#fndef-5c004257e64c45d272f04727feda64ddb9de47b9-0 https://web.archive.org/web/20180108090257/https://wiki.centos.org/About/Product#fndef-a91b3c0c287c782f9af063daff9e64b566d648c7-1 I've made that change in the attached PEP. > > Compatibility with kernels that lack ``vsyscall`` > > ~ > > This section is maybe not *strictly* necessary in the PEP but I think > we might as well keep it; maybe someone will find it useful. When I wrote this, building manylinux2 Docker image involved patching glibc, so I felt it was appropriate to include it in the PEP. Now that I've that process to its own Docker image I'm not so sure (see https://github.com/pypa/manylinux/pull/152#issuecomment-363289829). It'd be nice to keep around for posterity's sake, especially if I got something dreadfully wrong, but I'd be OK with removing it in favor of a footnote that links to this: https://github.com/markrwilliams/manylinux/blob/8b61bcb999bd064f0f0fd0cf9d279a69ddb8a2be/docker/glibc/README.rst > > Backwards compatibility with ``manylinux1`` wheels > > == > > ... > > I'm a little confused about what this section is trying to say > (especially the words "even when explicitly set"). Should we maybe > just say something like: I wanted to make it clear that even if you do this: pip install --platform=manylinux2 ... You might still get a manylinux1 wheel if no manylinux2 wheels are available, as demonstrated by this test case: https://github.com/markrwilliams/pip/blob/5f1ad7935050ee10f6aae0248144f033eb5e949d/tests/functional/test_download.py#L353-L373 That seemed right to me given that manylinuxes specify an upper bound for library symbols; a manylinux1 wheel that depends only GLIBC@2.5 symbols, for example, satisfies manylinux2's requirement of GLIBC <= 2.12. Maybe that's not a fair inference to make, especially since manylinux1 can depend on a installation of ncurses but manylinux2 can't. > In general, systems that can use manylinux2 wheels can also use > manylinux1 wheels; pip and similar installers should prefer manylinux2 > wheels where available. That's more succinct - do you think it makes the point about --platform clear enough? > -n > > -- > Nathaniel J. Smith -- https://vorpus.org (Changes are also available on GitHub: https://github.com/markrwilliams/peps/commits/manylinux2) - PEP: Title: The manylinux2 Platform Tag Version: $Revision$ Last-Modified: $Date$ Author: Mark Williams BDFL-Delegate: Nick Coghlan Discussions-To: Distutils SIG Status: Active Type: Informational Content-T
Re: [Distutils] draft PEP: manylinux2
On 02/03/2018 02:11 AM, Nathaniel Smith wrote: Docker Images - ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are provided for building binary ``linux`` wheels that can reliably be converted to ``manylinux2`` wheels. [8]_ These images come with a full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` 4.8.2) as well as the latest releases of Python and ``pip``. We can and should use newer compiler versions than that, and probably upgrade them again over the course of the image's lifespan, so let's just drop the version numbers from the PEP entirely. (Maybe s/6.9/6/ as well for the same reason.) Moving to GCC 5 and above will introduced the new libstd++ ABI. [1] The manylinux2 standard need to define which ABI compiled libraries should be compiled against as older version of libstdc++ will not support the new ABI. From what I recall the devtoolset packages for CentOS can only target the older, _GLIBCXX_USE_CXX11_ABI=0, ABI. Cheers, - Jonathan Helmus [1] https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 6 February 2018 at 00:38, Ivan Pozdeev wrote: > On 05.02.2018 16:51, Nick Coghlan wrote: >> Or, we can just put the year directly in the version number, so that >> publishers can go "I'm happy to target manylinux2010, because I'm fine >> with users of distros that are more than 7 years old needing to >> compile from source". > > The point is, a year has negative informativity in this case. > > The very reasoning "compatible with most distributions released since year > " is flawed 'cuz it's vague and nonintuitive. > > Which is "most" > distributions? Which part of the year X? Does that mean version Y> is included or not? How do I even know all that without checking > the spec? For most distribution decisions, that level of detail is going to be irrelevant: what's relevant is being able to get a rough sense of how exclusive you're being. Choosing manylinux2010 in 2018+ means "not very exclusive", as if someone is in a sufficiently conservative deployment environment that the base platform isn't even keeping within 8 years of upstream development, the fact they can't use community provided precompiled binaries is likely to be among the least of their concerns. By contrast, choosing manylinux2014 in 2018 *would* risk excluding quite a few common deployment environments, since it's still in that 5-7 year window where a lot of conservative environments are grudgingly admitting that they really do need to update their base platform. > (Normally, a year in an entity's name means that entity's release > year.) The year we happen to define any given manylinux version is pretty close to entirely irrelevant to anything (e.g. manylinux1 was defined in 2016, but the platform interface it describes has been available since 2007). The only reason the definition year isn't *completely* irrelevant is because pip et al can only start supporting it once we define it, so folks that aren't following our "always use a recent version of the installation toolchain, regardless of target platform" guidance may end up needing to care about when a particular version was defined. > That, provided I even remember the relevant years -- since > compatibility is governed by other things, I have absolutely no reason to. The year informs the versions of libraries that we pick to include the baseline: every ABI in manylinux2[010] will have been available to distros in 2010 (when RHEL 6 was released), just as everything in manylinux1 was around in 2007 (when RHEL 5 was released). The fuzziness is then conveyed by the "many" in manylinux, since we do things like assuming the use of glibc, which isn't a universally accurate assumption (e.g. the default Alpine Linux based images published by Docker use musl libc rather than glibc, so they don't provide a manylinux compatible platform). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 6 February 2018 at 00:35, Joni Orponen wrote: > On Mon, Feb 5, 2018 at 2:51 PM, Nick Coghlan wrote: >> >> As an illustrative example, manylinux1 was essentially manylinux2007, >> and it's now running into problems precisely because that baseline is >> more than a decade old. That's not obvious if all you know is the >> sequential number "1", but it makes intuitive sense once you realise >> the effective baseline year is back in 2007. > > > The 2007 baseline of a fairly conservative enterprise Linux distribution, > which relatively liberally backports features in point releases over the > lifespan. Red Hat only backports features that don't break ABI, so the year still sets the ABI baseline. > As discussed, the year does not ultimately mean all that much. It does, it drives the entire process, as we want to maintain compatibility with a broad range of environments, and the simplest metric for ensuring that is "How old is the baseline?". They fact you're not aware of this is problematic, since it means we're not conveying that clearly. > Just going with sequential version numbers exposes and/or hides just enough > for the end user. It doesn't though, since once we have a few versions out there, it conveys *no* information about which potential users and deployment environments are being excluded by targeting a particular manylinux version. Compare: - manylinux1 vs manylinux2 vs manylinux3 - manylinux2007 vs manylinux2010 vs manylinux2014 In the first one, you have to go look at the PEPs defining each version to get any sense whatsoever of who you might be excluding by targeting each variant. In the second, it's pretty clear that you'd be excluding users of pre-2007 distros, pre-2010 distros, and pre-2014 distros respectively. It's a heuristic, not a precise guideline (you'll still need to compare PEPs to distro ABIs to get the exact details), but it conveys a lot more useful information than the plain sequential numbering does. > Is there a particular reason for not picking RHEL 7 as the base for > manylinux2 at this point? Yes, it's too new - it would set the baseline at around 2014, which cuts out a lot of end user environments that are still in the process of being upgraded to newer alternatives (most notably RHEL/CentOS 6, since they're still supported, but other LTS distros still tend to linger past their nominal end of life dates). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 5, 2018, at 2:38 PM, Ivan Pozdeev via Distutils-SIG wrote: > The point is, a year has negative informativity in this case. > > The very reasoning "compatible with most distributions released since > year " is flawed 'cuz it's vague and nonintuitive. Which is "most" > distributions? Which part of the year X? Does that mean version Y> is included or not? How do I even know all that without > checking the spec? (Normally, a year in an entity's name means that > entity's release year.) That, provided I even remember the relevant > years -- since compatibility is governed by other things, I have > absolutely no reason to. > > A year would thus add confusion and/or encourage people to use that > "easy way out" reasoning and not actually check what they're signing up > for -- with the ensuing landmines to step on. People are going to take the easy way out anyway. I've made manylinux1 wheels, and I've never gone through and checked what distributions they are meant to be compatible with - I just assume that it will probably be most ones that I care about. I think the key question is how informative the year is. If there's a 80%+ chance that a distribution released in 2011 or 2012 supports manylinux2010, then I think it's helpful to make the year more obvious, even if there are a few counterexamples. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 05.02.2018 16:51, Nick Coghlan wrote: On 5 February 2018 at 18:03, Ivan Pozdeev via Distutils-SIG wrote: On 05.02.2018 5:15, Nick Coghlan wrote: The intended benefit of that is that it would allow folks to go ahead and propose newer manylinux variants that allow for ppc64le and aarch64 support as needed, without having to guess where those definitions should come in a sequential series. IMO this will bring forth more confusion that it'll solve. Technically, the ABI is linked to kernel and library versions rather than dates. Since Linux, unlike commercial products, is not locked into a particular vendor and thus doesn't have a single product life cycle forced upon it, those vary wildly between distibutions and running installations. We pick the library API & ABI versions based on "compatible with most distributions released since year ", though (that's the "many" in "manylinux"). As an illustrative example, manylinux1 was essentially manylinux2007, and it's now running into problems precisely because that baseline is more than a decade old. That's not obvious if all you know is the sequential number "1", but it makes intuitive sense once you realise the effective baseline year is back in 2007. Similarly, neither manylinux1 nor the proposed manylinux2[010] will ever support ppc64le or aarch64, because those instruction set architectures are too new relative to the API/ABI definitions. With the sequential numbering, any of that kind of reasoning based on relative dates requires looking up the PEP that defined that version, finding which version of RHEL/CentOS we used as a baseline, and then finding when the corresponding x.0 major version of RHEL was released. Or, we can just put the year directly in the version number, so that publishers can go "I'm happy to target manylinux2010, because I'm fine with users of distros that are more than 7 years old needing to compile from source". The point is, a year has negative informativity in this case. The very reasoning "compatible with most distributions released since year " is flawed 'cuz it's vague and nonintuitive. Which is "most" distributions? Which part of the year X? Does that mean version Y> is included or not? How do I even know all that without checking the spec? (Normally, a year in an entity's name means that entity's release year.) That, provided I even remember the relevant years -- since compatibility is governed by other things, I have absolutely no reason to. A year would thus add confusion and/or encourage people to use that "easy way out" reasoning and not actually check what they're signing up for -- with the ensuing landmines to step on. -- Regards, Ivan ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Mon, Feb 5, 2018 at 2:51 PM, Nick Coghlan wrote: > As an illustrative example, manylinux1 was essentially manylinux2007, > and it's now running into problems precisely because that baseline is > more than a decade old. That's not obvious if all you know is the > sequential number "1", but it makes intuitive sense once you realise > the effective baseline year is back in 2007. > The 2007 baseline of a fairly conservative enterprise Linux distribution, which relatively liberally backports features in point releases over the lifespan. As discussed, the year does not ultimately mean all that much. Just going with sequential version numbers exposes and/or hides just enough for the end user. Is there a particular reason for not picking RHEL 7 as the base for manylinux2 at this point? -- Joni Orponen ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 5 February 2018 at 18:03, Ivan Pozdeev via Distutils-SIG wrote: > On 05.02.2018 5:15, Nick Coghlan wrote: >> The intended benefit of that is that it would allow folks to go ahead >> and propose newer manylinux variants that allow for ppc64le and >> aarch64 support as needed, without having to guess where those >> definitions should come in a sequential series. > > IMO this will bring forth more confusion that it'll solve. Technically, the > ABI is linked to kernel and library versions rather than dates. > Since Linux, unlike commercial products, is not locked into a particular > vendor and thus doesn't have a single product life cycle forced upon it, > those vary wildly between distibutions and running installations. We pick the library API & ABI versions based on "compatible with most distributions released since year ", though (that's the "many" in "manylinux"). As an illustrative example, manylinux1 was essentially manylinux2007, and it's now running into problems precisely because that baseline is more than a decade old. That's not obvious if all you know is the sequential number "1", but it makes intuitive sense once you realise the effective baseline year is back in 2007. Similarly, neither manylinux1 nor the proposed manylinux2[010] will ever support ppc64le or aarch64, because those instruction set architectures are too new relative to the API/ABI definitions. With the sequential numbering, any of that kind of reasoning based on relative dates requires looking up the PEP that defined that version, finding which version of RHEL/CentOS we used as a baseline, and then finding when the corresponding x.0 major version of RHEL was released. Or, we can just put the year directly in the version number, so that publishers can go "I'm happy to target manylinux2010, because I'm fine with users of distros that are more than 7 years old needing to compile from source". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 05.02.2018 5:15, Nick Coghlan wrote: On 1 February 2018 at 10:01, Mark Williams wrote: Hi everyone! The manylinux1 platform tag has been tremendously useful, but unfortunately it's showing its age: https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html Nathaniel identified a list of things to do for its successor, manylinux2: https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html Please find below a draft PEP for manylinux2 that attempts to address these issues. Thanks for this! Something we've discussed in the past is switching manylinux over to a variant of CalVer, where the manylinux version number inherently conveys the era of operating system compatibility that each variant is targeting. In the case of this PEP, that would be `manylinux2010`, since the RHEL/CentOS 6 ABI was formally set with the release of RHEL 6 in November 2010. The intended benefit of that is that it would allow folks to go ahead and propose newer manylinux variants that allow for ppc64le and aarch64 support as needed, without having to guess where those definitions should come in a sequential series. IMO this will bring forth more confusion that it'll solve. Technically, the ABI is linked to kernel and library versions rather than dates. Since Linux, unlike commercial products, is not locked into a particular vendor and thus doesn't have a single product life cycle forced upon it, those vary wildly between distibutions and running installations. Would a manylinux2 -> manylinux2010 version numbering switch significantly complicate implementation of the PEP? Cheers, Nick. -- Regards, Ivan ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On 1 February 2018 at 10:01, Mark Williams wrote: > Hi everyone! > > The manylinux1 platform tag has been tremendously useful, but unfortunately > it's showing its age: > > https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html > > Nathaniel identified a list of things to do for its successor, manylinux2: > > https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html > > Please find below a draft PEP for manylinux2 that attempts to address these > issues. Thanks for this! Something we've discussed in the past is switching manylinux over to a variant of CalVer, where the manylinux version number inherently conveys the era of operating system compatibility that each variant is targeting. In the case of this PEP, that would be `manylinux2010`, since the RHEL/CentOS 6 ABI was formally set with the release of RHEL 6 in November 2010. The intended benefit of that is that it would allow folks to go ahead and propose newer manylinux variants that allow for ppc64le and aarch64 support as needed, without having to guess where those definitions should come in a sequential series. Would a manylinux2 -> manylinux2010 version numbering switch significantly complicate implementation of the PEP? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] draft PEP: manylinux2
On Wed, Jan 31, 2018 at 4:01 PM, Mark Williams wrote: > Hi everyone! > > The manylinux1 platform tag has been tremendously useful, but unfortunately > it's showing its age: > > https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html > https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html > > Nathaniel identified a list of things to do for its successor, manylinux2: > > https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html > > Please find below a draft PEP for manylinux2 that attempts to address these > issues. I've also opened a PR against python/peps: > > https://github.com/python/peps/pull/565 > > Docker images for x86_64 (and soon i686) are available to test drive: > > https://hub.docker.com/r/markrwilliams/manylinux2/tags/ Huzzah! This is an amazing bit of work, and I'm glad you got that weird email problem sorted out :-). I have a few minor comments below, but overall this all looks fine and sensible to me. Also, I think we should try to move quickly on this if we can, because the manylinux1 images are currently in the process of collapsing into unmaintainability. (For example: the openssl that CentOS 5 ships with is now so old that you can no longer use it to connect to the openssl web site to download a newer version.) > 4. If a wheel is built for any version of CPython 2 or CPython >versions 3.0 up to and including 3.2, it *must* include a CPython >ABI tag indicating its Unicode ABI. A ``manylinux2`` wheel built >against Python 2, then, must include either the ``cpy27mu`` tag >indicating it was built against an interpreter with the UCS-4 ABI >or the ``cpy27m`` tag indicating an interpeter with the UCS-2 >ABI. *[Citation for UCS ABI tags?]* For the citation: maybe PEP 3149? Or just https://github.com/pypa/pip/pull/3075 > Compilation of Compliant Wheels > === > > Like ``manylinux1``, the ``auditwheel`` tool adds ```manylinux2`` > platform tags to ``linux`` wheels built by ``pip wheel`` or > ``bdist_wheel`` a ``manylinux2`` Docker container. Missing word: "*in* a" > Docker Images > - > > ``manylinux2`` Docker images based on CentOS 6.9 x86_64 and i686 are > provided for building binary ``linux`` wheels that can reliably be > converted to ``manylinux2`` wheels. [8]_ These images come with a > full compiler suite installed (``gcc``, ``g++``, and ``gfortran`` > 4.8.2) as well as the latest releases of Python and ``pip``. We can and should use newer compiler versions than that, and probably upgrade them again over the course of the image's lifespan, so let's just drop the version numbers from the PEP entirely. (Maybe s/6.9/6/ as well for the same reason.) > Compatibility with kernels that lack ``vsyscall`` > ~ This section is maybe not *strictly* necessary in the PEP but I think we might as well keep it; maybe someone will find it useful. > Backwards compatibility with ``manylinux1`` wheels > == > > As explained in PEP 513, the specified symbol versions for > ``manylinux1`` whitelisted libraries constitute an *upper bound*. The > same is true for the symbol versions defined for ``manylinux2`` in > this PEP. As a result, ``manylinux1`` wheels are considered > ``manylinux2`` wheels. A ``pip`` that recognizes the ``manylinux2`` > platform tag will thus install ``manylinux1`` wheels for > ``manylinux2`` platforms -- even when explicitly set -- when no > ``manylinux2`` wheels are available. [20]_ I'm a little confused about what this section is trying to say (especially the words "even when explicitly set"). Should we maybe just say something like: In general, systems that can use manylinux2 wheels can also use manylinux1 wheels; pip and similar installers should prefer manylinux2 wheels where available. -n -- Nathaniel J. Smith -- https://vorpus.org ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] draft PEP: manylinux2
Hi everyone!
The manylinux1 platform tag has been tremendously useful, but unfortunately
it's showing its age:
https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html
https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html
Nathaniel identified a list of things to do for its successor, manylinux2:
https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html
Please find below a draft PEP for manylinux2 that attempts to address these
issues. I've also opened a PR against python/peps:
https://github.com/python/peps/pull/565
Docker images for x86_64 and i686 are available to test drive:
https://hub.docker.com/r/markrwilliams/manylinux2/tags/
Thanks!
PEP:
Title: The manylinux2 Platform Tag
Version: $Revision$
Last-Modified: $Date$
Author: Mark Williams
BDFL-Delegate: Nick Coghlan
Discussions-To: Distutils SIG
Status: Active
Type: Informational
Content-Type: text/x-rst
Created:
Post-History:
Resolution:
Abstract
This PEP proposes the creation of a ``manylinux2`` platform tag to
succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also
proposes that PyPI and ``pip`` both be updated to support uploading,
downloading, and installing ``manylinux2`` distributions on compatible
platforms.
Rationale
=
True to its name, the ``manylinux1`` platform tag has made the
installation of binary extension modules a reality on many Linux
systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are
more accessible to Python developers now that their installation on
common architectures does not depend on fragile development
environments and build toolchains.
``manylinux1`` wheels achieve their portability by allowing the
extension modules they contain to link against only a small set of
system-level shared libraries that export versioned symbols old enough
to benefit from backwards-compatibility policies. Extension modules
in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be
built against version 2.5 or earlier; they may then be run systems
that provide more recent ``glibc`` version that still export the
required symbols at version 2.5.
PEP 513 drew its whitelisted shared libraries and their symbol
versions from CentOS 5.11, which was the oldest supported CentOS
release at the time of its writing. Unfortunately, CentOS 5.11
reached its end-of-life on March 31st, 2017 with a clear warning
against its continued use. [4]_ No further updates, such as security
patches, will be made available. This means that its packages will
remain at obsolete versions that hamper the efforts of Python software
packagers who use the ``manylinux1`` Docker image.
CentOS 6.9 is now the oldest supported CentOS release, and will
receive maintenance updates through November 30th, 2020. [5]_ We
propose that a new PEP 425-style [6]_ platform tag called
``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux``
toolchain, PyPI, and ``pip`` be updated to support it.
The ``manylinux2`` policy
=
The following criteria determine a ``linux`` wheel's eligibility for
the ``manylinux2`` tag:
1. The wheel may only contain binary executables and shared objects
compiled for one of the two architectures supported by CentOS 6.9:
x86_64 or i686. [5]_
2. The wheel's binary executables or shared objects may not link
against externally-provided libraries except those in the following
whitelist: ::
libgcc_s.so.1
libstdc++.so.6
libm.so.6
libdl.so.2
librt.so.1
libcrypt.so.1
libc.so.6
libnsl.so.1
libutil.so.1
libpthread.so.0
libresolv.so.2
libX11.so.6
libXext.so.6
libXrender.so.1
libICE.so.6
libSM.so.6
libGL.so.1
libgobject-2.0.so.0
libgthread-2.0.so.0
libglib-2.0.so.0
This list is identical to the externally-provided libraries
whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and
``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for
inclusion for the same reasons outlined in PEP 513.
On Debian-based systems, these libraries are provided by the packages:
===
Package Libraries
===
libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1,
libnsl.so.1
libgcc1 libgcc_s.so.1
libgl1libGL.so.1
libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
libice6 libICE.so.6
libsm6libSM.so.6
libstdc++6libstdc++.so.6
libx11-6 libX11.so.6
libxext6 libXext.so.6
libxrender1 libXrender.so.1
===
On RPM-based systems, they are provided by the
[Distutils] draft PEP: manylinux2
Hi everyone!
The manylinux1 platform tag has been tremendously useful, but unfortunately
it's showing its age:
https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html
https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html
Nathaniel identified a list of things to do for its successor, manylinux2:
https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html
Please find below a draft PEP for manylinux2 that attempts to address these
issues. I've also opened a PR against python/peps:
https://github.com/python/peps/pull/565
Docker images for x86_64 and i686 are available to test drive:
https://hub.docker.com/r/markrwilliams/manylinux2/tags/
Thanks!
PEP:
Title: The manylinux2 Platform Tag
Version: $Revision$
Last-Modified: $Date$
Author: Mark Williams
BDFL-Delegate: Nick Coghlan
Discussions-To: Distutils SIG
Status: Active
Type: Informational
Content-Type: text/x-rst
Created:
Post-History:
Resolution:
Abstract
This PEP proposes the creation of a ``manylinux2`` platform tag to
succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also
proposes that PyPI and ``pip`` both be updated to support uploading,
downloading, and installing ``manylinux2`` distributions on compatible
platforms.
Rationale
=
True to its name, the ``manylinux1`` platform tag has made the
installation of binary extension modules a reality on many Linux
systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are
more accessible to Python developers now that their installation on
common architectures does not depend on fragile development
environments and build toolchains.
``manylinux1`` wheels achieve their portability by allowing the
extension modules they contain to link against only a small set of
system-level shared libraries that export versioned symbols old enough
to benefit from backwards-compatibility policies. Extension modules
in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be
built against version 2.5 or earlier; they may then be run systems
that provide more recent ``glibc`` version that still export the
required symbols at version 2.5.
PEP 513 drew its whitelisted shared libraries and their symbol
versions from CentOS 5.11, which was the oldest supported CentOS
release at the time of its writing. Unfortunately, CentOS 5.11
reached its end-of-life on March 31st, 2017 with a clear warning
against its continued use. [4]_ No further updates, such as security
patches, will be made available. This means that its packages will
remain at obsolete versions that hamper the efforts of Python software
packagers who use the ``manylinux1`` Docker image.
CentOS 6.9 is now the oldest supported CentOS release, and will
receive maintenance updates through November 30th, 2020. [5]_ We
propose that a new PEP 425-style [6]_ platform tag called
``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux``
toolchain, PyPI, and ``pip`` be updated to support it.
The ``manylinux2`` policy
=
The following criteria determine a ``linux`` wheel's eligibility for
the ``manylinux2`` tag:
1. The wheel may only contain binary executables and shared objects
compiled for one of the two architectures supported by CentOS 6.9:
x86_64 or i686. [5]_
2. The wheel's binary executables or shared objects may not link
against externally-provided libraries except those in the following
whitelist: ::
libgcc_s.so.1
libstdc++.so.6
libm.so.6
libdl.so.2
librt.so.1
libcrypt.so.1
libc.so.6
libnsl.so.1
libutil.so.1
libpthread.so.0
libresolv.so.2
libX11.so.6
libXext.so.6
libXrender.so.1
libICE.so.6
libSM.so.6
libGL.so.1
libgobject-2.0.so.0
libgthread-2.0.so.0
libglib-2.0.so.0
This list is identical to the externally-provided libraries
whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and
``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for
inclusion for the same reasons outlined in PEP 513.
On Debian-based systems, these libraries are provided by the packages:
===
Package Libraries
===
libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1,
libnsl.so.1
libgcc1 libgcc_s.so.1
libgl1libGL.so.1
libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
libice6 libICE.so.6
libsm6libSM.so.6
libstdc++6libstdc++.so.6
libx11-6 libX11.so.6
libxext6 libXext.so.6
libxrender1 libXrender.so.1
===
On RPM-based systems, they are provided by the
[Distutils] draft PEP: manylinux2
Hi everyone!
The manylinux1 platform tag has been tremendously useful, but unfortunately
it's showing its age:
https://mail.python.org/pipermail/distutils-sig/2017-April/030360.html
https://mail.python.org/pipermail/wheel-builders/2016-December/000239.html
Nathaniel identified a list of things to do for its successor, manylinux2:
https://mail.python.org/pipermail/distutils-sig/2017-April/030361.html
Please find below a draft PEP for manylinux2 that attempts to address these
issues. I've also opened a PR against python/peps:
https://github.com/python/peps/pull/565
Docker images for x86_64 (and soon i686) are available to test drive:
https://hub.docker.com/r/markrwilliams/manylinux2/tags/
Thanks!
PEP:
Title: The manylinux2 Platform Tag
Version: $Revision$
Last-Modified: $Date$
Author: Mark Williams
BDFL-Delegate: Nick Coghlan
Discussions-To: Distutils SIG
Status: Active
Type: Informational
Content-Type: text/x-rst
Created:
Post-History:
Resolution:
Abstract
This PEP proposes the creation of a ``manylinux2`` platform tag to
succeed the ``manylinux1`` tag introduced by PEP 513 [1]_. It also
proposes that PyPI and ``pip`` both be updated to support uploading,
downloading, and installing ``manylinux2`` distributions on compatible
platforms.
Rationale
=
True to its name, the ``manylinux1`` platform tag has made the
installation of binary extension modules a reality on many Linux
systems. Libraries like ``cryptography`` [2]_ and ``numpy`` [3]_ are
more accessible to Python developers now that their installation on
common architectures does not depend on fragile development
environments and build toolchains.
``manylinux1`` wheels achieve their portability by allowing the
extension modules they contain to link against only a small set of
system-level shared libraries that export versioned symbols old enough
to benefit from backwards-compatibility policies. Extension modules
in a ``manylinux1`` wheel that rely on ``glibc``, for example, must be
built against version 2.5 or earlier; they may then be run systems
that provide more recent ``glibc`` version that still export the
required symbols at version 2.5.
PEP 513 drew its whitelisted shared libraries and their symbol
versions from CentOS 5.11, which was the oldest supported CentOS
release at the time of its writing. Unfortunately, CentOS 5.11
reached its end-of-life on March 31st, 2017 with a clear warning
against its continued use. [4]_ No further updates, such as security
patches, will be made available. This means that its packages will
remain at obsolete versions that hamper the efforts of Python software
packagers who use the ``manylinux1`` Docker image.
CentOS 6.9 is now the oldest supported CentOS release, and will
receive maintenance updates through November 30th, 2020. [5]_ We
propose that a new PEP 425-style [6]_ platform tag called
``manylinux2`` be derived from CentOS 6.9 and that the ``manylinux``
toolchain, PyPI, and ``pip`` be updated to support it.
The ``manylinux2`` policy
=
The following criteria determine a ``linux`` wheel's eligibility for
the ``manylinux2`` tag:
1. The wheel may only contain binary executables and shared objects
compiled for one of the two architectures supported by CentOS 6.9:
x86_64 or i686. [5]_
2. The wheel's binary executables or shared objects may not link
against externally-provided libraries except those in the following
whitelist: ::
libgcc_s.so.1
libstdc++.so.6
libm.so.6
libdl.so.2
librt.so.1
libcrypt.so.1
libc.so.6
libnsl.so.1
libutil.so.1
libpthread.so.0
libresolv.so.2
libX11.so.6
libXext.so.6
libXrender.so.1
libICE.so.6
libSM.so.6
libGL.so.1
libgobject-2.0.so.0
libgthread-2.0.so.0
libglib-2.0.so.0
This list is identical to the externally-provided libraries
whitelisted for ``manylinux1``, minus ``libncursesw.so.5`` and
``libpanelw.so.5``. [7]_ ``libpythonX.Y`` remains ineligible for
inclusion for the same reasons outlined in PEP 513.
On Debian-based systems, these libraries are provided by the packages:
===
Package Libraries
===
libc6 libdl.so.2, libresolv.so.2, librt.so.1, libc.so.6,
libpthread.so.0, libm.so.6, libutil.so.1, libcrypt.so.1,
libnsl.so.1
libgcc1 libgcc_s.so.1
libgl1libGL.so.1
libglib2.0-0 libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0
libice6 libICE.so.6
libsm6libSM.so.6
libstdc++6libstdc++.so.6
libx11-6 libX11.so.6
libxext6 libXext.so.6
libxrender1 libXrender.so.1
===
On RPM-based systems, they are provided
