It sounds like I should explain better how things currently work :-).
The original manylinux1 spec is PEP 513. But of course it's just text
-- it's a useful reference, but it doesn't do much by itself. And when
we wrote it we had no idea how this would actually work out.
In practice, there are two pieces to manylinux1's implementation, that
work together to make it successful.
First, there's pip's gatekeeping logic. If you put up a manylinux1
wheel on pypi, then pip will install it on any python that's built
again glibc 2.5 or greater, on x86-64 or x86-32. That means not
ancient distros like CentOS 4 (its glibc is too old), and not exotic
distros like Alpine or Android (they don't use glibc), but it includes
all vaguely-modern mainstream desktop or server distros. So in
practice the definition of a manylinux1 wheel is "I promise this wheel
will work on any system with glibc 2.5 or greater and an Intel
processor".
But most maintainers have no idea how to actually fulfill that
promise, which is where the docker image and auditwheel come in. There
are a lot of ways a wheel can fail to work on a glibc 2.5+ system: it
might depend on a newer glibc, or it might depend on library that the
target system doesn't have installed, or a whole bunch of other super
arcane traps that we've discovered over time (e.g. the Python used for
the build has to be linked using the correct configure options). These
are all encoded into the docker image/auditwheel. (So for example,
auditwheel has some built-in knowledge of which libraries you can
expect to find on every Intel system with glibc 2.5 or greater, that
it uses to make decisions about which libraries need to be vendored.)
Technically you don't *have* to use these tools to build your wheel,
pip doesn't care, but they provide some nice padded guardrails that
make it possible for ordinary maintainers to fulfill the manylinux1
promise in practice.
How does this affect spec-writing? Well, we want to allow for non-pip
installers, so the part that pip does has to be specified. But pip's
part is really straightforward.
All the complicated bit is in the docker image/auditwheel. But, for
these, it turns out the spec doesn't actually matter that much. We can
observe that most wheels do work in practice, and whenever someone
discovers some new edge case that the PEP never thought of, then it's
not a disaster, it just means there's one broken wheel on pypi, and we
figure out how to fix the tools to catch the new edge case, they
upload a new wheel, and life goes on.
So the proposal here is to refactor the spec to match how this
actually works: the official definition of a manylinux_${glibc
version}_${arch} wheel would be "I promise this wheel will work on any
Linux system with glibc >=${glibc version} and an ${arch} processor".
We'll still need to make changes as old distros go out of support, new
architectures get supported, etc., but the difference is, those
changes won't require complex cross-ecosystem coordination with new
formal specs for each one; instead they'll be routine engineering
problems for the docker image+auditwheel maintainers to solve.
-n
On Fri, Nov 30, 2018 at 12:09 AM Nathaniel Smith <n...@pobox.com> wrote:
>
> Hi all,
>
> The manylinux1 -> manylinux2010 transition has turned out to be very
> difficult. Timeline so far:
>
> March 2017: CentOS 5 went EOL
> April 2018: PEP 517 accepted
> May 2018: support for manylinux2010 lands in warehouse
> November 2018: support lands in auditwheel, and pip master
> December 2018: 21 months after CentOS 5 EOL, wwee still don't have an
> official build environment, or support in a pip release
>
> We'll get through this, but it's been super painful and maybe we can change
> things somehow so it will suck less next time.
>
> We don't have anything like this pain on Windows or macOS. We never have to
> update pip, warehouse, etc., after those OSes hit EOLs. Why not?
>
> On Windows, we have just two tags: "win32" and "win_amd64". These are defined
> to mean something like "this wheel will run on any recent-ish Windows
> system". So the meaning of the tag actually changes over time: it used to be
> that if a wheel said it ran on win32, then that meant it would work on winxp,
> but since winxp hit EOL people started uploading "win32" wheels that don't
> work on winxp, and that's worked fine.
>
> On macOS, the tags look like "macosx_10_9_x86_64". So here we have the OS
> version embedded in the tag. This means that we do occasionally switch which
> tags we're using, kind of like how manylinux1 -> manylinux2010 is intended to
> work. But, unlike for the manylinux tags, defining a new macosx tag is
> totally trivial: every time a new OS version is released, the tag springs
> into existence without any human intervention. Warehouse already accepts
> uploads with this tag; pip already knows which systems can install wheels
> with this tag, etc.
>
> Can we take any inspiration from this for manylinux?
>
> We could do the Windows thing, and have a plain "manylinux" tag that means
> "any recent-ish glibc-based Linux". Today it would be defined to be "any
> distro newer than CentOS 6". When CentOS 6 goes out of service, we could
> tweak the definition to be "any distro newer than CentOS 7". Most parts of
> the toolchain wouldn't need to be updated, though, because the tag wouldn't
> change, and by assumption, enforcement wouldn't really be needed, because the
> only people who could break would be ones running on unsupported platforms.
> Just like happens on Windows.
>
> We could do the macOS thing, and have a "manylinux_${glibc version}" tag that
> means "this package works on any Linux using glibc newer than ${glibc
> version}". We're already using this as our heuristic to handle the current
> manylinux profiles, so e.g. manylinux1 is effectively equivalent to
> manylinux_2_5, and manylinux2010 will be equivalent to manylinux_2_12. That
> way we'd define the manylinux tags once, get support into pip and warehouse
> and auditwheel once, and then in the future the only thing that would have to
> change to support new distro releases or new architectures would be to set up
> a proper build environment.
>
> What do y'all think?
>
> -n
--
Nathaniel J. Smith -- https://vorpus.org
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/TKJA5VHQLNYOF5RD6EZVI4TBPRJWQRQU/
