On Tue, Jul 7, 2015 at 7:02 PM, Donald Stufft <don...@stufft.io> wrote:
> On July 7, 2015 at 10:22:55 AM, Nick Coghlan (ncogh...@gmail.com) wrote: > > On 8 July 2015 at 00:07, Antoine Pitrou wrote: > > > On Tue, 7 Jul 2015 23:53:59 +1000 > > > Nick Coghlan wrote: > > >> Unfortunately, the compatibility tagging for Linux wheels is currently > > >> so thoroughly inadequate that even in tightly controlled environments > > >> having a wheel file escape from its "intended" target platforms can > > >> cause hard to debug problems. > > > > > > I'm not sure what you're pointing to, could you elaborate a bit? > > > > That was a reference to a case of someone building for Debian (I > > think), and then having one of their wheel files end up installed on a > > CentOS system and wondering why things weren't working. > > > > > For the record, building against a well-known, old glibc + gcc has > > > served the Anaconda platform well. > > > > The key problem is that there's no straightforward way for us to > > verify that folks are actually building against a suitably limited set > > of platform APIs that all Linux distros in widespread use provide. > > > > And when it inevitably fails (which it will, Python and PyPI are too > > popular for it not to), the folks running into the problem are > > unlikely to be able to diagnose what has happened, and even once we > > figure out what has gone wrong, we'd be left having to explain how to > > blacklist wheel files, and the UX would just generally be terrible > > (and the burden of dealing with that would fall on the pip and PyPI > > maintainers, not on the individual projects publishing insufficiently > > conservative Linux wheel files). > > > > That's why the platform override tags are such a nice idea, as it > > becomes possible to start iterating on possible solutions to the > > problem without affecting the default installation UX in the near > > term. > > > > > > pip 7+ actually has the UI for blacklisting binary packages now, primarily > to > ask "no don't build a wheel for X", but it also functions to ask *not* to > accept wheels for a particular project from PyPI. > > In my mind, the biggest reason to not just open up the ability to upload > even > generic linux wheels right now is the lack of a safe-ish default. I think > if > we added a few things: > > * Default to per platform tags (e.g. ubuntu_14_04), but allow this to be > customized and also accept "Generic" Linux wheels as well. > * Put the libc into the file name as well since it's reasonable to build a > "generic" linux wheel that statically links all dependencies (afaik), > however > it does not really work to statically link glibc. This means that even if > you build against an old version of glibc if you're running on a Linux > that > *doesnt* use glibc (like Alpine which uses MUSL) you'll run into > problems. > > I think that it is entirely possible to build a generic linux wheel that > will > work on any Linux built the same libc* of the same or newer version, > however I > think that you have to be careful if you do it. You have to ensure all your > dependencies are statically linked (if you have any) and you have to ensure > that you build against an old enough Linux (likely some form of CentOS). > > * Side question, since I don't actually know how a computer works: Is it > even > possible to have a CPython extension link against a different libc than > CPython itself is linked against? What if static linking is involved > since > there are non glibc libcs which actually do support static linking? > You can use versioned symbols to manage some of those issues ( https://www.kernel.org/pub/software/libs/glibc/hjl/compat/). Some softwares even include their own libc, but this is getting hair fast. The common solution is to do what Antoine mentioned: build on the lowest common denominator, and reduce the dependencies on the system as much as possible. To be honest, glibc is rarely your problem: the kernel is actually more problematic (some common python packages don't build on 2.6.18 anymore), and C++ even more so. E.g. llvm 3.6 will not build on gcc 4.1 (the version of centos 5), so you need a new g++ which means a new libtstdc++. I am biased, but that's the kind of things where you may want to work with "professional" providers with people paid to work on those boring but critical issues. David > >
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
