On July 9, 2015 at 7:41:25 PM, Ionel Cristian Mărieș (cont...@ionelmc.ro) wrote: > Would be quite useful to see some references and details about the vague > issues being mentioned in the thread. It would help a lot the less versed > engineers (like me) understand the issues at hand (and hopefully reduce the > amount of disagreement overall). > > For example, for me it's not clear what's wrong with Antoine's proposal > (compile on Centos 5) - it seemed quite sensible approach to produce a > reasonably compatible binary. > > Some issues with kernel ABI have been mentioned - can anyone point me to > some resources describing the possible problems? Is it correct to assume > that it's about using vendor-specific kernel api?
It is about ABI. I'm not an expert but essentially anything that a Wheel doesn't statically link needs to be ABI compatible. For a plain C extension that doesn't link to anything else you have primarily: * The libc that it was linked against. * The Python that it was linked against. Currently on Python 3 we have a "stable" ABI (though I don't think it covers the entire thing) which is represented in the Wheel filename, however we don't have anything to cover the libc version. The most common version of libc in use is glibc and that is (as far as I know) basically always ABI compatible when going from an older version to a new version. However there is no guarentee that a glibc from Linux X will be ABI compatible with a glibC from Linux Z. There isn't even a guarentee that it will be glibc at all (for instance, Alpine linux uses MUSL). On top of that, you have the fact that a lot of C extensions are not self contained C extensions but instead are bindings for some other library. The problem starts becoming a lot bigger here, for example psycopg2 is a pretty popular library that links to libpsql. This may or may not be ABI compatible across CentOS to Ubuntu (for instance) and trying to install one that isn't will cause breakages. Circling back to Antoine's suggestion, the problem isn't that it wouldn't actually work, because it would sometimes, the problem is that you need to be careful about using it because it only works in some situations. In order for that to work you need to make sure that the project you're compiling is either a completely self contained C-extension or that you statically link everything that you're linking against. You also need to make sure that the person compiling the Wheel does so on a sufficiently ancient glibc to cover everyone that you care about covering. However this will still break in "weird" ways on even older versions of glibc, on Linux distributions which do not use glibc, or just because two Linux distributions decided to make their glibc slightly incompatible. This is the reason that it's currently blocked, because the edge cases are sufficiently sharp and you have to be very careful to build your Wheels in just the right way that I felt it was better to punt on it until someone puts in the effort to make it do something more resembling the right thing by default. > > Also, what does Conda do to solve the binary compatibility issues and > distutils or pip could never ever do (or implement)? > They don't do anything (to my knowledge, and hopefully Antoine or someone can correct me if I'm wrong) to solve the libc compatiblity problem besides building on a sufficiently old version of CentOS that they feel comfortable calling that their minimum level of support. I assume that if I tried to run Conda on something like Alpine the default repositories would break since it doesn't use glibc. They however _do_ own the ABI of everything else, so everything from the Python you're using to the libpsql to the openssl comes from their repositories. This means that they don't have to worry about the ABI differences of OpenSSL in CentOS/RHEL 5 and Ubuntu 14.04 since they'll never use them, they'll just use the OpenSSL they personally have packaged. This isn't something we can adopt because we're not trying to be another platform (in the vein of Conda) we're trying to provide a common tooling that can be used across a wide range of platforms to install Python projects. This will absolutely involve trying to figure out what the right line to walk is for us between defining a defacto platform and simply being something you "plug" into another platform. I think the best way to go about this is to figure out the best way to publish Wheels for a specific platform and focus on identifying the platforms and providing ways for people to override that. A generic "Linux" platform is a reasonable one, but I don't think it's a reasonable default since to actually support it requires taking the special precautions from above. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
