On Sun, Feb 7, 2016 at 7:48 PM, Nick Coghlan <ncogh...@gmail.com> wrote: > On 8 February 2016 at 08:33, Matthew Brett <matthew.br...@gmail.com> wrote: >> It seems reasonable to build to the same compatibility level as most >> Debian packaged modules. > > Right, one of the key things to remember with manylinux1 is that it > is, *quite deliberately*, only an 80% solution to the cross-distro > lack-of-ABI-compatibility problem: we want to solve the simple cases > now, and then move on to figuring out how to solve the more complex > cases later (and applications that embed their own Python runtimes are > a whole world of pain, in more ways than one). > > Since we know that extensions built against a statically linked > CPython will run correctly against a dynamically linked one, then it > probably makes sense to go down that path for the manylinux1 reference > build environment. > > However, there's one particular test case we should investigate before > committing to that path: loading manylinux1 wheels built against a > statically linked CPython into a system httpd environment running the > system mod_wsgi. If I've understood the problem description correctly, > that *should* work, but if it doesn't, then it would represent a > significant compatibility concern.
That's actually a great example of the case that "ought" to fail, because first you have a host program (apache2) that uses dlopen() to load in the CPython interpreter (implicitly, by dlopen'ing mod_wsgi.so, which is linked to libpython), and then the CPython interpreter turns around and tries to use dlopen() to load extension modules. Normally, this should work if and only if the extension modules are themselves linked against libpythonX.Y.so, --enable-shared / Fedora style. However, this is not Apache's first rodeo: revision 1.12 date: 1998/07/10 18:29:50; author: rasmus; state: Exp; lines: +2 -2 Set the RTLD_GLOBAL dlopen mode parameter to allow dynamically loaded modules to load their own modules dynamically. This improves mod_perl and mod_php3 when these modules are loaded dynamically into Apache. (Confirmation that this is still true: https://apr.apache.org/docs/apr/2.0/group__apr__dso.html#gaedc8609c2bb76e5c43f2df2281a9d8b6 -- also I ran mod_wsgi-express under LD_DEBUG=scopes and that also showed libpython2.7.so.1 getting added to the global scope.) Using RTLD_GLOBAL like this is the "wrong" thing -- it means that different Apache mods all get loaded into the same global namespace, and that means that they can have colliding symbols and step on each other's feet. E.g., this is the sole and entire reason why you can't load a python2 version of mod_wsgi and a python3 version of mod_wsgi into the same apache. But OTOH it means that Python extension modules will work even if they don't explicitly link against libpython. I also managed to track down two other programs that also follow this load-a-plugin-that-embeds-python pattern -- LibreOffice and xchat -- and they also both seem to use RTLD_GLOBAL. So even if it's the "wrong" thing, extension modules that don't explicitly link to libpython do seem to work reliably here in the world we have, and they're more compatible with Debian/Ubuntu and their massive market share, so... the whole manylinux1 strategy is nothing if not relentlessly pragmatic. I guess we should forbid linking to libpython in the PEP. [Note: I did not actually try loading any such modules into mod_wsgi or these other programs, because I have no idea how to use mod_wsgi or these other programs :-). The LD_DEBUG output is fairly definitive, but it wouldn't hurt for someone to double-check if they feel inspired...] -n -- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
