You don't necessarily need rpath actually. The Linux loader has a bug/feature where once it has successfully loaded a library with a given soname, then any future requests for that soname within the same process will automatically return that same library, regardless of rpath settings etc. So as long as the main interpreter has loaded libpython.whatever from the correct directory, then extension modules will all get that same version. The rpath won't matter at all.
It is annoying in general that on Linux, we have these two different ways to build extension modules. It definitely violates TOOWTDI :-). It would be nice at some point to get rid of one of them. Note that we can't get rid of the two different ways entirely though – on Windows, extension modules *must* link to libpython.dll, and on macOS, extension modules *can't* link to libpython.dylib. So the best we can hope for is to make Linux consistently do one of these, instead of supporting both. In principle, having extension modules link to libpython.so is a good thing. Suppose that someone wants to dynamically load the python interpreter into their program as some kind of plugin. (Examples: Apache's mod_python, LibreOffice's support for writing macros in Python.) It would be nice to be able to load python2 and python3 simultaneously into the same process as distinct plugins. And this is totally doable in theory, *but* it means that you can't assume that the interpreter's symbols will be automagically injected into extension modules, so it's only possible if extension modules link to libpython.so. In practice, extension modules have never consistently linked to libpython.so, so everybody who loads the interpreter as a plugin has already worked around this. Specifically, they use RTLD_GLOBAL to dump all the interpreter's symbols into the global namespace. This is why you can't have python2 and python3 mod_python at the same time in the same Apache. And since everyone is already working around this, linking to libpython.so currently has zero benefit... in fact manylinux wheels are actually forbidden to link to libpython.so, because this is the only way to get wheels that work on every interpreter. -n On Wed, Apr 24, 2019, 09:54 Victor Stinner <vstin...@redhat.com> wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When Python is built with --enable-shared, the python3.7 program is > linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly > linked to libpython3.7m as well: > > $ python3.7-config --ldflags > ... -lpython3.7m ... > > Example with numpy: > > $ ldd /usr/lib64/python3.7/site-packages/numpy/core/ > umath.cpython-37m-x86_64-linux-gnu.so > ... > libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) > ... > > When Python 3.7 is compiled in debug mode, libpython gets a "d" flag > for debug: libpython3.7dm.so.1.0. > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be > compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug > libpython. > > (2) If "libpython" gets a different filename in debug mode, C > extensions should not be linked to libpython explicitly but > *implicitly* to avoid picking the wrong libpython. For example, remove > "-lpython3.7m" from "python3.7-config --ldflags" output. > > The option (1) rely on rpath which is discouraged by Linux vendors and > may not be supported by all operating systems. > > The option (2) is simpler and likely more portable. > > Currently, C extensions of the standard library may or may not be > linked to libpython depending on they are built. In practice, both > work since python3.7 is already linked to libpython: so libpython is > already loaded in memory before C extensions are loaded. > > I opened https://bugs.python.org/issue34814 to discuss how C > extensions of the standard library should be linked but I closed it > because we failed to find a consensus and the initial use case became > a non-issue. It seems like we should reopen the discussion :-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com > On Wed, Apr 24, 2019, 09:54 Victor Stinner <vstin...@redhat.com> wrote: > Hum, I found issues with libpython: C extensions are explicitly linked > to libpython built in release mode. So a debug python loading a C > extension may load libpython in release mode, whereas libpython in > debug mode is already loaded. > > When Python is built with --enable-shared, the python3.7 program is > linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly > linked to libpython3.7m as well: > > $ python3.7-config --ldflags > ... -lpython3.7m ... > > Example with numpy: > > $ ldd /usr/lib64/python3.7/site-packages/numpy/core/ > umath.cpython-37m-x86_64-linux-gnu.so > ... > libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...) > ... > > When Python 3.7 is compiled in debug mode, libpython gets a "d" flag > for debug: libpython3.7dm.so.1.0. > > I see 2 solutions: > > (1) Use a different directory. If "libpython" gets the same filename > in release and debug mode, at least, they must be installed in > different directories. If libpython build in debug mode is installed > in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be > compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug > libpython. > > (2) If "libpython" gets a different filename in debug mode, C > extensions should not be linked to libpython explicitly but > *implicitly* to avoid picking the wrong libpython. For example, remove > "-lpython3.7m" from "python3.7-config --ldflags" output. > > The option (1) rely on rpath which is discouraged by Linux vendors and > may not be supported by all operating systems. > > The option (2) is simpler and likely more portable. > > Currently, C extensions of the standard library may or may not be > linked to libpython depending on they are built. In practice, both > work since python3.7 is already linked to libpython: so libpython is > already loaded in memory before C extensions are loaded. > > I opened https://bugs.python.org/issue34814 to discuss how C > extensions of the standard library should be linked but I closed it > because we failed to find a consensus and the initial use case became > a non-issue. It seems like we should reopen the discussion :-) > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com >
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com