Hi Ole
On 05/01/2026 01:27, Ole Streicher wrote:
Hi Stuart,
thank you very much for your investigation and patch!
I must however say that I do not fully understand the bug and also not
your solution:
Your approach is (if I understand yoour execute_after_dh_auto_install
correctly) not to rebuild the libcasa_python lib for each Python
version, but to copy it and then patch the right soname into. This works
only if the libs would be ABI compatible.
Not quite. The configure, build, and install steps *are* run for each
interpreter - this is done by having debhelper use the pybuild
buildsystem which in turn runs cmake with appropriate arguments, rather
than running cmake directly from debhelper. At the end of the build
stage there are two complete build trees created:
.pybuild/cpython3_3.13/build
.pybuild/cpython3_3.14/build
with .so that links against the correct libboost_pythonXYZ.so.A.BC
within each of these trees. At the end of the install step, they are
sitting in debian/tmp/usr/lib/<triplet>-pythonX.YZ/ and need merging.
For the non-Python-using libraries, it doesn't matter which one is used
and so they are all collected from the compilation that used the Python
default interpreter.
If the casacore .so were like a 'normal' Python extension module, then
we'd just compile it against each interpreter and then throw all the
compiled modules into the Python dist-packages with their tagged
filenames and it would all work out.
However, these are not like the 'normal' Python extension modules that
we are more familiar with.
- libcasa-python3-X is not a normal Python extension module itself but
rather a .so that Python extension modules will be compiled against;
none of the tooling understand this extra level of indirection
- nothing in the upstream code for casacore or boost-python ensures that
when running Python 3.14, that libboost_python314 is used, and in fact
what happens is that libboost_python313 is accidentally loaded into
Python 3.14 which is the cause of the segfaults when trying to build
python-casaore.
When I say that nothing prevents loading wrong library, that's because
the loader and linker are based on sonames and search paths but
libcases_python3 contains no information about what Python version it
was built against in either its soname or on-disk location. To make sure
that Python 3.13 loads the 3.13-compatible .so, we either need to fiddle
with those to ensure that both linker and loader find the right .so...
that is we can:
- fiddle with sonames (that is what I've done in this work), making
the soname include the Python version just as boost itself does, so that
the linker and then loader have an unambiguous name to load, by
injecting the Python version and arch triplet into the soname and
filename: libcasa_python3.cpython-313-x86_64-linux-gnu.so.9
- fiddle with the search path by putting the ambiguously named
libcasa_python3.so.9 into an interpreter-specific directory, and rely on
rpath munging somewhere to ensure that the correct version is found.
(Neither is lovely, but the latter felt more fragile; further, the
former is more familiar with libboost_pythonXYZ following this path, so
that's the approach I took here. Setting the compatibility as a tag in
the filename also mirrors the standard Python approach to .so)
The complementary change to python-casacore debian/rules makes it find
the pyver-specific libcasa_python3.<tag>.so when compiling the actual
python extension module - both linker and loader will now use the right
.so because it has the pyver-specific name, with pybuild doing all the
normal loops over supported interpreters for us, and dh_python3 doing
the normal work to merge those build outputs to create a package that
combines support for all supported Python interpreters.
I will note that there is one remaining wart but it's not a blocker for
this work and can be considered later. What should happen to the old
'libcasa_python3.so.9' given that it is now unneeded thanks to the
pyver-specific files? If we knew that no-one outside of Debian's
python-casacore used this soname then I'd suggest dropping that file
from the packages entirely and sticking only with the nicely versioned
sonames described above. Perhaps, however, it is a public-enough-soname
to deserve to live on for compatibility, even though it has all these
problems with Python interpreter transitions. I left it in for
backwards-compatibility in this work, although I suspect it's broken
enough to think that it should go.
However, if usr/lib/<arch>/
libcasa_python3.so.9 is ABI compatible between Python versions, why
can't we just keep using this name and a single library?
It is indeed not compatible due to transitive linking - I suspect that
you had missed the detail that my proposed changes have casacore being
recompiled multiple times (for each supported Python interpreter); the
shell snippet to set the sonames based on the interpreter version is
also dense and Makefiles don't lend themselves to comments within loops,
unfortunately.
If there is an ABI incompatibility, we would in any case need to do
something else (like limiting to only one Python version), right?
Indeed - building multiple times as done here is the best-practice for
Debian Python packaging. But yes, if all else fails, then only
supporting `py3versions -d` (and not build-depending on
python3-all/python3-all-dev) would be the path to go... however, the
more packages that build correctly against all supported Python 3
versions, the easier each interpreter transition is, being more a
sequence of binNMUs with minimal flag days, less work for the release
team to hint incompatible packages out of testing so that the transition
can proceed, and a much less stressful critical path for all involved.
I know that "it compiles, ship it" is not an explanation that these
things are right, but with the changes proposed, it compiles, running
tests at compile time and autopkgtests later that all pass, whereas
without these changes to sonames, we see the segfaults from loading the
wrong version of the library.
BTW I found that looking at the output of `ldd` for each of the
libraries in python3-casacore and libcasa-python3-9 to check the
specific versions of libboost_python and libpython that are being used
was most instructive - for both the current (failing) situation and
checking this work was doing the right thing.
Please let me know if I've now explained what this is doing better and
sufficiently! Adding some detailed comments above the d/rules
execute_after_dh_auto_install block seems necessary to encompass all
this. Would you like me to add that in git? or would you prefer trying
to write it yourself to ensure you are across what this is all trying to
do? (happy to review said text if you want)
regards
Stuart
--
Stuart Prescott http://www.nanonanonano.net/ [email protected]
Debian Developer http://www.debian.org/ [email protected]
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7
