Bug#944364: dpkg: ldconfig is not invoked for Depends or even Pre-Depends
Control: reassign -1 python3.7
Control: severity -1 important
On Wed, 13 Nov 2019 01:27:11 +0100 Guillem Jover wrote:
> Control: reassign -1 python3-augeas
> Control: severity -1 serious
>
> Hi!
>
> On Sat, 2019-11-09 at 11:06:15 +0100, Jakub Wilk wrote:
> > * Alexander Thomas , 2019-11-08, 15:50:
> > > Traceback (most recent call last):
> > > File "/usr/lib/the-package/setup-package.py", line 3, in
> > >from augeas import Augeas
> > > File "/usr/lib/python2.7/dist-packages/augeas.py", line 78, in
> > >class Augeas(object):
> > > File "/usr/lib/python2.7/dist-packages/augeas.py", line 82, in Augeas
> > >_libaugeas = _dlopen("augeas")
> > > File "/usr/lib/python2.7/dist-packages/augeas.py", line 75, in _dlopen
> > >raise ImportError("Unable to import lib%s!" % args[0])
> > > ImportError: Unable to import libaugeas!
> >
> > The _dlopen() function uses ctypes.util.find_library(), which is a
> > fundamentally broken API. This is the culprit, not dpkg.
>
> Ah, thanks for tracking this down! I was not sure whether the ld caching
> logic in glibc might have regressed perhaps.
>
> > > ldconfig is one of the triggers of libc-bin. It seems that its
> > > invocation is now postponed too late.
> >
> > ldconfig is declared as a noawait trigger, so dpkg is allowed to configure
> > the triggering package immediately, without waiting for the trigger to be
> > run.
> >
> > This declaration is correct, because running ldconfig shouldn't have any
> > effect on software functionality, unless there's a bug somewhere else.
>
> Exactly. So I guess this needs to be reassigned first to augeas, which
> I'm doing now. And perhaps cloned or a new bug filed to python too for
> the brokeness in the ctypes.util.find_library() API, but I'd leave that
> to you Jakub, as I've not checked any of that.
Hello,
When reporting this bug, python-augeas was used as an example to
demonstrate the problem when a python library is used by a package that
depends on it. The problem is not specific to pythone-augeas and is
reproducible on other python packages that use
ctypes.util.find_library(). I have tried this with the library
python3-magic which opens its library as follows:
"ctypes.cdll.LoadLibrary(find_library('magic'))"
...
Setting up libpython3.7-stdlib:amd64 (3.7.6-1) ...
Setting up libpython3-stdlib:amd64 (3.7.5-3) ...
Setting up python3.7 (3.7.6-1) ...
Setting up python3 (3.7.5-3) ...
running python rtupdate hooks for python3.7...
running python post-rtupdate hooks for python3.7...
Setting up python3-magic (2:0.4.15-2) ...
Setting up the-package (1.2.3ubuntu1-2-1) ...
Traceback (most recent call last):
File "/usr/lib/the-package/setup-package.py", line 3, in
import magic
File "/usr/lib/python3/dist-packages/magic/__init__.py", line 361, in
add_compat(globals())
File "/usr/lib/python3/dist-packages/magic/__init__.py", line 325, in
add_compat
from magic import compat
File "/usr/lib/python3/dist-packages/magic/compat.py", line 61, in
_open = _libraries['magic'].magic_open
File "/usr/lib/python3.7/ctypes/__init__.py", line 377, in __getattr__
func = self.__getitem__(name)
File "/usr/lib/python3.7/ctypes/__init__.py", line 382, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/bin/python3: undefined symbol: magic_open
dpkg: error processing package the-package (--configure):
installed the-package package post-installation script subprocess
returned error exit status 1
Processing triggers for libc-bin (2.29-6) ...
Errors were encountered while processing:
the-package
E: Sub-process /usr/bin/dpkg returned an error code (1)
This is either a bug with all python libraries that use ctypes'
find_library() for simply using the method or it is a bug with ctypes
find_library(). I am inclined to think the latter because judging by
ctypes documentation, using find_library() followed by LoadLibrary() is
the expected usage pattern on GNU/Linux. Hence reassigning this bug to
Python.
Also, severity 'serious' does not fit this bug as the affected packages
(and ctypes API) are fully usable expect from within Debian postinst
scripts. Demoting severity to important again.
Thanks,
--
Sunil
signature.asc
Description: OpenPGP digital signature
Bug#944364: dpkg: ldconfig is not invoked for Depends or even Pre-Depends
Control: reassign -1 python3-augeas
Control: severity -1 serious
Hi!
On Sat, 2019-11-09 at 11:06:15 +0100, Jakub Wilk wrote:
> * Alexander Thomas , 2019-11-08, 15:50:
> > Traceback (most recent call last):
> > File "/usr/lib/the-package/setup-package.py", line 3, in
> >from augeas import Augeas
> > File "/usr/lib/python2.7/dist-packages/augeas.py", line 78, in
> >class Augeas(object):
> > File "/usr/lib/python2.7/dist-packages/augeas.py", line 82, in Augeas
> >_libaugeas = _dlopen("augeas")
> > File "/usr/lib/python2.7/dist-packages/augeas.py", line 75, in _dlopen
> >raise ImportError("Unable to import lib%s!" % args[0])
> > ImportError: Unable to import libaugeas!
>
> The _dlopen() function uses ctypes.util.find_library(), which is a
> fundamentally broken API. This is the culprit, not dpkg.
Ah, thanks for tracking this down! I was not sure whether the ld caching
logic in glibc might have regressed perhaps.
> > ldconfig is one of the triggers of libc-bin. It seems that its
> > invocation is now postponed too late.
>
> ldconfig is declared as a noawait trigger, so dpkg is allowed to configure
> the triggering package immediately, without waiting for the trigger to be
> run.
>
> This declaration is correct, because running ldconfig shouldn't have any
> effect on software functionality, unless there's a bug somewhere else.
Exactly. So I guess this needs to be reassigned first to augeas, which
I'm doing now. And perhaps cloned or a new bug filed to python too for
the brokeness in the ctypes.util.find_library() API, but I'd leave that
to you Jakub, as I've not checked any of that.
Thanks,
Guillem
Bug#944364: dpkg: ldconfig is not invoked for Depends or even Pre-Depends
* Alexander Thomas , 2019-11-08, 15:50:
Traceback (most recent call last):
File "/usr/lib/the-package/setup-package.py", line 3, in
from augeas import Augeas
File "/usr/lib/python2.7/dist-packages/augeas.py", line 78, in
class Augeas(object):
File "/usr/lib/python2.7/dist-packages/augeas.py", line 82, in Augeas
_libaugeas = _dlopen("augeas")
File "/usr/lib/python2.7/dist-packages/augeas.py", line 75, in _dlopen
raise ImportError("Unable to import lib%s!" % args[0])
ImportError: Unable to import libaugeas!
The _dlopen() function uses ctypes.util.find_library(), which is a
fundamentally broken API. This is the culprit, not dpkg.
ldconfig is one of the triggers of libc-bin. It seems that its
invocation is now postponed too late.
ldconfig is declared as a noawait trigger, so dpkg is allowed to
configure the triggering package immediately, without waiting for the
trigger to be run.
This declaration is correct, because running ldconfig shouldn't have any
effect on software functionality, unless there's a bug somewhere else.
--
Jakub Wilk
Bug#944364: dpkg: ldconfig is not invoked for Depends or even Pre-Depends
Package: dpkg
Version: 1.19.7
Severity: important
Dear Maintainer,
This was first discovered in Ubuntu 18.04 and has already been filed as
Ubuntu bug #1851675, but was later on found to have originated in changes
to dpkg in Debian itself.
https://bugs.launchpad.net/ubuntu/+source/dpkg/+bug/1851675
In a nutshell, if a package has a Depends: that will cause a library to
be installed, and the package relies on that library in a postinst hook,
then this hook will fail if the library was not already installed before
performing an apt-get install of the package itself.
As an example, I have a package that has a "Depends: python-augeas".
A stripped-down example of this package can be downloaded from the
Ubuntu bug report.
In its postinst script, a Python script is invoked that tries to do:
from augeas import Augeas
This should work because all the requirements to make this import work,
should be made available by installing the python-augeas dependency. The
actual result however, is:
Unpacking the-package (1.2.3) ...
Setting up the-package (1.2.3) ...
Traceback (most recent call last):
File "/usr/lib/the-package/setup-package.py", line 3, in
from augeas import Augeas
File "/usr/lib/python2.7/dist-packages/augeas.py", line 78, in
class Augeas(object):
File "/usr/lib/python2.7/dist-packages/augeas.py", line 82, in Augeas
_libaugeas = _dlopen("augeas")
File "/usr/lib/python2.7/dist-packages/augeas.py", line 75, in _dlopen
raise ImportError("Unable to import lib%s!" % args[0])
ImportError: Unable to import libaugeas!
dpkg: error processing package the-package (--configure):
installed the-package package post-installation script subprocess returned
error exit status 1
The reason why this fails is because python-augeas depends on libaugeas0,
but this library has not yet been completely configured. This can be seen
by adding `ldconfig -p >/tmp/ldconfig-output` to the postinst script.
Simply running apt-get -f install afterwards, will work, because then
libaugeas0 is fully configured.
Moving python-augeas or even libaugeas0 to Pre-Depends, does not help as
a workaround (and according to my interpretation of debian-policy,
should not be needed).
This worked as expected until recently. The regression must have been
introduced by one of the changes listed here, most likely one of the
changes related to trigger loops:
http://launchpadlibrarian.net/43334/dpkg_1.19.0.5ubuntu2.1_1.19.0.5ubuntu2.2.diff.gz
ldconfig is one of the triggers of libc-bin. It seems that its
invocation is now postponed too late. If a package 'Depends' directly or
indirectly on libraries that were installed during the same run, then
ldconfig must be invoked before setting up the package to ensure it can
use the libraries in its postinst hook.
-- Package-specific info:
-- System Information:
Debian Release: 10.1
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.3.0-19-generic (SMP w/8 CPU cores)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_WARN, TAINT_OOT_MODULE
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C.UTF-8
(charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages dpkg depends on:
ii libbz2-1.0 1.0.6-9.2~deb10u1
ii libc62.28-10
ii liblzma5 5.2.4-1
ii libselinux1 2.8-1+b1
ii tar 1.30+dfsg-6
ii zlib1g 1:1.2.11.dfsg-1
dpkg recommends no packages.
Versions of packages dpkg suggests:
ii apt1.8.2
pn debsig-verify
-- no debconf information
