Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
On Sun, 2021-01-31 at 18:49 +0100, Andrea Bolognani wrote: > I've added a few extra Depends to make sure everything is really > updated in lockstep and tested with unattended-upgrades: the result > was much better this time! ... > I believe we can merge the MR and consider the issue resolved. Looks much better indeed, thanks for your work! -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
On Sun, Jan 31, 2021 at 04:36:21PM +0100, Andrea Bolognani wrote: > I've opened > > https://salsa.debian.org/libvirt-team/libvirt/-/merge_requests/98 > > with the proposed patch, and I'm going to use the information you > provided above to give it some testing now. I've added a few extra Depends to make sure everything is really updated in lockstep and tested with unattended-upgrades: the result was much better this time! Log started: 2021-01-31 18:20:43 (Reading database ... 87062 files and directories currently installed.) Preparing to unpack .../00-libnss-libvirt_7.0.0-1_amd64.deb ... Unpacking libnss-libvirt:amd64 (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../01-libvirt-sanlock_7.0.0-1_amd64.deb ... Unpacking libvirt-sanlock (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../02-libvirt-login-shell_7.0.0-1_amd64.deb ... Unpacking libvirt-login-shell (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../03-libvirt-dev_7.0.0-1_amd64.deb ... Unpacking libvirt-dev:amd64 (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../04-libvirt-daemon-config-nwfilter_7.0.0-1_all.deb ... Unpacking libvirt-daemon-config-nwfilter (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../05-libvirt-daemon-config-network_7.0.0-1_all.deb ... Unpacking libvirt-daemon-config-network (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../06-libvirt-daemon-system_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-system (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../07-libvirt-daemon-driver-qemu_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-qemu (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../08-libvirt-clients_7.0.0-1_amd64.deb ... Unpacking libvirt-clients (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../09-libvirt0_7.0.0-1_amd64.deb ... Unpacking libvirt0:amd64 (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../10-libvirt-daemon-driver-xen_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-xen (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../11-libvirt-daemon-driver-vbox_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-vbox (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../12-libvirt-daemon-driver-storage-zfs_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-storage-zfs (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../13-libvirt-daemon-driver-storage-rbd_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-storage-rbd (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../14-libvirt-daemon-driver-storage-gluster_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-storage-gluster (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../15-libvirt-daemon_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../16-libvirt-daemon-driver-lxc_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-lxc (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../17-libvirt-daemon-system-systemd_7.0.0-1_all.deb ... Unpacking libvirt-daemon-system-systemd (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-daemon-config-network (7.0.0-1) ... Setting up libvirt-daemon-system-systemd (7.0.0-1) ... Setting up libvirt0:amd64 (7.0.0-1) ... Setting up libvirt-daemon-config-nwfilter (7.0.0-1) ... Setting up libvirt-clients (7.0.0-1) ... Setting up libvirt-dev:amd64 (7.0.0-1) ... Setting up libvirt-sanlock (7.0.0-1) ... Setting up libnss-libvirt:amd64 (7.0.0-1) ... Setting up libvirt-daemon-driver-qemu (7.0.0-1) ... Setting up libvirt-daemon (7.0.0-1) ... Setting up libvirt-daemon-driver-vbox (7.0.0-1) ... Setting up libvirt-daemon-driver-storage-rbd (7.0.0-1) ... Setting up libvirt-daemon-driver-lxc (7.0.0-1) ... Setting up libvirt-login-shell (7.0.0-1) ... Setting up libvirt-daemon-driver-storage-gluster (7.0.0-1) ... Setting up libvirt-daemon-driver-xen (7.0.0-1) ... Setting up libvirt-daemon-driver-storage-zfs (7.0.0-1) ... Setting up libvirt-daemon-system (7.0.0-1) ... Installing new version of config file /etc/apparmor.d/abstractions/libvirt-lxc ... Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ... Installing new version of config file /etc/default/libvirt-guests ... Installing new version of config file /etc/libvirt/qemu.conf ... virtlockd.service is a disabled or a static unit, not starting it. virtlogd.service is a disabled or a static unit, not starting it. Processing triggers for man-db (2.9.3-2) ... Processing triggers for libc-bin (2.31-9) ... Log ended: 2021-01-31 18:20:48 Log started: 2021-01-31 18:20:49 (Reading database ... 87071 files and directories currently installed.) Preparing to unpack .../libvirt-wireshark_7.0.0-1_amd64.deb ... Unpacking libvirt-wireshark (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-wireshark (7.0.0-1) ... Log ended: 2021-01-31 18:20:49 Log started: 2021-01-31 18:20:49 (Reading database ... 87071 files and directories currently instal
Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
On Sun, Jan 31, 2021 at 11:04:13PM +0800, Paul Wise wrote:
> On Sun, 2021-01-31 at 15:34 +0100, Andrea Bolognani wrote:
> > As I've never used unattended-upgrades myself, I'm not familiar with
> > it. Is there any chance you could provide some quick tips on how to
> > set up a reproducer environment? Specifically how to set up the same
> > upgrade strategy you're using, and whether it's possible to manually
> > trigger an unattended-upgrades run? That would help a lot!
>
> You can probably also reproduce it without unattended-upgrades by just
> upgrading libvirt-daemon itself using apt but with unattended-upgrades
> the key is to enable the minimal steps option, but do this overall:
>
> Install a Debian system that has the old libvirt using the Debian
> wayback machine (snapshot.debian.org), install unattended-upgrades and
> add something like the following in /etc/apt/apt.conf.d/99override-u-u.conf
> and then run this command: systemctl start apt-daily{,-upgrade}.service
>
> APT::Periodic::Update-Package-Lists "always";
> APT::Periodic::Download-Upgradeable-Packages "always";
> APT::Periodic::AutocleanInterval "always";
> APT::Periodic::Unattended-Upgrade "always";
> Unattended-Upgrade::Origins-Pattern { "origin=Debian"; };
> Unattended-Upgrade::MinimalSteps "true";
> Unattended-Upgrade::Mail "root";
> Unattended-Upgrade::Remove-Unused-Dependencies "true";
> Unattended-Upgrade::Automatic-Reboot "false";
> // Temporary options for debugging
> //Unattended-Upgrade::Verbose "true";
> //Unattended-Upgrade::Debug "true";
Thanks, I'll try this.
> > As for the issue itself, I think it's caused by some of the
> > dependencies not being strict enough
>
> I'm not sure but I think the issue is caused by the libvirt0 symbol
> file generating incorrect dependencies for the private symbols,
> probably it should be much more restrictive for those.
>
> The missing symbols are pretty concerning, those look like broken ABI?
The exact error is
Failed to load module
'/usr/lib/x86_64-linux-gnu/libvirt/connection-driver/libvirt_driver_qemu.so':
/usr/lib/x86_64-linux-gnu/libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not
found (required by
/usr/lib/x86_64-linux-gnu/libvirt/connection-driver/libvirt_driver_qemu.so)
Note how it's talking about LIBVIRT_PRIVATE_x.y.z rather than
LIBVIRT_x.y.z: the latter contains the public API, while the former
is used for internal symbols that are not exposed publicly and are
only needed by other libvirt components, such as utility functions
that are provided by libvirt.so for use in the various drivers.
This is part of the reason why we want upgrades to happen in
lockstep: for any given version of libvirt, its various components
are really only meant to work with other components when these come
from the very same build.
I've opened
https://salsa.debian.org/libvirt-team/libvirt/-/merge_requests/98
with the proposed patch, and I'm going to use the information you
provided above to give it some testing now.
--
Andrea Bolognani
Resistance is futile, you will be garbage collected.
signature.asc
Description: PGP signature
Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
On Sun, 2021-01-31 at 15:34 +0100, Andrea Bolognani wrote:
> As I've never used unattended-upgrades myself, I'm not familiar with
> it. Is there any chance you could provide some quick tips on how to
> set up a reproducer environment? Specifically how to set up the same
> upgrade strategy you're using, and whether it's possible to manually
> trigger an unattended-upgrades run? That would help a lot!
You can probably also reproduce it without unattended-upgrades by just
upgrading libvirt-daemon itself using apt but with unattended-upgrades
the key is to enable the minimal steps option, but do this overall:
Install a Debian system that has the old libvirt using the Debian
wayback machine (snapshot.debian.org), install unattended-upgrades and
add something like the following in /etc/apt/apt.conf.d/99override-u-u.conf
and then run this command: systemctl start apt-daily{,-upgrade}.service
APT::Periodic::Update-Package-Lists "always";
APT::Periodic::Download-Upgradeable-Packages "always";
APT::Periodic::AutocleanInterval "always";
APT::Periodic::Unattended-Upgrade "always";
Unattended-Upgrade::Origins-Pattern { "origin=Debian"; };
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "false";
// Temporary options for debugging
//Unattended-Upgrade::Verbose "true";
//Unattended-Upgrade::Debug "true";
> As for the issue itself, I think it's caused by some of the
> dependencies not being strict enough
I'm not sure but I think the issue is caused by the libvirt0 symbol
file generating incorrect dependencies for the private symbols,
probably it should be much more restrictive for those.
The missing symbols are pretty concerning, those look like broken ABI?
--
bye,
pabs
https://wiki.debian.org/PaulWise
signature.asc
Description: This is a digitally signed message part
Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
On Sun, Jan 31, 2021 at 05:43:43PM +0800, Paul Wise wrote: > Source: libvirt > Version: 7.0.0-1 > Severity: serious > > There is some sort of missing dependency that means that the daemon is > stopped during certain kinds of upgrades. I use unattended-upgrades in > the minimal-steps mode rather than upgrading all packages at once. As a > result the order of upgrades seems to cause a failure to restart the > libvirt systemd service. It looks like the libvirt-daemon-* packages > need to be upgraded before the libvirt-daemon-system package. > > I have included below the apt and systemd logs for this upgrade. > > Log started: 2021-01-31 17:07:53 > apt-listchanges: Reading changelogs... > apt-listchanges: Mailing root: apt-listchanges: changelogs for chianamo > Preconfiguring packages ... > apt-listchanges: Reading changelogs... > Preconfiguring packages ... > Preparing to unpack .../libvirt0-dbgsym_7.0.0-1_amd64.deb ... > Unpacking libvirt0-dbgsym:amd64 (7.0.0-1) over (6.9.0-4) ... > Preparing to unpack .../libvirt-daemon-system_7.0.0-1_amd64.deb ... > Unpacking libvirt-daemon-system (7.0.0-1) over (6.9.0-4) ... > Preparing to unpack .../libvirt-daemon_7.0.0-1_amd64.deb ... > Unpacking libvirt-daemon (7.0.0-1) over (6.9.0-4) ... > Preparing to unpack .../libvirt-clients_7.0.0-1_amd64.deb ... > Unpacking libvirt-clients (7.0.0-1) over (6.9.0-4) ... > Preparing to unpack .../libvirt0_7.0.0-1_amd64.deb ... > Unpacking libvirt0:amd64 (7.0.0-1) over (6.9.0-4) ... > Setting up libvirt0:amd64 (7.0.0-1) ... > Setting up libvirt-daemon (7.0.0-1) ... > Setting up libvirt-clients (7.0.0-1) ... > Setting up libvirt0-dbgsym:amd64 (7.0.0-1) ... > Setting up libvirt-daemon-system (7.0.0-1) ... > Installing new version of config file > /etc/apparmor.d/abstractions/libvirt-lxc ... > Installing new version of config file > /etc/apparmor.d/abstractions/libvirt-qemu ... > Installing new version of config file /etc/default/libvirt-guests ... > Installing new version of config file /etc/libvirt/qemu.conf ... > Job for libvirtd.service failed because the control process exited with error > code. > See "systemctl status libvirtd.service" and "journalctl -xe" for details. > virtlockd.service is a disabled or a static unit, not starting it. > virtlogd.service is a disabled or a static unit, not starting it. > Log ended: 2021-01-31 17:09:05 > > Log started: 2021-01-31 17:13:02 > apt-listchanges: Reading changelogs... > apt-listchanges: Reading changelogs... > Preparing to unpack .../libvirt-daemon-config-nwfilter_7.0.0-1_all.deb ... > Unpacking libvirt-daemon-config-nwfilter (7.0.0-1) over (6.9.0-4) ... > Setting up libvirt-daemon-config-nwfilter (7.0.0-1) ... > Log ended: 2021-01-31 17:13:24 > > Log started: 2021-01-31 17:15:12 > apt-listchanges: Reading changelogs... > apt-listchanges: Reading changelogs... > Preparing to unpack .../libvirt-daemon-config-network_7.0.0-1_all.deb ... > Unpacking libvirt-daemon-config-network (7.0.0-1) over (6.9.0-4) ... > Setting up libvirt-daemon-config-network (7.0.0-1) ... > Log ended: 2021-01-31 17:15:32 > > Log started: 2021-01-31 17:15:33 > apt-listchanges: Reading changelogs... > apt-listchanges: Reading changelogs... > Preparing to unpack .../libvirt-daemon-system-systemd_7.0.0-1_all.deb ... > Unpacking libvirt-daemon-system-systemd (7.0.0-1) over (6.9.0-4) ... > Setting up libvirt-daemon-system-systemd (7.0.0-1) ... > Log ended: 2021-01-31 17:15:53 > > Log started: 2021-01-31 17:15:54 > apt-listchanges: Reading changelogs... > apt-listchanges: Reading changelogs... > Preparing to unpack .../libvirt-daemon-driver-qemu_7.0.0-1_amd64.deb ... > Unpacking libvirt-daemon-driver-qemu (7.0.0-1) over (6.9.0-4) ... > Setting up libvirt-daemon-driver-qemu (7.0.0-1) ... > Processing triggers for man-db (2.9.3-2) ... > Log ended: 2021-01-31 17:16:22 > > Jan 31 17:08:34 kernel: audit: type=1400 audit(1612084114.263:107): > apparmor="STATUS" operation="profile_replace" info="same as current profile, > skipping" profile="unconfined" name="libvirtd" pid=1318234 > comm="apparmor_parser" > Jan 31 17:08:34 kernel: audit: type=1400 audit(1612084114.263:108): > apparmor="STATUS" operation="profile_replace" info="same as current profile, > skipping" profile="unconfined" name="libvirtd//qemu_bridge_helper" > pid=1318234 comm="apparmor_parser" > Jan 31 17:08:34 audit[1318234]: AVC apparmor="STATUS" > operation="profile_replace" info="same as current profile, skipping" > profile="unconfined" name="libvirtd" pid=1318234 comm="apparmor_parser" > Jan 31 17:08:34 audit[1318234]: AVC apparmor="STATUS" > operation="profile_replace" info="same as current profile, skipping" > profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=1318234 > comm="apparmor_parser" > Jan 31 17:08:34 systemd[1]: Reloading. > Jan 31 17:08:36 systemd[1]: Stopping Virtualization daemon... > Jan 31 17:08:36 systemd[1]: libvirtd.service: Succeeded. > Jan 31 17:08:36 systemd[1]: Stopped Virtualization daemon. > Jan 3
Bug#981435: libvirt: stops on upgrade: internal error: Failed to load module 'libvirt_driver_qemu.so': libvirt.so.0: version `LIBVIRT_PRIVATE_6.9.0' not found (required by libvirt_driver_qemu.so)
Source: libvirt Version: 7.0.0-1 Severity: serious There is some sort of missing dependency that means that the daemon is stopped during certain kinds of upgrades. I use unattended-upgrades in the minimal-steps mode rather than upgrading all packages at once. As a result the order of upgrades seems to cause a failure to restart the libvirt systemd service. It looks like the libvirt-daemon-* packages need to be upgraded before the libvirt-daemon-system package. I have included below the apt and systemd logs for this upgrade. Log started: 2021-01-31 17:07:53 apt-listchanges: Reading changelogs... apt-listchanges: Mailing root: apt-listchanges: changelogs for chianamo Preconfiguring packages ... apt-listchanges: Reading changelogs... Preconfiguring packages ... Preparing to unpack .../libvirt0-dbgsym_7.0.0-1_amd64.deb ... Unpacking libvirt0-dbgsym:amd64 (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../libvirt-daemon-system_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-system (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../libvirt-daemon_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../libvirt-clients_7.0.0-1_amd64.deb ... Unpacking libvirt-clients (7.0.0-1) over (6.9.0-4) ... Preparing to unpack .../libvirt0_7.0.0-1_amd64.deb ... Unpacking libvirt0:amd64 (7.0.0-1) over (6.9.0-4) ... Setting up libvirt0:amd64 (7.0.0-1) ... Setting up libvirt-daemon (7.0.0-1) ... Setting up libvirt-clients (7.0.0-1) ... Setting up libvirt0-dbgsym:amd64 (7.0.0-1) ... Setting up libvirt-daemon-system (7.0.0-1) ... Installing new version of config file /etc/apparmor.d/abstractions/libvirt-lxc ... Installing new version of config file /etc/apparmor.d/abstractions/libvirt-qemu ... Installing new version of config file /etc/default/libvirt-guests ... Installing new version of config file /etc/libvirt/qemu.conf ... Job for libvirtd.service failed because the control process exited with error code. See "systemctl status libvirtd.service" and "journalctl -xe" for details. virtlockd.service is a disabled or a static unit, not starting it. virtlogd.service is a disabled or a static unit, not starting it. Log ended: 2021-01-31 17:09:05 Log started: 2021-01-31 17:13:02 apt-listchanges: Reading changelogs... apt-listchanges: Reading changelogs... Preparing to unpack .../libvirt-daemon-config-nwfilter_7.0.0-1_all.deb ... Unpacking libvirt-daemon-config-nwfilter (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-daemon-config-nwfilter (7.0.0-1) ... Log ended: 2021-01-31 17:13:24 Log started: 2021-01-31 17:15:12 apt-listchanges: Reading changelogs... apt-listchanges: Reading changelogs... Preparing to unpack .../libvirt-daemon-config-network_7.0.0-1_all.deb ... Unpacking libvirt-daemon-config-network (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-daemon-config-network (7.0.0-1) ... Log ended: 2021-01-31 17:15:32 Log started: 2021-01-31 17:15:33 apt-listchanges: Reading changelogs... apt-listchanges: Reading changelogs... Preparing to unpack .../libvirt-daemon-system-systemd_7.0.0-1_all.deb ... Unpacking libvirt-daemon-system-systemd (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-daemon-system-systemd (7.0.0-1) ... Log ended: 2021-01-31 17:15:53 Log started: 2021-01-31 17:15:54 apt-listchanges: Reading changelogs... apt-listchanges: Reading changelogs... Preparing to unpack .../libvirt-daemon-driver-qemu_7.0.0-1_amd64.deb ... Unpacking libvirt-daemon-driver-qemu (7.0.0-1) over (6.9.0-4) ... Setting up libvirt-daemon-driver-qemu (7.0.0-1) ... Processing triggers for man-db (2.9.3-2) ... Log ended: 2021-01-31 17:16:22 Jan 31 17:08:34 kernel: audit: type=1400 audit(1612084114.263:107): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirtd" pid=1318234 comm="apparmor_parser" Jan 31 17:08:34 kernel: audit: type=1400 audit(1612084114.263:108): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=1318234 comm="apparmor_parser" Jan 31 17:08:34 audit[1318234]: AVC apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirtd" pid=1318234 comm="apparmor_parser" Jan 31 17:08:34 audit[1318234]: AVC apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirtd//qemu_bridge_helper" pid=1318234 comm="apparmor_parser" Jan 31 17:08:34 systemd[1]: Reloading. Jan 31 17:08:36 systemd[1]: Stopping Virtualization daemon... Jan 31 17:08:36 systemd[1]: libvirtd.service: Succeeded. Jan 31 17:08:36 systemd[1]: Stopped Virtualization daemon. Jan 31 17:08:36 systemd[1]: Starting Virtualization daemon... Jan 31 17:08:36 libvirtd[1318287]: libvirt version: 7.0.0, package: 1 (Andrea Bolognani Thu, 28 Jan 2021 22:06:43 +0100) Jan 31 17:08:36 libvirtd[1318287]: hostname: chianamo Jan 31 17:08:36 libvirtd[
