Bug#932759: [Pkg-xen-devel] Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
On 7/23/19 4:07 PM, niek wrote: > [...] > 2019-07-21 07:38:40 status installed xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:40 remove xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 > 2019-07-21 07:38:41 status installed libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 remove libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status half-configured libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status half-installed libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status config-files libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 > [...] > 2019-07-21 07:38:42 status installed xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:42 remove xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:42 status half-configured > xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:42 status half-installed xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:39:41 status config-files xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 Ok, so, the most interesting question for me is... On line 50 in the init script: https://salsa.debian.org/xen-team/debian-xen/blob/master/debian/xen-utils-common.xen.init case $DPKG_MAINTSCRIPT_PACKAGE in xen-utils-$VERSION) ;; # xen-utils-V maintscript, under Xen X=V xen-utils-*)exit 0;; # xen-utils-V maintscript, but under Xen X!=V *) ;; # maybe not under dpkg, etc. esac What is the value of this $DPKG_MAINTSCRIPT_PACKAGE when it happens? Could it be something else than something beginning with xen-utils-? I have a suspicion that the systemd[1]: Reloading. has something to do with it. Or the triggers? Anyway, if DPKG_MAINTSCRIPT_PACKAGE gets lost *anywhere* in whatever happens, it might end up as empty, and then matching just *. But, we really need find out how to reproduce it in a test environment. :| Hans
Bug#932759: [Pkg-xen-devel] Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
Some more lines from /var/log/syslog of that same time. Looks like some process is running various scripts in /usr/lib/linux-boot-probes/ and /usr/lib/os-probes. Jul 21 07:38:41 host systemd[1]: xen.service: Succeeded. Jul 21 07:38:41 host systemd[1]: Stopped LSB: Xen daemons. Jul 21 07:38:53 host kernel: SGI XFS with ACLs, security attributes, realtime, no debug enabled Jul 21 07:38:53 host kernel: JFS: nTxBlock = 7137, nTxLock = 57100 Jul 21 07:38:53 host kernel: QNX4 filesystem 0.2.3 registered. Jul 21 07:38:54 host kernel: raid6: sse2x1 gen() 4028 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x1 xor() 3036 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x2 gen() 4666 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x2 xor() 3596 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x4 gen() 5263 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x4 xor() 3941 MB/s Jul 21 07:38:54 host kernel: raid6: using algorithm sse2x4 gen() 5263 MB/s Jul 21 07:38:54 host kernel: raid6: xor() 3941 MB/s, rmw enabled Jul 21 07:38:54 host kernel: raid6: using ssse3x2 recovery algorithm Jul 21 07:38:54 host kernel: xor: measuring software checksum speed Jul 21 07:38:54 host kernel:prefetch64-sse: 6630.000 MB/sec Jul 21 07:38:54 host kernel:generic_sse: 5853.000 MB/sec Jul 21 07:38:54 host kernel: xor: using function: prefetch64-sse (6630.000 MB/sec) Jul 21 07:38:54 host kernel: Btrfs loaded, crc32c=crc32c-intel Jul 21 07:38:54 host kernel: fuse init (API version 7.27) Jul 21 07:38:54 host systemd[1]: Mounting FUSE Control File System... Jul 21 07:38:54 host systemd[1]: Mounted FUSE Control File System. Jul 21 07:38:55 host os-prober[11566]: debug: running /usr/lib/os-probes/mounted/05efi on mounted /dev/sda1 (...) Jul 21 07:39:38 host 50mounted-tests[16816]: debug: /usr/lib/linux-boot-probes/mounted/40grub2 succeeded Jul 21 07:39:38 host systemd[1828]: var-lib-os\x2dprober-mount.mount: Succeeded. Jul 21 07:39:38 host systemd[1]: var-lib-os\x2dprober-mount.mount: Succeeded. Jul 21 07:39:38 host linux-boot-prober[16820]: debug: linux detected by /usr/lib/linux-boot-probes/50mounted-tests niek
Bug#932759: [Pkg-xen-devel] Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
On 22-7-2019 18:57, Hans van Kranenburg wrote: > Hi niek, > > Thanks for the report! > > On 7/22/19 8:32 PM, niek wrote: >> Package: xen-hypervisor-4.11-amd64 >> Version: 4.11.1+92-g6c33308a8d-2 >> >> What happened: >> - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as >> described in >> https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html >> >> - started some Linux pv domu without problems >> >> - removed obsolete packages with 'apt autoremove'. This removed (among >> others) >> xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), >> libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), >> xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) >> >> [...] >> - xenconsoled was not running >> >> - searching system logs revealed that xenconsoled seemed to have stopped >> when 'apt autoremove' removed the obsolete xen 4.8 >> packages after upgrading to xen 4.11. > > Well, there it is again. We tried to make a fix, exactly for this... > > https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a > > ...and apparently there's another scenario in which even this doesn't work? > > Can you show the lines from /var/log/dpkg.log from that moment, the > seconds around 07:38:40? It tells exactly what got removed, in what > second, just to confirm? > > I'm pretty sure I tried to reproduce this after we added the fix I just > referenced, and I was unable to. So, I'm very interested in finding out > what's still going on here. > > Usually being able to reproduce a problem is one of the biggest steps > towards finding a solution. (since it can be done over and over again, > finding out what exactly causes it). So, finding the right sequence of > steps to make it happen again is crucial here. > > Do you think the systemd reload has anything to do with it? Maybe the > whole systemd init-script-wrapper-trickery is misbehaving in some way? > > Can you reproduce this by manually grabbing the > xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch > again, installing them and removing them again? Do you have any other idea? > > Thanks, > Hans > Hi Hans, Thanks for your work with Ian getting xen 4.11 in buster! Very happy with that. I seem to remember seeing 'processing triggers for systemd' somewhere, sometime, but I can't confirm that from the logs so it probably was at another stage of the upgrade. This is a production system, so I will not try to reproduce this by reinstalling the 4.8 packages as you suggested. There is, however an identical server that I still need to upgrade, so I will be able to look closely what happens with that. Here are the relevant lines from dpkg.conf: 2019-07-21 07:38:39 status installed python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:39 remove python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:39 status half-configured python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:39 status half-installed python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status config-files python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status installed libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 remove libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status half-configured libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status half-installed libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status config-files libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status installed xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:40 remove xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 2019-07-21 07:38:41 status installed libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 remove libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status half-configured libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status half-installed libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status config-files libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 2019-07-21 07:38:41 status installed python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 remove python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status half-configured python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status half-installed python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status config-files python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status not-installed python3-distutils:all 2019-07-21 07:38:41 status installed
Bug#932759: [Pkg-xen-devel] Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
Hi niek, Thanks for the report! On 7/22/19 8:32 PM, niek wrote: > Package: xen-hypervisor-4.11-amd64 > Version: 4.11.1+92-g6c33308a8d-2 > > What happened: > - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as > described in > https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html > > - started some Linux pv domu without problems > > - removed obsolete packages with 'apt autoremove'. This removed (among > others) > xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), > libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), > xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) > > [...] > - xenconsoled was not running > > - searching system logs revealed that xenconsoled seemed to have stopped > when 'apt autoremove' removed the obsolete xen 4.8 > packages after upgrading to xen 4.11. Well, there it is again. We tried to make a fix, exactly for this... https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a ...and apparently there's another scenario in which even this doesn't work? Can you show the lines from /var/log/dpkg.log from that moment, the seconds around 07:38:40? It tells exactly what got removed, in what second, just to confirm? I'm pretty sure I tried to reproduce this after we added the fix I just referenced, and I was unable to. So, I'm very interested in finding out what's still going on here. Usually being able to reproduce a problem is one of the biggest steps towards finding a solution. (since it can be done over and over again, finding out what exactly causes it). So, finding the right sequence of steps to make it happen again is crucial here. Do you think the systemd reload has anything to do with it? Maybe the whole systemd init-script-wrapper-trickery is misbehaving in some way? Can you reproduce this by manually grabbing the xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch again, installing them and removing them again? Do you have any other idea? Thanks, Hans
