Bug#934992: network-manager breaks systemd autopkgtest
Hi Michael, You already got a bit further, but nevertheless. On 18-08-2019 18:39, Michael Biebl wrote: > Paul, which lxc version was used to run the autopkgtest tests? We are currently running stretch on 11/12 of the workers and buster on the other one (number 12). > Is it possible to get access to a system with the exact same > configuration as on ci.debian.net so one can interactively debug this issue? Is this still needed? It can be done of course. Paul FYI, we filed a regression bug against lxc the other day, so yes, there is an issue with the latest one in sid/bullseye: https://bugs.debian.org/934983 lxc: lxc-attach loses exit code of executed command signature.asc Description: OpenPGP digital signature
Bug#934992: network-manager breaks systemd autopkgtest
Control: reassign -1 systemd 241-7 Am 18.08.19 um 18:39 schrieb Michael Biebl: > Am 18.08.19 um 17:18 schrieb Michael Biebl: >> I've created an LXC container via >> autopkgtest-build-lxc debian bullseye >> and got the same autopkgtest failure using this container. >> So this looks like a regression that was introduced outside of >> systemd/network-manager >> >> I've also tried a qemu backed autopkgtest run and there the tests >> completed successfully. >> Might this be a regression introduced by LXC itself (and or in >> combination with a newer kernel?) > > My test system is running unstable, i.e. I was using > lxc_1:3.1.0+really3.0.4-1 > After downgrading lxc to 1:3.1.0+really3.0.3-8, boot-smoke did > successfully pass again. > > Paul, which lxc version was used to run the autopkgtest tests? > > Is it possible to get access to a system with the exact same > configuration as on ci.debian.net so one can interactively debug this issue? Dear lxc maintainers, sorry for the noise. Afaics this is actually a bug in the boot-smoke autopkgtest and not related to lxc, so tentatively reassigning to systemd. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#934992: [Pkg-utopia-maintainers] Bug#934992: network-manager breaks systemd autopkgtest
Am 18.08.19 um 13:01 schrieb Andreas Henriksson: > So apparently polkitd is not running after the first reboot attempt. I > have no clue why or how to investigate why so I guess I'll leave that to > the professionals. > > Maybe it would be useful to change the test to do > "pidof polkitd || fail" to gather more autopkgtest artifacts information > for debugging purposes? i.e. the boot-smoke-journal.txt file. Hm, I have to say I don't quite get the check for "pidof polkitd" either. polkit.service is a dbus activated service, i.e. it is started on demand and not statically during boot. So boot-smoke was relying on "something" triggering the start of polkit.service. Seems that older network manager versions did trigger the start of polkit.service, so this test only worked "by accident". See the attached bullseye.txt and sid.txt which confirm this suspicion. Martin, do you remember why you chose to check for "pidof polkitd"? This seems rather unreliable when we rely on "some other component" to trigger it's start. Shouldn't we rather check for a service which is installed by the test and hooked into multi-user.target, like "pidof NetworkManager"? Afaics, this looks like a bug in boot-smoke to me for making wrong assumption. I would like Martin's feedback on this though. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? root@autopkgtest-bullseye:/tmp/autopkgtest-lxc.qoaowm2l/downtmp/build.Ira/src# systemctl status polkit.service NetworkManager.service * polkit.service - Authorization Manager Loaded: loaded (/lib/systemd/system/polkit.service; static; vendor preset: enabled) Active: inactive (dead) Docs: man:polkit(8) * NetworkManager.service - Network Manager Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sun 2019-08-18 17:16:00 UTC; 11s ago Docs: man:NetworkManager(8) Process: 63 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=exited, status=0/SUCCESS) Main PID: 63 (code=exited, status=0/SUCCESS) root@autopkgtest-bullseye:/tmp/autopkgtest-lxc.qoaowm2l/downtmp/build.Ira/src# systemctl start NetworkManager.service root@autopkgtest-bullseye:/tmp/autopkgtest-lxc.qoaowm2l/downtmp/build.Ira/src# systemctl status polkit.service NetworkManager.service * polkit.service - Authorization Manager Loaded: loaded (/lib/systemd/system/polkit.service; static; vendor preset: enabled) Active: active (running) since Sun 2019-08-18 17:16:25 UTC; 1s ago Docs: man:polkit(8) Main PID: 225 (polkitd) Tasks: 4 (limit: 4915) Memory: 3.4M CGroup: /system.slice/polkit.service `-225 /usr/lib/policykit-1/polkitd --no-debug Aug 18 17:16:25 autopkgtest-bullseye systemd[1]: Starting Authorization Manager... Aug 18 17:16:25 autopkgtest-bullseye polkitd[225]: started daemon version 0.105 using authority implementation `local' version `0.105' Aug 18 17:16:25 autopkgtest-bullseye systemd[1]: Started Authorization Manager. * NetworkManager.service - Network Manager Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2019-08-18 17:16:24 UTC; 1s ago Docs: man:NetworkManager(8) Main PID: 215 (NetworkManager) Tasks: 4 (limit: 4915) Memory: 2.9M CGroup: /system.slice/NetworkManager.service `-215 /usr/sbin/NetworkManager --no-daemon Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4084] Loaded device plugin: NMAtmManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1.1 Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4101] Loaded device plugin: NMWwanFactory (/usr/lib/x86_64-linux-gnu/NetworkManager/1. Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4106] Loaded device plugin: NMBluezManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1 Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4123] Loaded device plugin: NMTeamFactory (/usr/lib/x86_64-linux-gnu/NetworkManager/1. Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4133] device (lo): carrier: link connected Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4137] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1) Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4144] device (eth0): carrier: link connected Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4147] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2) Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4154] manager: NetworkManager state is now CONNECTED_GLOBAL Aug 18 17:16:25 autopkgtest-bullseye NetworkManager[215]: [1566148585.4196] manager: startup complete root@autopkgtest-bullseye:/tmp/autopkgtest-lxc.qoaowm2l/downtmp/build.Ira/src# apt-cache
Bug#934992: network-manager breaks systemd autopkgtest
Am 18.08.19 um 17:18 schrieb Michael Biebl: > Am 17.08.19 um 21:23 schrieb Paul Gevers: >> Source: network-manager, systemd >> Control: found -1 network-manager/1.20.0-1 >> Control: found -1 network-manager/1.19.90-2 >> Control: found -1 systemd/241-7 >> Severity: serious >> X-Debbugs-CC: debian...@lists.debian.org >> User: debian...@lists.debian.org >> Usertags: breaks needs-update >> >> Dear maintainers, >> >> With a recent upload of network-manager the autopkgtest of systemd fails >> in testing when that autopkgtest is run with the binary packages of >> network-manager from unstable. It passes when run with only packages >> from testing. In tabular form: >>passfail >> network-managerfrom testing1.20.0-1 >> systemdfrom testing241-7 >> versioned deps [0] from testingfrom unstable >> all others from testingfrom testing >> >> I copied some of the output at the bottom of this report. (It doesn't >> show anything that isn't there in a successful run, normally it continues). >> >> Currently this regression is blocking the migration of network-manager >> to testing [1]. Due to the nature of this issue, I filed this bug report >> against both packages. Can you please investigate the situation and >> reassign the bug to the right package? >> >> More information about this bug and the reason for filing it can be found on >> https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformation >> >> Paul >> >> [0] You can see what packages were added from the second line of the log >> file quoted below. The migration software adds source package from >> unstable to the list if they are needed to install packages from >> network-manager/1.20.0-1. I.e. due to versioned dependencies or >> breaks/conflicts. >> [1] https://qa.debian.org/excuses.php?package=network-manager >> >> https://ci.debian.net/data/autopkgtest/testing/amd64/s/systemd/2754701/log.gz >> >> autopkgtest [12:20:19]: test boot-smoke: [--- >> reboot #0 >> bash: line 1: 6405 Killed >> /tmp/autopkgtest-lxc.d0a1dm24/downtmp/build.Yzc/src/debian/tests/boot-smoke >> 2> >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stderr >&2) (tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stdout) >> autopkgtest [12:20:20]: test process requested reboot with marker 1 >> checking for failed unmounts for user systemd >> checking for connection timeouts >> checking that polkitd runs >> autopkgtest [12:20:31]: test boot-smoke: ---] >> > > I've created an LXC container via > autopkgtest-build-lxc debian bullseye > and got the same autopkgtest failure using this container. > So this looks like a regression that was introduced outside of > systemd/network-manager > > I've also tried a qemu backed autopkgtest run and there the tests > completed successfully. > Might this be a regression introduced by LXC itself (and or in > combination with a newer kernel?) My test system is running unstable, i.e. I was using lxc_1:3.1.0+really3.0.4-1 After downgrading lxc to 1:3.1.0+really3.0.3-8, boot-smoke did successfully pass again. Paul, which lxc version was used to run the autopkgtest tests? Is it possible to get access to a system with the exact same configuration as on ci.debian.net so one can interactively debug this issue? -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#934992: network-manager breaks systemd autopkgtest
Am 17.08.19 um 21:23 schrieb Paul Gevers: > Source: network-manager, systemd > Control: found -1 network-manager/1.20.0-1 > Control: found -1 network-manager/1.19.90-2 > Control: found -1 systemd/241-7 > Severity: serious > X-Debbugs-CC: debian...@lists.debian.org > User: debian...@lists.debian.org > Usertags: breaks needs-update > > Dear maintainers, > > With a recent upload of network-manager the autopkgtest of systemd fails > in testing when that autopkgtest is run with the binary packages of > network-manager from unstable. It passes when run with only packages > from testing. In tabular form: >passfail > network-managerfrom testing1.20.0-1 > systemdfrom testing241-7 > versioned deps [0] from testingfrom unstable > all others from testingfrom testing > > I copied some of the output at the bottom of this report. (It doesn't > show anything that isn't there in a successful run, normally it continues). > > Currently this regression is blocking the migration of network-manager > to testing [1]. Due to the nature of this issue, I filed this bug report > against both packages. Can you please investigate the situation and > reassign the bug to the right package? > > More information about this bug and the reason for filing it can be found on > https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformation > > Paul > > [0] You can see what packages were added from the second line of the log > file quoted below. The migration software adds source package from > unstable to the list if they are needed to install packages from > network-manager/1.20.0-1. I.e. due to versioned dependencies or > breaks/conflicts. > [1] https://qa.debian.org/excuses.php?package=network-manager > > https://ci.debian.net/data/autopkgtest/testing/amd64/s/systemd/2754701/log.gz > > autopkgtest [12:20:19]: test boot-smoke: [--- > reboot #0 > bash: line 1: 6405 Killed > /tmp/autopkgtest-lxc.d0a1dm24/downtmp/build.Yzc/src/debian/tests/boot-smoke > 2> >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stderr >&2) >>> (tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stdout) > autopkgtest [12:20:20]: test process requested reboot with marker 1 > checking for failed unmounts for user systemd > checking for connection timeouts > checking that polkitd runs > autopkgtest [12:20:31]: test boot-smoke: ---] > I've created an LXC container via autopkgtest-build-lxc debian bullseye and got the same autopkgtest failure using this container. So this looks like a regression that was introduced outside of systemd/network-manager I've also tried a qemu backed autopkgtest run and there the tests completed successfully. Might this be a regression introduced by LXC itself (and or in combination with a newer kernel?) Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#934992: network-manager breaks systemd autopkgtest
Hi, I had a quick look and hoping my incomplete findings might be helpful... On Sat, Aug 17, 2019 at 09:23:17PM +0200, Paul Gevers wrote: [...] > With a recent upload of network-manager the autopkgtest of systemd fails > in testing when that autopkgtest is run with the binary packages of > network-manager from unstable. [...] >From the autopkgtest log: << autopkgtest [12:20:19]: test boot-smoke: [--- reboot #0 bash: line 1: 6405 Killed /tmp/autopkgtest-lxc.d0a1dm24/downtmp/build.Yzc/src/debian/tests/boot-smoke 2> >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stderr >&2) > >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stdout) autopkgtest [12:20:20]: test process requested reboot with marker 1 checking for failed unmounts for user systemd checking for connection timeouts checking that polkitd runs autopkgtest [12:20:31]: test boot-smoke: ---] autopkgtest [12:20:32]: test boot-smoke: - - - - - - - - - - results - - - - - - - - - - boot-smoke FAIL non-zero exit status 1 << Given the above above output I assume this is where we bail out (note: script is 'set -e'): https://sources.debian.org/src/systemd/241-7/debian/tests/boot-smoke/#L51 (Otherwise we should have seen the "checking that there are no running jobs" output as well.) So apparently polkitd is not running after the first reboot attempt. I have no clue why or how to investigate why so I guess I'll leave that to the professionals. Maybe it would be useful to change the test to do "pidof polkitd || fail" to gather more autopkgtest artifacts information for debugging purposes? i.e. the boot-smoke-journal.txt file. HTH Regards, Andreas Henriksson
Bug#934992: network-manager breaks systemd autopkgtest
Source: network-manager, systemd Control: found -1 network-manager/1.20.0-1 Control: found -1 network-manager/1.19.90-2 Control: found -1 systemd/241-7 Severity: serious X-Debbugs-CC: debian...@lists.debian.org User: debian...@lists.debian.org Usertags: breaks needs-update Dear maintainers, With a recent upload of network-manager the autopkgtest of systemd fails in testing when that autopkgtest is run with the binary packages of network-manager from unstable. It passes when run with only packages from testing. In tabular form: passfail network-managerfrom testing1.20.0-1 systemdfrom testing241-7 versioned deps [0] from testingfrom unstable all others from testingfrom testing I copied some of the output at the bottom of this report. (It doesn't show anything that isn't there in a successful run, normally it continues). Currently this regression is blocking the migration of network-manager to testing [1]. Due to the nature of this issue, I filed this bug report against both packages. Can you please investigate the situation and reassign the bug to the right package? More information about this bug and the reason for filing it can be found on https://wiki.debian.org/ContinuousIntegration/RegressionEmailInformation Paul [0] You can see what packages were added from the second line of the log file quoted below. The migration software adds source package from unstable to the list if they are needed to install packages from network-manager/1.20.0-1. I.e. due to versioned dependencies or breaks/conflicts. [1] https://qa.debian.org/excuses.php?package=network-manager https://ci.debian.net/data/autopkgtest/testing/amd64/s/systemd/2754701/log.gz autopkgtest [12:20:19]: test boot-smoke: [--- reboot #0 bash: line 1: 6405 Killed /tmp/autopkgtest-lxc.d0a1dm24/downtmp/build.Yzc/src/debian/tests/boot-smoke 2> >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stderr >&2) > >(tee -a /tmp/autopkgtest-lxc.d0a1dm24/downtmp/boot-smoke-stdout) autopkgtest [12:20:20]: test process requested reboot with marker 1 checking for failed unmounts for user systemd checking for connection timeouts checking that polkitd runs autopkgtest [12:20:31]: test boot-smoke: ---] signature.asc Description: OpenPGP digital signature
