[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
This bug was fixed in the package systemd - 229-4ubuntu21.29 --- systemd (229-4ubuntu21.29) xenial; urgency=medium [ Dan Streetman ] * d/p/lp1881312/0001-core-don-t-dispatch-load-queue-when-setting-Slice-fo.patch, d/p/lp1881312/0002-run-make-slice-work-in-conjunction-with-scope.patch: - run new scope unit in specified slice (LP: #1881312) https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b5a778c3cd93e3d684602146f281f315f30778a8 * d/p/lp1877176-sd-dhcp-client-validate-hostnames-stricter-7308.patch: - strictly validate hostname sent to dhcp server (LP: #1877176) https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7a1f1c41dcd860fcba6be42a2153cfe4315ebf17 * d/e/rules-ubuntu/40-vm-hotadd.rules: - Hotadd only offline memory and CPUs (LP: #1876018) https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=43357bd260a1c3e7b4951d33cbacaebda1d086b3 * d/p/lp1698388-journal-remote-Ensure-reallocation-of-source-buf-doe.patch: - Fix assertion failure when journald source->filled < source->size (LP: #1698388) https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=d1cc15b88113510227e5a76654e103ef54a8b8f3 [ Heitor Alves de Siqueira ] * d/p/lp1876600-sd-bus-deal-with-cookie-overruns.patch: - deal with dbus cookie overruns (LP: #1876600) https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8dc9ef188944349169044d8974a4ffa55c66be9d -- Heitor Alves de Siqueira Tue, 07 Jul 2020 17:28:11 -0400 ** Changed in: systemd (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Released Status in systemd source package in Bionic: Fix Released Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Validated systemd/229-4ubuntu21.29 from xenial-proposed, according to test case from description: ubuntu@systemd-cookie-xenial:~$ dpkg -l systemd | grep systemd ii systemd229-4ubuntu21.29 amd64system and service manager ubuntu@systemd-cookie-xenial:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@systemd-cookie-xenial:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0x400ddb: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (65s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x603040) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 (65s) cookie: 0x8000reply-cookie: 0x8000 (129s) cookie: 0x8001 reply-cookie: 0x8001 (188s) cookie: 0x8002 reply-cookie: 0x8002 ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Committed Status in systemd source package in Bionic: Fix Released Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. To compile and debug the test
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Hello Heitor, or anyone else affected, Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.29 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags removed: verification-done ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Fix Committed Status in systemd source package in Bionic: Fix Released Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
This bug was fixed in the package systemd - 237-3ubuntu10.41 --- systemd (237-3ubuntu10.41) bionic; urgency=medium [ Dan Streetman ] * d/p/lp1867375/0001-network-Allow-to-configure-GW-even-UseRoutes-false.patch, d/p/lp1867375/0002-network-add-a-flag-to-ignore-gateway-provided-by-DHC.patch, d/p/lp1867375/0003-network-change-UseGateway-default-to-UseRoutes-setti.patch: - Move gateway ignoring from UseRoutes= to UseGateway= (LP: #1867375) * d/p/lp1873607/0002-core-make-sure-to-restore-the-control-command-id-too.patch: - Avoid segfault during serialization (LP: #1873607) * d/p/lp1529152/0001-bash-completion-systemctl-use-systemctl-no-pager.patch, d/p/lp1529152/0002-bash-completion-systemctl-pass-current-partial-unit-.patch, d/p/lp1529152/0003-shell-completion-systemctl-pass-current-word-to-all-.patch, d/p/lp1529152/0004-bash-completion-systemctl-re-implement-__filter_unit.patch, d/p/lp1529152/0005-strip-value-from-property-names.patch: - fix slow systemctl tab completion (LP: #1529152) * d/p/lp1877159-networkd-fix-attribute-length-for-wireguard-10380.patch: - avoid kernel err msg setting wireguard param (LP: #1877159) [ Heitor Alves de Siqueira ] * d/p/lp1876600-sd-bus-deal-with-cookie-overruns.patch: - deal with dbus cookie overruns (LP: #1876600) -- Heitor Alves de Siqueira Sun, 03 May 2020 11:30:25 + ** Changed in: systemd (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: Fix Released Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Validated systemd 237-3ubuntu10.41 from bionic-proposed, according to test case from bug description: ubuntu@systemd-cookie-bionic:~$ dpkg -l systemd | grep systemd ii systemd237-3ubuntu10.41 amd64system and service manager ubuntu@systemd-cookie-bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@systemd-cookie-bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (15s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 (15s) cookie: 0x8000reply-cookie: 0x8000 (29s) cookie: 0x8001reply-cookie: 0x8001 (43s) cookie: 0x8002reply-cookie: 0x8002 ** Tags removed: verification-needed verification-needed-bionic ** Tags added: verification-done verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: Fix Committed Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. To compile and debug the
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Hello Heitor, or anyone else affected, Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.41 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-bionic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: Fix Committed Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
** Tags removed: sts-sponsor ** Tags added: sts-sponsor-ddstreet -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: In Progress Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1876600/+subscriptions -- Mailing list:
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
** Changed in: systemd (Ubuntu Xenial) Status: Confirmed => In Progress ** Changed in: systemd (Ubuntu Bionic) Status: Confirmed => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: In Progress Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests. To manage notifications about this bug go to:
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
** Description changed: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. - u To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: In Progress Status in systemd source package in Bionic: In Progress Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
** Merge proposal linked: https://code.launchpad.net/~halves/ubuntu/+source/systemd/+git/systemd/+merge/383417 ** Merge proposal linked: https://code.launchpad.net/~halves/ubuntu/+source/systemd/+git/systemd/+merge/383418 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Confirmed Status in systemd source package in Bionic: Confirmed Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. u To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests. To
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
Test builds for the proposed merge can be found at the lp1876600 PPA [0]. [0] https://launchpad.net/~halves/+archive/ubuntu/lp1876600 ** Tags added: sts-sponsor -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1876600 Title: cookie overruns can cause org.freedesktop.systemd1 dbus to hang Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Xenial: Confirmed Status in systemd source package in Bionic: Confirmed Bug description: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd systemd | 229-4ubuntu4 | xenial | source, ... systemd | 229-4ubuntu21.27 | xenial-security | source, ... systemd | 229-4ubuntu21.27 | xenial-updates | source, ... systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... systemd | 237-3ubuntu10| bionic | source, ... systemd | 237-3ubuntu10.38 | bionic-security | source, ... systemd | 237-3ubuntu10.39 | bionic-updates | source, ... systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects It's essentially the example code from the sd_bus_message_new_method_call() manpage, with minor modifications: this is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. u To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not handling the cookie reuse properly. In practice, this shouldn't cause serious problems as most dbus messages should not stall long enough for a possible overlap in the 2^31 space. The patch has been present in other stable Ubuntu Series and upstream, and has been validated and tested through the systemd test suite and autopkgtests. To manage notifications about this bug go to:
[Touch-packages] [Bug 1876600] Re: cookie overruns can cause org.freedesktop.systemd1 dbus to hang
** Attachment added: "test.gdb" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1876600/+attachment/5366780/+files/test.gdb ** Description changed: [Impact] Long-running services overflow the sd_bus->cookie counter, causing further communication with org.freedesktop.systemd1 to stall. [Description] Systemd dbus messages include a "cookie" value to uniquely identify them in their bus context. This value is obtained from the bus header, and incremented for each exchanged message in the same bus object. For services that run for longer periods of time and keep communicating through dbus, it's possible to overflow the cookie value, causing further messages to the org.freedesktop.systemd1 dbus to fail. This can lead to these services becoming unresponsive, as they get stuck trying to communicate with invalid bus cookie values. This issue has been fixed upstream by the commit below: - sd-bus: deal with cookie overruns (1f82f5bb4237) $ git describe --contains 1f82f5bb4237 v242-rc1~228 $ rmadison systemd - systemd | 229-4ubuntu4 | xenial | source, ... - systemd | 229-4ubuntu21.27 | xenial-security | source, ... - systemd | 229-4ubuntu21.27 | xenial-updates | source, ... - systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... - systemd | 237-3ubuntu10| bionic | source, ... - systemd | 237-3ubuntu10.38 | bionic-security | source, ... - systemd | 237-3ubuntu10.39 | bionic-updates | source, ... - systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < - systemd | 242-7ubuntu3 | eoan| source, ... + systemd | 229-4ubuntu4 | xenial | source, ... + systemd | 229-4ubuntu21.27 | xenial-security | source, ... + systemd | 229-4ubuntu21.27 | xenial-updates | source, ... + systemd | 229-4ubuntu21.28 | xenial-proposed | source, ... + systemd | 237-3ubuntu10| bionic | source, ... + systemd | 237-3ubuntu10.38 | bionic-security | source, ... + systemd | 237-3ubuntu10.39 | bionic-updates | source, ... + systemd | 237-3ubuntu10.40 | bionic-proposed | source, ... < + systemd | 242-7ubuntu3 | eoan| source, ... Releases starting with Eoan already have this fix. [Test Case] There doesn't seem to be an easy test case for this, as the cookie values start at zero and won't overflow until (1<<32). There have been reports from users hitting this on Kubernetes clusters continuously running for longer periods (~5 months). Using GDB, we can construct an artificial test case to test the cookie overflow. The test case below performs the following steps: 1. Create a new system bus object through sd_bus_default_system() 2. Allocate and append a new method_call message to the bus 3. Send the message through sd_bus_call() 4. Handle the response message and free up the message objects - This is done continuously, to keep incrementing the bus cookie value. We step in with GDB when it reaches 0x1, and set its value to 0xff00 which then causes the test program to fail shortly afterwards. An example test run of an impacted system: + It's essentially the example code from the + sd_bus_message_new_method_call() manpage, with minor modifications: this + is done continuously, to keep incrementing the bus cookie value. We step + in with GDB when it reaches 0x1, and set its value to 0xff00 + which then causes the test program to fail shortly afterwards. An + example test run of an impacted system: + ubuntu@bionic:~$ gcc -Wall test.c -o cookie -lsystemd -g ubuntu@bionic:~$ gdb --batch --command=test.gdb --args ./cookie Breakpoint 1 at 0xe61: file test.c, line 38. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". (16s) cookie: 0x0001reply-cookie: 0x0001 Breakpoint 1, print_unit_path (bus=0x55757290) at test.c:38 38 r = sd_bus_message_new_method_call(bus, , $1 = 0x1 $2 = 0xff00 Call failed: Operation not supported Sleeping and retrying... Call failed: Invalid argument Assertion 'm->n_ref > 0' failed at ../src/libsystemd/sd-bus/bus-message.c:934, function sd_bus_message_unref(). Aborting. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=0x6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. u To compile and debug the test case above, libsystemd-dev and libsystemd0-dbgsym are required. Both test.c and test.gdb source code are attached to this LP bug. [Regression Potential] This fix introduces some changes in the way cookie incrementation is handled. We now have a reduced number of available values, since the patch makes use of a high order bit to indicate whether we have overflowed or not. Potential issues could arise from two distinct messages repeating the cookie value, or from us not
