[Touch-packages] [Bug 1850009] Re: Issue with detecting new state in dhclient-enter-hooks.d/resolved
This bug is also being handled in #1805183. Comment #9 has a much better patch (along the same lines as mine but also captures STDERR from md5sum). Though it doesn't have Eoan included in the list of affected versions. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1850009 Title: Issue with detecting new state in dhclient-enter-hooks.d/resolved Status in systemd package in Ubuntu: Confirmed Bug description: After upgrading to 19.10 I noticed that resolving hostnames stopped working. Quick release, which package and version info: root@caret:~# lsb_release -r Release:19.10 root@caret:~# dpkg -S /etc/dhcp/dhclient-enter-hooks.d/resolved systemd: /etc/dhcp/dhclient-enter-hooks.d/resolved root@caret:~# apt-cache policy systemd systemd: Installed: 242-7ubuntu3 Candidate: 242-7ubuntu3 Version table: *** 242-7ubuntu3 500 500 http://au.archive.ubuntu.com/ubuntu eoan/main amd64 Packages 100 /var/lib/dpkg/status The file /etc/resolv.conf was pointing to the local instance of systemd-resolved however there was no search line, indicating something wasn't quite right. Restarting the systemd-resolved service did recreate /etc/resolv.conf with expected search line. After much debugging, I've worked out what was going wrong. The dhcp hook file /etc/dhcp/dhclient-enter-hooks.d/resolved has a bug in the resolver state change logic. This may also impact on tickets 1805183, 1832050, 1832053 and maybe others. When doing the md5sum of the old and new state, the command is wrong. It isn't sending STDOUT (and optionally STDERR) to the temp file but rather putting the md5sum command into the background and then creating an empty file for both states. Which obviously isn't different, so systemd-resolved doesn't restart. Quick diff of a working version (I'm not capturing STDERR but one can with "2>&1") === BEGIN PATCH === *** .brk-resolved-20191027 2019-10-27 11:19:56.210069032 +1030 --- resolved2019-10-27 11:30:17.156381734 +1030 *** *** 30,36 mkdir -p $statedir oldstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $oldstate if [ -n "$new_domain_name_servers" ] ; then cat <$statedir/isc-dhcp-v4-$interface.conf [Resolve] --- 30,36 mkdir -p $statedir oldstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $oldstate if [ -n "$new_domain_name_servers" ] ; then cat <$statedir/isc-dhcp-v4-$interface.conf [Resolve] *** *** 55,61 fi newstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $newstate if ! cmp $oldstate $newstate; then systemctl try-reload-or-restart systemd-resolved.service fi --- 55,61 fi newstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $newstate if ! cmp $oldstate $newstate; then systemctl try-reload-or-restart systemd-resolved.service fi === END PATCH === Also, the script removes the script removes the $oldstate file but doesn't remove the $newstate file. Not sure how this made it through QA testing... But breaking DHCP client shouldn't happen like this. ivan. ProblemType: Bug DistroRelease: Ubuntu 19.10 Package: network-manager 1.20.4-2ubuntu2 ProcVersionSignature: Ubuntu 5.3.0-19.20-generic 5.3.1 Uname: Linux 5.3.0-19-generic x86_64 ApportVersion: 2.20.11-0ubuntu8 Architecture: amd64 Date: Sun Oct 27 11:48:33 2019 InstallationDate: Installed on 2017-04-21 (918 days ago) InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412) IpRoute: default via 172.16.1.14 dev enp0s5 150.101.89.32/28 dev enp0s5 scope link 169.254.0.0/16 dev enp0s5 scope link metric 1000 172.16.1.0/24 dev enp0s5 proto kernel scope link src 172.16.1.120 IwConfig: lono wireless extensions. enp0s5no wireless extensions. NetworkManager.state: [main] NetworkingEnabled=false WirelessEnabled=true WWANEnabled=true ProcEnviron: LANGUAGE=en_AU:en TERM=vt100 PATH=(custom, no user) LANG=en_AU.UTF-8 SHELL=/bin/bash RfKill: SourcePackage: network-manager UpgradeStatus: Upgraded to eoan on 2019-10-26 (0 days ago) nmcli-con: nmcli-dev: DEVICE TYPE STATE
[Touch-packages] [Bug 1805183] Re: systemd-resolved constantly restarts on Bionic upgraded from Xenial
I've found the patch applied to Eoan was the broken version where it used the bash'ism instead of the much more basic (and more portable) sh'ism of redirecting both STDOUT and STDERR. I have confirmed this behaviour in Eoan (see bug #1850009) after doing an upgrade to this release. I did a quick fix patch that is basically the same as #9 (I redirect only STDOUT however where as #9 redirects both STDOUT and STDERR). -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1805183 Title: systemd-resolved constantly restarts on Bionic upgraded from Xenial Status in systemd package in Ubuntu: Triaged Status in systemd source package in Bionic: In Progress Status in systemd source package in Cosmic: Won't Fix Status in systemd source package in Disco: Confirmed Bug description: [Impact] Log noise due to needless restart of resolved on lease expiry, maybe loss of cached state? Application that require Name Resolution may fail while the service is being unnecessarily restarted [Test case] (1) Append make_resolv_conf to the end of the file, so it gets executed (2) Execute the file with bash -x and different settings and ensure there are no restarts if the settings are the same, and that there are if settings change; for example: sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook => no restart sudo new_domain_name_servers=8.8.8.8 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook => should restart sudo new_domain_name_servers=8.8.8.8 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook => no restart sudo new_domain_name_servers=8.8.4.4 interface="wlp61s0" reason=REBIND bash -x debian/extra/dhclient-enter-resolved-hook => should restart [Regression potential] The change only restarts resolved when the settings change. If there's a bug in the logic, resolved might not be restarted when it should be. Also, since there will be less restarts of resolved, it will run longer, so if there are memory leaks they will become more apparent. [Original bug report] If a cloud server is upgraded from Xenial to Bionic, the dhclient system remains in place and any DHCP lease refreshes cause a needless restart of the system-resolved daemon Nov 26 16:59:41 srv-qvjhx dhclient[825]: DHCPREQUEST of 10.226.209.106 on ens3 to 10.226.209.105 port 67 (xid=0x2bd41d7d) Nov 26 16:59:41 srv-qvjhx dhclient[825]: DHCPACK of 10.226.209.106 from 10.226.209.105 Nov 26 16:59:41 srv-qvjhx systemd[1]: Stopping Network Name Resolution... Nov 26 16:59:41 srv-qvjhx systemd[1]: Stopped Network Name Resolution. Nov 26 16:59:41 srv-qvjhx systemd[1]: Starting Network Name Resolution... Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Positive Trust Anchors: Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5 Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 1 Nov 26 16:59:41 srv-qvjhx systemd-resolved[1609]: Using system hostname 'srv-qvjhx'. Nov 26 16:59:41 srv-qvjhx systemd[1]: Started Network Name Resolution. Nov 26 16:59:41 srv-qvjhx systemd[1]: Starting resolvconf-pull-resolved.service... Nov 26 16:59:41 srv-qvjhx dhclient[825]: bound to 10.226.209.106 -- renewal in 1466 seconds. Nov 26 16:59:41 srv-qvjhx systemd[1]: Started resolvconf-pull-resolved.service. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: ubuntu-release-upgrader-core 1:16.04.25 ProcVersionSignature: Ubuntu 4.4.0-139.165-generic 4.4.160 Uname: Linux 4.4.0-139-generic x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 CrashDB: ubuntu Date: Mon Nov 26 16:17:52 2018 PackageArchitecture: all SourcePackage: ubuntu-release-upgrader UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1805183/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1850009] [NEW] Issue with detecting new state in dhclient-enter-hooks.d/resolved
Public bug reported: After upgrading to 19.10 I noticed that resolving hostnames stopped working. Quick release, which package and version info: root@caret:~# lsb_release -r Release:19.10 root@caret:~# dpkg -S /etc/dhcp/dhclient-enter-hooks.d/resolved systemd: /etc/dhcp/dhclient-enter-hooks.d/resolved root@caret:~# apt-cache policy systemd systemd: Installed: 242-7ubuntu3 Candidate: 242-7ubuntu3 Version table: *** 242-7ubuntu3 500 500 http://au.archive.ubuntu.com/ubuntu eoan/main amd64 Packages 100 /var/lib/dpkg/status The file /etc/resolv.conf was pointing to the local instance of systemd-resolved however there was no search line, indicating something wasn't quite right. Restarting the systemd-resolved service did recreate /etc/resolv.conf with expected search line. After much debugging, I've worked out what was going wrong. The dhcp hook file /etc/dhcp/dhclient-enter-hooks.d/resolved has a bug in the resolver state change logic. This may also impact on tickets 1805183, 1832050, 1832053 and maybe others. When doing the md5sum of the old and new state, the command is wrong. It isn't sending STDOUT (and optionally STDERR) to the temp file but rather putting the md5sum command into the background and then creating an empty file for both states. Which obviously isn't different, so systemd- resolved doesn't restart. Quick diff of a working version (I'm not capturing STDERR but one can with "2>&1") === BEGIN PATCH === *** .brk-resolved-20191027 2019-10-27 11:19:56.210069032 +1030 --- resolved2019-10-27 11:30:17.156381734 +1030 *** *** 30,36 mkdir -p $statedir oldstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $oldstate if [ -n "$new_domain_name_servers" ] ; then cat <$statedir/isc-dhcp-v4-$interface.conf [Resolve] --- 30,36 mkdir -p $statedir oldstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $oldstate if [ -n "$new_domain_name_servers" ] ; then cat <$statedir/isc-dhcp-v4-$interface.conf [Resolve] *** *** 55,61 fi newstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf &> $newstate if ! cmp $oldstate $newstate; then systemctl try-reload-or-restart systemd-resolved.service fi --- 55,61 fi newstate="$(mktemp)" ! md5sum $statedir/isc-dhcp-v4-$interface.conf $statedir/isc-dhcp-v6-$interface.conf > $newstate if ! cmp $oldstate $newstate; then systemctl try-reload-or-restart systemd-resolved.service fi === END PATCH === Also, the script removes the script removes the $oldstate file but doesn't remove the $newstate file. Not sure how this made it through QA testing... But breaking DHCP client shouldn't happen like this. ivan. ProblemType: Bug DistroRelease: Ubuntu 19.10 Package: network-manager 1.20.4-2ubuntu2 ProcVersionSignature: Ubuntu 5.3.0-19.20-generic 5.3.1 Uname: Linux 5.3.0-19-generic x86_64 ApportVersion: 2.20.11-0ubuntu8 Architecture: amd64 Date: Sun Oct 27 11:48:33 2019 InstallationDate: Installed on 2017-04-21 (918 days ago) InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412) IpRoute: default via 172.16.1.14 dev enp0s5 150.101.89.32/28 dev enp0s5 scope link 169.254.0.0/16 dev enp0s5 scope link metric 1000 172.16.1.0/24 dev enp0s5 proto kernel scope link src 172.16.1.120 IwConfig: lono wireless extensions. enp0s5no wireless extensions. NetworkManager.state: [main] NetworkingEnabled=false WirelessEnabled=true WWANEnabled=true ProcEnviron: LANGUAGE=en_AU:en TERM=vt100 PATH=(custom, no user) LANG=en_AU.UTF-8 SHELL=/bin/bash RfKill: SourcePackage: network-manager UpgradeStatus: Upgraded to eoan on 2019-10-26 (0 days ago) nmcli-con: nmcli-dev: DEVICE TYPE STATE IP4-CONNECTIVITY IP6-CONNECTIVITY DBUS-PATH CONNECTION CON-UUID CON-PATH enp0s5 ethernet unmanaged unknown unknown /org/freedesktop/NetworkManager/Devices/2 -- ---- lo loopback unmanaged unknown unknown /org/freedesktop/NetworkManager/Devices/1 -- ---- nmcli-nm: RUNNING VERSION STATE STARTUP CONNECTIVITY NETWORKING WIFI-HW WIFI WWAN-HW WWAN running 1.20.4 connected started unknown disabledenabled enabled enabled enabled ** Affects: systemd (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug eoan ** Package changed:
