** Also affects: systemd via https://github.com/systemd/systemd/issues/17577 Importance: Unknown Status: Unknown
** Description changed: + [impact] + + networking changes, like RA events, can cause systemd-resolved to re- + write the resolv.conf file, even if the contents didn't change, + resulting in unnecessary increased amount of inotify events + + [test case] + + see original description + + [regression potential] + + regressions would result in incorrect or missing data in the resolv.conf + file, possibly resulting in dns failures or errors + + [scope] + + TBD + + [original description] + # Issue description: On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?. The log noise is the observable problem that would be nice to see addressed: - root@lxd02:~# uptime - 17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05 - root@lxd02:~# journalctl -b0 | grep -cF dnsmasq - 158609 + root@lxd02:~# uptime + 17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05 + root@lxd02:~# journalctl -b0 | grep -cF dnsmasq + 158609 Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received. 1) One can observe that by setting systemd-resolved's service in debug mode: $ sudo systemctl edit systemd-resolved and in the editor that is opened, add and save this content: [Service] Environment=SYSTEMD_LOG_LEVEL=debug then restart systemd-resolved and watch the logs scroll by with: $ journalctl -fu systemd-resolved 3) In another terminal, watch the files be recreated with: watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf 3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by: sudo tcpdump -ni eth0 icmp6 + You will see that every time a RA packet comes in, resolved's journal + will log this: - You will see that every time a RA packet comes in, resolved's journal will log this: - - Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message + Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error- message=n/a And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file. - # Additional information: root@lxd02:~# lsb_release -rd Description: Ubuntu 20.04.1 LTS Release: 20.04 root@lxd02:~# apt-cache policy systemd systemd: - Installed: 245.4-4ubuntu3.2 - Candidate: 245.4-4ubuntu3.2 - Version table: - *** 245.4-4ubuntu3.2 500 - 500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages - 100 /var/lib/dpkg/status - 245.4-4ubuntu3 500 - 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages + Installed: 245.4-4ubuntu3.2 + Candidate: 245.4-4ubuntu3.2 + Version table: + *** 245.4-4ubuntu3.2 500 + 500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages + 100 /var/lib/dpkg/status + 245.4-4ubuntu3 500 + 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages root@lxd02:~# uname -a Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ** Changed in: systemd (Ubuntu) Status: Incomplete => In Progress ** Also affects: systemd (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: systemd (Ubuntu Groovy) Importance: Undecided Status: New ** Description changed: [impact] networking changes, like RA events, can cause systemd-resolved to re- write the resolv.conf file, even if the contents didn't change, resulting in unnecessary increased amount of inotify events [test case] see original description [regression potential] regressions would result in incorrect or missing data in the resolv.conf file, possibly resulting in dns failures or errors [scope] - TBD + this is needed for h and eralier + + this is (potentially) fixed upstream by + f3e1f00d03445911ee73729219cea88c8a70c612 which in first included in + v248, so this is needed in hirsute and earlier [original description] # Issue description: On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?. The log noise is the observable problem that would be nice to see addressed: root@lxd02:~# uptime 17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05 root@lxd02:~# journalctl -b0 | grep -cF dnsmasq 158609 Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received. 1) One can observe that by setting systemd-resolved's service in debug mode: $ sudo systemctl edit systemd-resolved and in the editor that is opened, add and save this content: [Service] Environment=SYSTEMD_LOG_LEVEL=debug then restart systemd-resolved and watch the logs scroll by with: $ journalctl -fu systemd-resolved 3) In another terminal, watch the files be recreated with: watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf 3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by: sudo tcpdump -ni eth0 icmp6 You will see that every time a RA packet comes in, resolved's journal will log this: Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error- message=n/a And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file. # Additional information: root@lxd02:~# lsb_release -rd Description: Ubuntu 20.04.1 LTS Release: 20.04 root@lxd02:~# apt-cache policy systemd systemd: Installed: 245.4-4ubuntu3.2 Candidate: 245.4-4ubuntu3.2 Version table: *** 245.4-4ubuntu3.2 500 500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 245.4-4ubuntu3 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages root@lxd02:~# uname -a Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ** Changed in: systemd (Ubuntu) Status: In Progress => Fix Released ** Changed in: systemd (Ubuntu Focal) Status: New => In Progress ** Changed in: systemd (Ubuntu Groovy) Status: New => In Progress ** Changed in: systemd (Ubuntu Hirsute) Status: New => In Progress ** Changed in: systemd (Ubuntu Hirsute) Importance: Undecided => Wishlist ** Changed in: systemd (Ubuntu Groovy) Importance: Undecided => Wishlist ** Changed in: systemd (Ubuntu Focal) Importance: Undecided => Wishlist ** Changed in: systemd (Ubuntu Hirsute) Assignee: (unassigned) => Dan Streetman (ddstreet) ** Changed in: systemd (Ubuntu Groovy) Assignee: (unassigned) => Dan Streetman (ddstreet) ** Changed in: systemd (Ubuntu Focal) Assignee: (unassigned) => Dan Streetman (ddstreet) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1891215 Title: systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for every IPv6 RA received To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1891215/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
