Public bug reported: # Issue description:
On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?. The log noise is the observable problem that would be nice to see addressed: root@lxd02:~# uptime 17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05 root@lxd02:~# journalctl -b0 | grep -cF dnsmasq 158609 Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received. 1) One can observe that by setting systemd-resolved's service in debug mode: $ sudo systemctl edit systemd-resolved and in the editor that is opened, add and save this content: [Service] Environment=SYSTEMD_LOG_LEVEL=debug then restart systemd-resolved and watch the logs scroll by with: $ journalctl -fu systemd-resolved 3) In another terminal, watch the files be recreated with: watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf 3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by: sudo tcpdump -ni eth0 icmp6 You will see that every time a RA packet comes in, resolved's journal will log this: Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error- message=n/a And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file. # Additional information: root@lxd02:~# lsb_release -rd Description: Ubuntu 20.04.1 LTS Release: 20.04 root@lxd02:~# apt-cache policy systemd systemd: Installed: 245.4-4ubuntu3.2 Candidate: 245.4-4ubuntu3.2 Version table: *** 245.4-4ubuntu3.2 500 500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 245.4-4ubuntu3 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages root@lxd02:~# uname -a Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ** Affects: systemd (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1891215 Title: systemd-resolved re-creates /run/systemd/resolve/*resolv.conf for every IPv6 RA received Status in systemd package in Ubuntu: New Bug description: # Issue description: On 2 Linode VMs that are used as lxd hosts, we noticed that /run/systemd/resolve/*resolv.conf were re-created quite frequently (~ once per second). We noticed because of the log noise from lxd's dnsmasq instance using inotify to watch the target of /etc/resolv.conf (which points to the stub-resolv.conf in our case). This was (wrongly) reported as a lxd bug (https://github.com/lxc/lxd/issues/7765) until it became apparent it was more likely to be a problem with systemd(-resolved)?. The log noise is the observable problem that would be nice to see addressed: root@lxd02:~# uptime 17:55:48 up 9:52, 1 user, load average: 0.18, 0.11, 0.05 root@lxd02:~# journalctl -b0 | grep -cF dnsmasq 158609 Upon further investigation, it seems that systemd-resolved re-creates the resolv.conf and stub-resolv.conf files whenever an IPv6 RA is received. 1) One can observe that by setting systemd-resolved's service in debug mode: $ sudo systemctl edit systemd-resolved and in the editor that is opened, add and save this content: [Service] Environment=SYSTEMD_LOG_LEVEL=debug then restart systemd-resolved and watch the logs scroll by with: $ journalctl -fu systemd-resolved 3) In another terminal, watch the files be recreated with: watch -d -n 0.1 stat /run/systemd/resolve/stub-resolv.conf 3) In yet another terminal, run a packet capture and watch "ICMP6, router advertisement" messages come by: sudo tcpdump -ni eth0 icmp6 You will see that every time a RA packet comes in, resolved's journal will log this: Aug 11 17:33:55 lxd02 systemd-resolved[15368]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=244 reply_cookie=0 signature=sa{sv}as error-name=n/a error- message=n/a And the stat monitoring terminal will blink to highlight the new inode and timestamps of the freshly replaced stub-resolv.conf file. # Additional information: root@lxd02:~# lsb_release -rd Description: Ubuntu 20.04.1 LTS Release: 20.04 root@lxd02:~# apt-cache policy systemd systemd: Installed: 245.4-4ubuntu3.2 Candidate: 245.4-4ubuntu3.2 Version table: *** 245.4-4ubuntu3.2 500 500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 245.4-4ubuntu3 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages root@lxd02:~# uname -a Linux lxd01 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1891215/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
