[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Attached is a debdiff for systemd on Bionic which fixes this bug. ** Description changed: - Two servers today that updated systemd to "systemd 237-3ubuntu10.54" - https://ubuntu.com/security/notices/USN-5583-1 + [Impact] - could not resolve dns anymore. - no dns servers, normally set through dhcp. + A widespread outage was caused on Azure instances earlier today, when + systemd 237-3ubuntu10.54 was published to the bionic-security pocket. + Instances could no longer resolve DNS queries, breaking networking. - Ubuntu 18.04 + For affected users, the following workarounds are available. Use whatever is most convenient. + - Reboot your instances + - or - + - Issue "udevadm trigger -cadd -yeth0 && systemctl restart systemd-networkd" as root - Temp fix. - 1. Edit /etc/systemd/resolved.conf - 1. Add/Uncomment # FallbackDNS=168.63.129.16 - 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service - 1. Confirm dns working with systemd-resolve google.com + The trigger was found to be open-vm-tools issuing "udevadm trigger". + Azure has a specific netplan setup that uses the `driver` match to set + up networking. If a udevadm trigger is executed, the KV pair that + contains this info is lost. Next time netplan is executed, the server + loses it's DNS information. + + This is the same as bug 1902960 experienced on Focal two years ago. + + The root cause was found to be a bug in systemd, where if we receive a + "Remove" action from a change uevent, we need to run net_setup_link(), + we need to skip device rename and keep the old name. + + [Testcase] + + Start an instance up on Azure, any type. Simply issue udevadm trigger + and reload systemd-networkd: + + $ ping google.com + PING google.com (172.253.62.102) 56(84) bytes of data. + 64 bytes from bc-in-f102.1e100.net (172.253.62.102): icmp_seq=1 ttl=56 time=1.85 ms + $ sudo udevadm trigger && sudo systemctl restart systemd-networkd + $ ping google.com + ping: google.com: Temporary failure in name resolution + + To fix a broken instance, you can run: + + $ sudo udevadm trigger -cadd -yeth0 && sudo systemctl restart systemd- + networkd + + and then install the test packages below: + + Test packages are available in the following ppa: + + https://launchpad.net/~mruffell/+archive/ubuntu/sf343528-test + + If you install them, the issue should no longer occur. + + [Where problems could occur] + + If a regression were to occur, it would affect systemd-udevd processing + 'change' events from network devices, which could lead to network + outages. Since this would happen when systemd-networkd is restarted on + postinstall, a regression would cause widespread outages due to this SRU + being targeted to the security pocket, where unattended-upgrades will + automatically install from. + + Side effects could include incorrect udevd device properties. + + It is very important that this SRU is well tested before release. + + [Other info] + + This was fixed in Systemd 247 with the following commit: + + commit e0e789c1e97e2cdf1cafe0c6b7d7e43fa054f151 + Author: Yu Watanabe + Date: Mon, 14 Sep 2020 15:21:04 +0900 + Subject: udev: re-assign ID_NET_DRIVER=, ID_NET_LINK_FILE=, ID_NET_NAME= properties on non-'add' uevent + Link: https://github.com/systemd/systemd/commit/e0e789c1e97e2cdf1cafe0c6b7d7e43fa054f151 + + This was backported to Focal's systemd 245.4-4ubuntu3.4 in bug 1902960 + two years ago. Focal required a heavy backport, which was performed by + Dan Streetman. Focals backport can be found in d/p/lp1902960-udev-re- + assign-ID_NET_DRIVER-ID_NET_LINK_FILE-ID_NET.patch, or the below + pastebin: + + https://paste.ubuntu.com/p/K5k7bGt3Wx/ + + The changes between the Focal backport and the Bionic backport are: + + - We use udev_device_get_action() instead of device_get_action() + - device_action_from_string() is used to get to enum DeviceAction + - We return 0 from the "if (a == DEVICE_ACTION_MOVE) " hunk instead of "goto no_rename" + - log_device_* has been changed to log_*. + + See attached debdiff for Bionic backport. ** Summary changed: - Update to systemd 237-3ubuntu10.54 broke dns + systemd-udevd: Run net_setup_link on 'change' uevents to prevent DNS outages on Azure ** Patch added: "Debdiff for systemd on Bionic" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+attachment/5612617/+files/lp1988119_bionic.debdiff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: systemd-udevd: Run net_setup_link on 'change' uevents to prevent DNS outages on Azure Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Bionic: In Progress Bug description: [Impact] A widespread outage was caused on Azure instances earlier today, when systemd 237-3ubuntu10.54 was published to the bionic-security pocket. Instances could no longer
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
** Changed in: systemd (Ubuntu Bionic) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Tags added: bionic sts -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Bionic: In Progress Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
manually scaling node pool/reboot nodes solves this issue. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Bionic: In Progress Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
** Also affects: systemd (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: systemd (Ubuntu Bionic) Status: New => In Progress ** Changed in: systemd (Ubuntu) Status: Confirmed => Fix Released ** Changed in: systemd (Ubuntu Bionic) Importance: Undecided => Critical -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Bionic: In Progress Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
You can find here some simple Python script to run a command to the VMSS instances for all subscriptions [or filtered ones]: https://github.com/foreachtoil/execute-command-on-all-vmss I still lack threading, so this might take a little bit. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
this worked for us: https://github.com/joaguas/aksdnsfallback#if-the- above-method-fails-because-dhclient-might-stall-another-alternative-is- to-configure-resolved-to-use-a-fallback-dns-server-which-we-can- hardcode-in-its-configuration -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
I've added a few alternatives how to fix the problem, mainly using az cli for vmss, ansible or running a daemonset in this gist : https://gist.github.com/naioja/eb8bac307a711e704b7923400b10bc14 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Microsoft Support provided fix for AKS, which I also tested successfully
is:
kubectl get no -o json | jq -r '.items[].spec.providerID' | cut -c 9- | az vmss
run-command invoke --ids @- \
--command-id RunShellScript \
--scripts 'grep nameserver /etc/resolv.conf || { dhclient -x; dhclient -i
eth0; sleep 10; pkill dhclient; grep nameserver /etc/resolv.conf; }'
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update to systemd 237-3ubuntu10.54 broke dns
Status in systemd package in Ubuntu:
Confirmed
Bug description:
Two servers today that updated systemd to "systemd 237-3ubuntu10.54"
https://ubuntu.com/security/notices/USN-5583-1
could not resolve dns anymore.
no dns servers, normally set through dhcp.
Ubuntu 18.04
Temp fix.
1. Edit /etc/systemd/resolved.conf
1. Add/Uncomment # FallbackDNS=168.63.129.16
1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service
1. Confirm dns working with systemd-resolve google.com
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
@atykhyy thank you that worked for VMSS! I also had some VMs without scale sets, fixed those with: az vm availability-set list -g --query "[].virtualMachines[].id" --output tsv | az vm run-command invoke --scripts "echo FallbackDNS=168.63.129.16 >> /etc/systemd/resolved.conf; systemctl restart systemd-resolved.service" --command-id RunShellScript --ids @- -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
** Tags added: regression-update -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Is it safe to downgrade from systemd 237-3ubuntu10.54 to the previous 237-3ubuntu10.50? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
az cli from cmd.exe, something like this for AKS nodes experiencing the issue: az vmss list-instances -g -n vmss --query "[].id" --output tsv | az vmss run-command invoke --scripts "echo FallbackDNS=168.63.129.16 >> /etc/systemd/resolved.conf; systemctl restart systemd-resolved.service" --command-id RunShellScript --ids @- -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
To temporary mitigate the ImagePullBackOff I scaled up a new functional node (DNS wise) and used this command to reconcile the AKS cluster: az resource update --resource-group --name --namespace Microsoft.ContainerService --resource-type ManagedClusters This recovered CoreDNS in the kube-system namespace, which fixed the ImagePullBackOff -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Yes @richardprammer, it appears ImagePullBackOff is one of the symptoms of this issue. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
> Could this be a related issue, when deployment to aks fails, due to a connection refused when pulling images from azure container registry(ImagePullBackOff). I you look closer at the message accompanying the ImagePullBackOff, you should see something like: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:36288->[::1]:53: read: connection refused Port 53 is the port a DNS server usually listens on. If this is what you're seeing, then yes: your problems are caused by the issue described in here. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Could this be a related issue, when deployment to aks fails, due to a connection refused when pulling images from azure container registry(ImagePullBackOff). This problem started this morning out of the blue. Credentials for azure container service are ok and about every 20 image pulls I get one, and the container would start. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
We are testing this in our AKS clusters now, but we were able to manually scale up a node pool which brought up new "working" nodes. Then manually scaled the pool back down to remove the "non-working" nodes. This left only new nodes up and the services are functioning properly now. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
For people having this is issue on AKS clusters with custom DNS... We have done this on all affected nodepools: $ VMSS=XXX-vmss $ nodeResourceGroup=XXX-worker $ az vmss list-instances -g $nodeResourceGroup -n $VMSS --query "[].id" --output tsv | az vmss run-command invoke --scripts "systemd-resolve --set-dns=your_dns --set-dns=your_dns --set-domain=reddog.microsoft.com --interface=eth0" --command-id RunShellScript --ids @- -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
For anyone hitting this issue with AKS clusters: I have embedded the workaround above in a daemonset to avoid having to restart all nodes (for now) https://gist.github.com/skinny/96e7feb6b347299ebfacaa76295a82e7 - Please check the image+tag used in the daemonset to whatever is available in your cluster. - Deploy this daemonset to the cluster (default namespace is used) - After all pods are running 1/1 for a bit, you can delete it - Images can be pulled again :) HTH -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Hey guys, nothing is working. My application has been out since this early morning. We have already tried to restart the nodes, restart the VM, but nothing has been working and we don't have any update from Microsoft. 4 hours ago they said "More information will be provided within 60 minutes, when we expect to know more about the root cause and mitigation workstreams.". -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Not sure if this is the best place to help people out understanding if nodes are impacted. We already saw 2 different types of impact on our Azure AKS clusters. - Pod not able to Terminate - New images being pulled from ACR (or any container registry Sometimes it was very clear that we saw the nodes where "Not Ready` in order cases it's very hard to detect. We have found a way to detect if your nodes are affected. kubectl logs When you get the following error you know it's impacted: Error from server (InternalError): Internal error occurred: Authorization error (user=masterclient, verb=get, resource=nodes, subresource=proxy) So restarting the node will help and especially if your cluster is sensitive then you can be more granular about the restart. I hope it helps some visitors from the azure status page -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
** Also affects: systemd Importance: Undecided Status: New ** No longer affects: systemd -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
> This won't help anyone that has already received the broken update - I think the advice there is to restart, or there is a workaround in the OP here - but it should prevent any further occurrences. Do note this is not a solution for those using non-Azure resolvers provided via DHCP through their VNET. These users must reboot or manually set the fallback servers to their custom DNS resolver addresses -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
I've removed the update from bionic-security and bionic-updates, and restored the versions which were previously in there. This won't help anyone that has already received the broken update - I think the advice there is to restart, or there is a workaround in the OP here - but it should prevent any further occurrences. Note that there will be a delay of up to an hour or so for mirrors to receive the deletion. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Microsoft has created an incident for this. https://azure.status.microsoft/en-us/status reports: Azure customers running Canonical Ubuntu 18.04 experiencing DNS errors - Investigating Starting at approximately 06:00 UTC on 30 Aug 2022, a number of customers running Ubuntu 18.04 (bionic) VMs recently upgraded to systemd version 237-3ubuntu10.54 reported experiencing DNS errors when trying to access their resources. Reports of this issue are confined to this single Ubuntu version. This bug and a potential fix have been highlighted on the Canonical / Ubuntu site, which we encourage impacted customers to read: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119 An additional potential workaround customers can consider is to reboot impacted VM instances so that they receive a fresh DHCP lease and new DNS resolver(s). Any Azure service, including AKS, that uses Canonical Ubuntu version 18.04 of Linux may have some impact from this issue. We are working on mitigations across Azure services that are impacted. More information will be provided within 60 minutes, when we expect to know more about the root cause and mitigation workstreams. This message was last updated at 09:20 UTC on 30 August 2022 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Seems to be a duplicate of https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1938791 - same symptoms. [Workaround] Reboot the node, DNS should return back to normal. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
We've just had the same problem, on multiple VMs running in Azure. In the dpkg log we can see that systemd was indeed updated (times in UTC): 2022-08-30 06:31:18 status unpacked udev:amd64 237-3ubuntu10.54 2022-08-30 06:31:18 status half-configured udev:amd64 237-3ubuntu10.54 2022-08-30 06:31:19 status installed udev:amd64 237-3ubuntu10.54 2022-08-30 06:31:19 status triggers-pending initramfs-tools:all 0.130ubuntu3.13 2022-08-30 06:31:19 trigproc man-db:amd64 2.8.3-2ubuntu0.1 2022-08-30 06:31:19 status half-configured man-db:amd64 2.8.3-2ubuntu0.1 2022-08-30 06:31:19 status installed man-db:amd64 2.8.3-2ubuntu0.1 2022-08-30 06:31:19 trigproc ureadahead:amd64 0.100.0-21 2022-08-30 06:31:19 status half-configured ureadahead:amd64 0.100.0-21 2022-08-30 06:31:20 status installed ureadahead:amd64 0.100.0-21 2022-08-30 06:31:20 trigproc libc-bin:amd64 2.27-3ubuntu1.5 2022-08-30 06:31:20 status half-configured libc-bin:amd64 2.27-3ubuntu1.5 2022-08-30 06:31:20 status installed libc-bin:amd64 2.27-3ubuntu1.5 2022-08-30 06:31:20 trigproc systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:20 status half-configured systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:20 status installed systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:20 trigproc initramfs-tools:all 0.130ubuntu3.13 2022-08-30 06:31:20 status half-configured initramfs-tools:all 0.130ubuntu3.13 2022-08-30 06:31:34 status installed initramfs-tools:all 0.130ubuntu3.13 2022-08-30 06:31:37 startup archives unpack 2022-08-30 06:31:38 upgrade libnss-systemd:amd64 237-3ubuntu10.53 237-3ubuntu10.54 2022-08-30 06:31:38 status triggers-pending libc-bin:amd64 2.27-3ubuntu1.5 2022-08-30 06:31:38 status half-configured libnss-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status unpacked libnss-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status half-installed libnss-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status triggers-pending man-db:amd64 2.8.3-2ubuntu0.1 2022-08-30 06:31:38 status half-installed libnss-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status unpacked libnss-systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:38 status unpacked libnss-systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:38 upgrade libpam-systemd:amd64 237-3ubuntu10.53 237-3ubuntu10.54 2022-08-30 06:31:38 status half-configured libpam-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status unpacked libpam-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status half-installed libpam-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status half-installed libpam-systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status unpacked libpam-systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:38 status unpacked libpam-systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:38 upgrade systemd:amd64 237-3ubuntu10.53 237-3ubuntu10.54 2022-08-30 06:31:38 status half-configured systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status unpacked systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:38 status half-installed systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:39 status triggers-pending ureadahead:amd64 0.100.0-21 2022-08-30 06:31:39 status triggers-pending dbus:amd64 1.12.2-1ubuntu1.3 2022-08-30 06:31:39 status triggers-pending dbus:amd64 1.12.2-1ubuntu1.3 2022-08-30 06:31:39 status half-installed systemd:amd64 237-3ubuntu10.53 2022-08-30 06:31:39 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:39 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 upgrade libsystemd0:amd64 237-3ubuntu10.53 237-3ubuntu10.54 2022-08-30 06:31:40 status half-configured libsystemd0:amd64 237-3ubuntu10.53 2022-08-30 06:31:40 status unpacked libsystemd0:amd64 237-3ubuntu10.53 2022-08-30 06:31:40 status half-installed libsystemd0:amd64 237-3ubuntu10.53 2022-08-30 06:31:40 status half-installed libsystemd0:amd64 237-3ubuntu10.53 2022-08-30 06:31:40 status unpacked libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 startup packages configure 2022-08-30 06:31:40 configure libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status half-configured libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status installed libsystemd0:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 startup packages configure 2022-08-30 06:31:40 configure systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64 237-3ubuntu10.54 2022-08-30 06:31:40 status unpacked systemd:amd64
[Touch-packages] [Bug 1988119] Re: Update to systemd 237-3ubuntu10.54 broke dns
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: systemd (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1988119 Title: Update to systemd 237-3ubuntu10.54 broke dns Status in systemd package in Ubuntu: Confirmed Bug description: Two servers today that updated systemd to "systemd 237-3ubuntu10.54" https://ubuntu.com/security/notices/USN-5583-1 could not resolve dns anymore. no dns servers, normally set through dhcp. Ubuntu 18.04 Temp fix. 1. Edit /etc/systemd/resolved.conf 1. Add/Uncomment # FallbackDNS=168.63.129.16 1. Restart systemd-resolved sudo systemctl restart systemd-resolved.service 1. Confirm dns working with systemd-resolve google.com To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
