Attached is a debdiff for systemd on Bionic which fixes this bug.
** Description changed:
- Two servers today that updated systemd to "systemd 237-3ubuntu10.54"
- https://ubuntu.com/security/notices/USN-5583-1
+ [Impact]
- could not resolve dns anymore.
- no dns servers, normally set through
** Changed in: systemd (Ubuntu Bionic)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Tags added: bionic sts
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
manually scaling node pool/reboot nodes solves this issue.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update to systemd 237-3ubuntu10.54 broke dns
** Also affects: systemd (Ubuntu Bionic)
Importance: Undecided
Status: New
** Changed in: systemd (Ubuntu Bionic)
Status: New => In Progress
** Changed in: systemd (Ubuntu)
Status: Confirmed => Fix Released
** Changed in: systemd (Ubuntu Bionic)
Importance: Undecided
You can find here some simple Python script to run a command to the VMSS
instances for all subscriptions [or filtered ones]:
https://github.com/foreachtoil/execute-command-on-all-vmss
I still lack threading, so this might take a little bit.
--
You received this bug notification because you are
this worked for us: https://github.com/joaguas/aksdnsfallback#if-the-
above-method-fails-because-dhclient-might-stall-another-alternative-is-
to-configure-resolved-to-use-a-fallback-dns-server-which-we-can-
hardcode-in-its-configuration
--
You received this bug notification because you are a
I've added a few alternatives how to fix the problem, mainly using az
cli for vmss, ansible or running a daemonset in this gist :
https://gist.github.com/naioja/eb8bac307a711e704b7923400b10bc14
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which
Microsoft Support provided fix for AKS, which I also tested successfully
is:
kubectl get no -o json | jq -r '.items[].spec.providerID' | cut -c 9- | az vmss
run-command invoke --ids @- \
--command-id RunShellScript \
--scripts 'grep nameserver /etc/resolv.conf || { dhclient -x; dhclient -i
@atykhyy thank you that worked for VMSS!
I also had some VMs without scale sets, fixed those with:
az vm availability-set list -g --query
"[].virtualMachines[].id" --output tsv | az vm run-command invoke
--scripts "echo FallbackDNS=168.63.129.16 >> /etc/systemd/resolved.conf;
systemctl restart
** Tags added: regression-update
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update to systemd 237-3ubuntu10.54 broke dns
Status in systemd package in
Is it safe to downgrade from systemd 237-3ubuntu10.54 to the previous
237-3ubuntu10.50?
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update to systemd
az cli from cmd.exe, something like this for AKS nodes experiencing the
issue: az vmss list-instances -g -n vmss --query "[].id"
--output tsv | az vmss run-command invoke --scripts "echo
FallbackDNS=168.63.129.16 >> /etc/systemd/resolved.conf; systemctl
restart systemd-resolved.service"
To temporary mitigate the ImagePullBackOff I scaled up a new functional node
(DNS wise) and used this command to reconcile the AKS cluster:
az resource update --resource-group --name
--namespace Microsoft.ContainerService --resource-type ManagedClusters
This recovered CoreDNS in the
Yes @richardprammer, it appears ImagePullBackOff is one of the symptoms
of this issue.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update to systemd
> Could this be a related issue, when deployment to aks fails, due to a
connection refused when pulling images from azure container
registry(ImagePullBackOff).
I you look closer at the message accompanying the ImagePullBackOff, you should
see something like:
dial tcp: lookup
Could this be a related issue, when deployment to aks fails, due to a
connection refused when pulling images from azure container
registry(ImagePullBackOff). This problem started this morning out of the blue.
Credentials for azure container service are ok and about every 20 image pulls I
get
We are testing this in our AKS clusters now, but we were able to
manually scale up a node pool which brought up new "working" nodes. Then
manually scaled the pool back down to remove the "non-working" nodes.
This left only new nodes up and the services are functioning properly
now.
--
You
For people having this is issue on AKS clusters with custom DNS...
We have done this on all affected nodepools:
$ VMSS=XXX-vmss
$ nodeResourceGroup=XXX-worker
$ az vmss list-instances -g $nodeResourceGroup -n $VMSS --query "[].id"
--output tsv | az vmss run-command invoke --scripts
For anyone hitting this issue with AKS clusters: I have embedded the
workaround above in a daemonset to avoid having to restart all nodes
(for now)
https://gist.github.com/skinny/96e7feb6b347299ebfacaa76295a82e7
- Please check the image+tag used in the daemonset to whatever is available in
Hey guys, nothing is working. My application has been out since this
early morning. We have already tried to restart the nodes, restart the
VM, but nothing has been working and we don't have any update from
Microsoft. 4 hours ago they said "More information will be provided
within 60 minutes, when
Not sure if this is the best place to help people out understanding if nodes
are impacted.
We already saw 2 different types of impact on our Azure AKS clusters.
- Pod not able to Terminate
- New images being pulled from ACR (or any container registry
Sometimes it was very clear that we saw the
** Also affects: systemd
Importance: Undecided
Status: New
** No longer affects: systemd
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1988119
Title:
Update
> This won't help anyone that has already received the broken update - I
think the advice there is to restart, or there is a workaround in the OP
here - but it should prevent any further occurrences.
Do note this is not a solution for those using non-Azure resolvers
provided via DHCP through
I've removed the update from bionic-security and bionic-updates, and
restored the versions which were previously in there.
This won't help anyone that has already received the broken update - I
think the advice there is to restart, or there is a workaround in the OP
here - but it should prevent
Microsoft has created an incident for this.
https://azure.status.microsoft/en-us/status reports:
Azure customers running Canonical Ubuntu 18.04 experiencing DNS errors
- Investigating
Starting at approximately 06:00 UTC on 30 Aug 2022, a number of
customers running Ubuntu 18.04 (bionic) VMs
Seems to be a duplicate of
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1938791 - same
symptoms.
[Workaround]
Reboot the node, DNS should return back to normal.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to
We've just had the same problem, on multiple VMs running in Azure.
In the dpkg log we can see that systemd was indeed updated (times in
UTC):
2022-08-30 06:31:18 status unpacked udev:amd64 237-3ubuntu10.54
2022-08-30 06:31:18 status half-configured udev:amd64 237-3ubuntu10.54
2022-08-30 06:31:19
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: systemd (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
28 matches
Mail list logo
