[Bug 1810583] Re: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency
Thanks Karl. This solution from Chris Aumann seems perfect, and I've just deployed it onto our HAPROXY pair. Just restarted udev and networkd, and everything survives as expected now. Much appreciated. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1810583 Title: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1810583] Re: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency
We had this happen again this morning, causing an outage. Same issue, apt daily leads to a udev restart, which in turn restarted the network service and caused VRRP address to be lost on both haproxy nodes. I am going to try and completely disable the apt daily scheduled job while this bug remains. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1810583 Title: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1810583] Re: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency
Andreas, in our case this was a one off. The system had been running for 2 months without any issues, and this sudden network restart due to a daily update check was not expected. We did a lot of testing different failover events (disconnecting vNIC, powering off a single node, stopping keepalived service etc), but we never specifically tested a restart of the networkd service. This bug has potentially gone unnoticed for some time because of this aspect, and the frequency of this event occurring (in our case), is low. Just for visibility, the specific workaround I attempted to implement which recycled keepalived on netowrk restart was to add an override to networkd unit file using the following commands. This results in the immediate issues being fixed (keepalived restarts as desired), but prevents the network daemon from starting up after a reboot causing the system to become stuck in a wait loop. I had to boot to recovery mode and remove the override file again to restore functionality. - sudo systemctl edit systemd-networkd then in the override file via NANO: [Service] ExecStartPost=!/bin/systemctl restart keepalived - -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1810583 Title: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1810583] Re: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency
Just to add, we also attempted to work around this by adding a systemd override to netplan to recycle the keepalived service whenever network management was restarted. While it corrected the issue, it also created another problem whereby the system hung on startup after a reboot waiting endlessly for the network daemon to start. I had to revert this change in light of this. For now, I've disabled ubuntu auto update task completely and hopefully this will avoid any network service restarts until the issue is resolved within the package. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1810583 Title: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1810583] Re: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency
Hi Karl. I can confirm this issue also, we encountered it this morning on a 2 node keepalived cluster consisting of 2 VMWARE ubuntu 18.04.1 VMs. In our case, a daily update task had restarted UDEV, which in turn restarted systemd-networkd. When this service restarted, the virtual ip on the MASTER node's NIC was lost, but nothing was recognised by keepalived and the ip was never restored on either MASTER or BACKUP. This caused an outage of services hosted on the virtualip. When we investigated, we found that both MASTER and BACKUP nodes only had their own primary ip addresses, and neither node had the virtual ip. The virtual ip was unreachable. No managed failover by keepalived had occurred. We restarted keepalived on both nodes, which caused the virtual ip to re-appear on the MASTER node's NIC. We can reproduce this on demand right now by manually restarting systemd-networkd, which causes the virtual ip to vanish. The only way to get it to return is to then manually restart keepalived. Notably, when this problem occurs, nothing is logged by keepalived in syslog at all, which suggests it's not recognising the restart of networkd, or the loss of the virtual ip, and therefore not announcing it to the BACKUP node. There is a good discussion on the ubuntu forums about this, and someone has confirmed that downgrading the keepalived package to the previous one resolves this behaviour, so it does look like the patch in the latest package version has potentially introduced this. Here is the thread for ref: https://ubuntuforums.org/showthread.php?t=2406400=13819524#post13819524 I'm happy to test anything required on a VM if necessary. We haven't taken any action to workaround this yet. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1810583 Title: Daily cron restarts network on unattended updates but keepalived .service is not restarted as a dependency To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
