[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Hi xavier, while it is true that the PIDFile directive is not in those nwer releases as outlined by yutani before in comment #6 the issue itself does not show up, see comment #10 I ran the test in Bionic again today and can confirm that it still does not trigger. So I have to assume it is

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Affects aswell 17.10 and 18.04. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title: keepalived fails to restart cleanly due to the wrong systemd settings To manage notifications about

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Changed in: keepalived (Debian) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title: keepalived fails to restart cleanly due to the wrong systemd

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

> Anything else needed on src:systemd side of things? Hi xnox, Well, in a perfect world and with a time machine to help you could look what in zesty's systemd fixed it to work reliably even with the suboptimal service file. But then you have more important tasks all around you and it is fixed

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Anything else needed on src:systemd side of things? ** Changed in: systemd (Ubuntu) Status: New => Incomplete ** Changed in: systemd (Ubuntu Xenial) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

This bug was fixed in the package keepalived - 1:1.2.19-1ubuntu0.2 --- keepalived (1:1.2.19-1ubuntu0.2) xenial; urgency=medium * Add PIDFile to avoid misdetection of MainPID on restart (LP: #1644530). -- Christian Ehrhardt Mon, 13 Mar 2017

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Changed in: systemd (Ubuntu) Assignee: Dimitri John Ledkov (xnox) => (unassigned) ** Changed in: systemd (Ubuntu) Milestone: ubuntu-17.03 => None -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Tested on Xenial pre-Proposed: 15 hits with-Proposed: 0 hits Also for general regression things seem to work normal. Setting verification-done. Yutani - if you could also verify Proposed that would make it even better! ** Tags removed: verification-needed ** Tags added: verification-done --

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Hello Yutani, or anyone else affected, Accepted keepalived into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keepalived/1:1.2.19-1ubuntu0.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package.

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Fix prepared via bileto at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2564/ I verified the fix against the ppa and that we need no extra systemd-daemon reload. Also in the meantime all dep8 tests finished. That said SRU template ready, fix ready, Tests ok => syncing into the

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Changed in: keepalived (Debian) Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title: keepalived fails to restart cleanly due to the wrong systemd settings

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Added SRU Template for Xenial - Yakkety/Zesty not showing the issue (likely due to systemd version) - Trusty is pre systemd and even the old version uses a --pidfile in its call to start-stop-daemon ** Description changed: + [Impact] + + * Restarts of keepalived can leave stale processes with

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

I tested yakkety to check where an SRU makes sense and can confirm that there with systemd 231-9ubuntu3 it already is slow on the loop restarts and working without the issue this bug is about. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Reported the request to pick up the PIDFile statement to Debian. Linking up the debbug here. ** Bug watch added: Debian Bug tracker #857618 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=857618 ** Also affects: keepalived (Debian) via

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

While I'd like to understand what systemd changed, I still think adding the PIDFile is correct as a fix for keepalived. Understanding systemd might just open up to fix more than just this service. Never the less for type forking (that is the case here) PIDFile is recommended

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

@Yutani - would you mind reporting on the two diffs to upstream to Debian and mention the bug number here? - After=syslog.target which might be reasonable to pick - PIDFile=/var/run/keepalived.pid (which is not bad and we thought would If you are unwilling or unable let me know, but it would be

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

I can confirm that the adding of a PIDFile as suggested makes it survive the looped test. We should try to understand the changes behind it working in zesty. But sooner or later adding the PIDFile might be the less invasive option to make it working in an SRU. -- You received this bug

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Changed in: systemd (Ubuntu) Milestone: None => ubuntu-17.03 ** Changed in: systemd (Ubuntu) Assignee: (unassigned) => Dimitri John Ledkov (xnox) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

What I think is happening in our case: Since no ExecStop= was specified, systemd will send SIGTERM [...] Details: https://www.freedesktop.org/software/systemd/man/systemd.kill.html# KillMode is "process" in the service file. That means "If set to process, only the main process itself is killed."

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Note: with my loop test above I could show that on Xenial this triggers "often" but not 100%, more like 75%. So to be sure on a fix one should run that loop with 10 seconds sleeps in between multiple times. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

First check with the head-to-head restart is confirming my former assumption on Zesty it is: 1. not failing 2. taking longer each restart I wanted to go further but a trivial test comparing with this: $ time for i in $(seq 1 200); do sudo systemctl restart keepalived; sudo systemctl status

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

The -HUP is only on reload, the issue is reported on restart. Which is really killing AFAIK, that makes even more sense (PIDs not fully updated on next kill) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

That makes some sort of sense. The action for reload is "Just" a signal to MAINPID ExecReload=/bin/kill -HUP $MAINPID I think the failure is a second -HUP signal before the former reload is fully complete. And on a more complex keepalived setup this might also be a longer time-window to trigger.

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Same setup as above in two Xenial VMs Again I see the backup taking over and reverting to backup once the restart is complete. Note: the restart is much faster now, like it is just sending a signal or so while on zesty it was feeling like waiting for completion. 0) pre restart Process: 2416

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Note: I compared the upstream and Debian/Ubuntu Service file. Differences are: 1. Service Name (don't care) 2. After=syslog.target which might be reasonable to pick 3. PIDFile=/var/run/keepalived.pid (which is not bad and we thought would be the fix here, but it seems is not required as my zesty

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

On Zesty (still without the pidfile in the service) it seems to work fine (Version 1:1.3.2-1). Test: 2x zesty KVM guests sudo apt-get install keepalived Set up as above, but with ens3 (virtual ethenet device). I think the name of backup/master conf names are interchanged in the example above,

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Hi Yutani, thank you so much. No need to excuse for adding the repro late, I OTOH missed to subscribe back then and threby missed to see your reply on the upstream PR in time. Anyway - lets see what we can do to get your great work and analysis to help all of the Ubunut community. I'll start

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Changed in: keepalived (Ubuntu) Status: Fix Released => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title: keepalived fails to restart cleanly due to the wrong systemd

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Description changed: Because "PIDFile=" directive is missing in the systemd unit file, keepalived sometimes fails to kill all old processes. The old processes remain with old settings and cause unexpected behaviors. The detail of this bug is described in this ticket in upstream:

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

I've provided the minimal reproducible example in Bug Description (sorry for this late). Can you confirm? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title: keepalived fails to restart

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Hi, As I have pointed above, this problem is NOT solved! Even the newest version of keepalived package (1.3.2) still lacks "PIDFile=" directive. $ wget http://launchpadlibrarian.net/296728051/keepalived_1.3.2-1_amd64.deb $ dpkg-deb --extract keepalived_1.3.2-1_amd64.deb keepalived $ cat

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

** Description changed: Because "PIDFile=" directive is missing in the systemd unit file, keepalived sometimes fails to kill all old processes. The old processes remain with old settings and cause unexpected behaviors. The detail of this bug is described in this ticket in upstream:

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Thank you for your comment! > 1.2.24 is already in the coming Ubuntu relase, so that is fixed there. I'm afraid not. The issue still exists since the upstream continues to use their own systemd unit file. I've created the same patch to the very upstream:

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Thank you for your report and your help to make Ubuntu better. 1.2.24 is already in the coming Ubuntu relase, so that is fixed there. To consider an SRU [1] there should be some clear steps to reproduce this issue. Did you find on your debugging a reliable way to trigger this issue so it can be

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

The attachment "add PIDFile= directive in systemd unit file" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team. [This is an automated message performed by a

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Sorry, I repost my patch. ** Patch removed: "add PIDFile= directive in systemd unit file" https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1644530/+attachment/4782407/+files/fix-systemd-unit.patch ** Patch added: "add PIDFile= directive in systemd unit file"

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: keepalived (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1644530 Title:

[Bug 1644530] Re: keepalived fails to restart cleanly due to the wrong systemd settings

This patch should fix the issue. ** Patch added: "add PIDFile= directive in systemd unit file" https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1644530/+attachment/4782407/+files/fix-systemd-unit.patch -- You received this bug notification because you are a member of Ubuntu Bugs,