Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 17:54 schrieb Michael Biebl: > Am 10.03.19 um 17:36 schrieb Harlan Lieberman-Berg: >> On Sun, Mar 10, 2019 at 12:29 PM Michael Biebl wrote: >>> Can you provide the output of >>> systemctl status certbot.timer >>> journalctl -u certbot.timer >> >> The output of `systemctl show certbot.timer` is at >> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=922031;filename=systemctl-show-certbot.timer.txt;msg=20 >> . One of the reporters will have to follow up with the output of >> journalctl -u certbot.timer, as I can't replicate the problem. >> >>> Is certbot.timer restarted as part of the package update? >> >> Not unless dh_installsystemd is doing it automagically, no. > > I had a quick look, it's a bug in your package. > What happens is roughly this: > > 0.12 used and older compat level where > dh_systemd_start defaults to stop in old/prerm, and start in > new/postinst. So during the upgrade, certbot.prerm stops certbot.timer. > > Your newer certbot package uses a newer compat level which defaults to > restart after upgrade. > If you check /var/lib/dpkg/info/certbot.postinst, you'll see a > systemctl try-restart certbot-timer. So, I need to clarify this a bit. dh_systemd_start (or dh_installsystemd) is supposed to use "systemctl restart" for --restart-after-upgrade (the default action with compat 10 and above). In your case, dh_systemd_start generated code which uses "systemctl try-restart", which is an important difference. "try-restart" is a nop if the unit is not already running, "restart" will start the unit in such a case. Now, dh_systemd_start is only supposed to use "try-restart" as action when used in combination with "--no-start". You only use that for certbot.service though and not for certbot.timer. It seems dh_systemd_start in stable get's confused by override_dh_systemd_start: dh_systemd_start --package=certbot certbot.timer dh_systemd_start --package=certbot --no-start certbot.service and generates this imo faulty code. I also checked dh_systemd_start in buster which does the expected thing here. This doesn't really change anything regarding that the best fix is to simply roll back the compat level to 9 in certbot for the stable upload. The takeaway here is, that it helps to check the generated maintainer scripts code, especially when you do a compat level bump. Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 18:23 schrieb Harlan Lieberman-Berg: > On Sun, Mar 10, 2019 at 1:19 PM Michael Biebl wrote: >> Let's bump this to serious. I think you should consider making another >> stable upload for this. >> As users have pointed out, systems which aren't rebooted regularly are >> otherwise up for a nasty experience. >> You might consider asking the security team to distribute the update >> with stable-security more quickly and not wait for the next stable upload. > > I agree; this needs an update. I'll open a ticket with the SRMs and > see what they think about it going to -security. > >> You have a couple of options to fix this: >> 1/ Switch the compat level back to what 0.10 was using > > I think this is the most reasonable approach, since newer versions > aren't using dh_installsystemd anyway. > > The bigger question for me is... what do we do about users who have > already upgraded? If my understanding is correct, if we switch back to > compat level 9, they'll all be forcibly started again by the postinst, > correct? Or do we need to explicitly start it again? > debhelper should generate code which starts certbot.timer in postinst. Please double check the generated postinst to be sure. Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
On Sun, Mar 10, 2019 at 1:19 PM Michael Biebl wrote: > Let's bump this to serious. I think you should consider making another > stable upload for this. > As users have pointed out, systems which aren't rebooted regularly are > otherwise up for a nasty experience. > You might consider asking the security team to distribute the update > with stable-security more quickly and not wait for the next stable upload. I agree; this needs an update. I'll open a ticket with the SRMs and see what they think about it going to -security. > You have a couple of options to fix this: > 1/ Switch the compat level back to what 0.10 was using I think this is the most reasonable approach, since newer versions aren't using dh_installsystemd anyway. The bigger question for me is... what do we do about users who have already upgraded? If my understanding is correct, if we switch back to compat level 9, they'll all be forcibly started again by the postinst, correct? Or do we need to explicitly start it again? -- Harlan Lieberman-Berg ~hlieberman
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Control: severity -1 serious Let's bump this to serious. I think you should consider making another stable upload for this. As users have pointed out, systems which aren't rebooted regularly are otherwise up for a nasty experience. You might consider asking the security team to distribute the update with stable-security more quickly and not wait for the next stable upload. You have a couple of options to fix this: 1/ Switch the compat level back to what 0.10 was using 2/ Keep the current compat level 10, but change your debian/rules override_dh_systemd_start: dh_systemd_start --no-restart-after-upgrade --package=certbot certbot.timer 3/ Keep the restart-after-upgrade behaviour and start certbot.timer manually on this particular. This should be guarded by a version check. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 17:54 schrieb Michael Biebl: > Am 10.03.19 um 17:36 schrieb Harlan Lieberman-Berg: >> On Sun, Mar 10, 2019 at 12:29 PM Michael Biebl wrote: >>> Can you provide the output of >>> systemctl status certbot.timer >>> journalctl -u certbot.timer >> >> The output of `systemctl show certbot.timer` is at >> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=922031;filename=systemctl-show-certbot.timer.txt;msg=20 >> . One of the reporters will have to follow up with the output of >> journalctl -u certbot.timer, as I can't replicate the problem. >> >>> Is certbot.timer restarted as part of the package update? >> >> Not unless dh_installsystemd is doing it automagically, no. > > I had a quick look, it's a bug in your package. > What happens is roughly this: > > 0.12 used and older compat level where > dh_systemd_start defaults to stop in old/prerm, and start in > new/postinst. So during the upgrade, certbot.prerm stops certbot.timer. Check 0.10.12's /var/lib/dpkg/info/certbot.prerm: if [ -d /run/systemd/system ]; then deb-systemd-invoke stop certbot.timer >/dev/null fi Your timer is stopped at this point. > Your newer certbot package uses a newer compat level which defaults to > restart after upgrade. > If you check /var/lib/dpkg/info/certbot.postinst, you'll see a > systemctl try-restart certbot-timer. > This will only restart systemd if it was started. > Since the old package stopped certbot timer in prerm, your certbot.timer > is not (re)started on upgrades. 0.28.0's /var/lib/dpkg/info/certbot.postinst # Automatically added by dh_systemd_start if [ -d /run/systemd/system ]; then systemctl --system daemon-reload >/dev/null || true if [ -n "$2" ]; then _dh_action=try-restart else _dh_action=start fi deb-systemd-invoke $_dh_action certbot.timer >/dev/null || true fi from man systemctl: try-restart PATTERN... Restart one or more units specified on the command line if the units are running. This does nothing if units are not running. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 17:59 schrieb Michael Biebl: > Am 10.03.19 um 17:54 schrieb Michael Biebl: >> Am 10.03.19 um 17:36 schrieb Harlan Lieberman-Berg: >>> On Sun, Mar 10, 2019 at 12:29 PM Michael Biebl wrote: Can you provide the output of systemctl status certbot.timer journalctl -u certbot.timer >>> >>> The output of `systemctl show certbot.timer` is at >>> https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=922031;filename=systemctl-show-certbot.timer.txt;msg=20 >>> . One of the reporters will have to follow up with the output of >>> journalctl -u certbot.timer, as I can't replicate the problem. >>> Is certbot.timer restarted as part of the package update? >>> >>> Not unless dh_installsystemd is doing it automagically, no. >> >> I had a quick look, it's a bug in your package. >> What happens is roughly this: >> >> 0.12 used and older compat level where >> dh_systemd_start defaults to stop in old/prerm, and start in >> new/postinst. So during the upgrade, certbot.prerm stops certbot.timer. > > Check 0.10.12's /var/lib/dpkg/info/certbot.prerm: > > if [ -d /run/systemd/system ]; then > deb-systemd-invoke stop certbot.timer >/dev/null > fi > > Your timer is stopped at this point. > >> Your newer certbot package uses a newer compat level which defaults to >> restart after upgrade. >> If you check /var/lib/dpkg/info/certbot.postinst, you'll see a >> systemctl try-restart certbot-timer. >> This will only restart systemd if it was started. >> Since the old package stopped certbot timer in prerm, your certbot.timer >> is not (re)started on upgrades. > > 0.28.0's /var/lib/dpkg/info/certbot.postinst > > # Automatically added by dh_systemd_start > if [ -d /run/systemd/system ]; then > systemctl --system daemon-reload >/dev/null || true > if [ -n "$2" ]; then > _dh_action=try-restart > else > _dh_action=start > fi > deb-systemd-invoke $_dh_action certbot.timer >/dev/null || true > fi > > from man systemctl: > >try-restart PATTERN... >Restart one or more units specified on the command line if > the units are running. This does nothing if units are not running. > restart-after-upgrade is the default starting with compat level 10. See the debhelper man page. -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 17:36 schrieb Harlan Lieberman-Berg: > On Sun, Mar 10, 2019 at 12:29 PM Michael Biebl wrote: >> Can you provide the output of >> systemctl status certbot.timer >> journalctl -u certbot.timer > > The output of `systemctl show certbot.timer` is at > https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=922031;filename=systemctl-show-certbot.timer.txt;msg=20 > . One of the reporters will have to follow up with the output of > journalctl -u certbot.timer, as I can't replicate the problem. > >> Is certbot.timer restarted as part of the package update? > > Not unless dh_installsystemd is doing it automagically, no. I had a quick look, it's a bug in your package. What happens is roughly this: 0.12 used and older compat level where dh_systemd_start defaults to stop in old/prerm, and start in new/postinst. So during the upgrade, certbot.prerm stops certbot.timer. Your newer certbot package uses a newer compat level which defaults to restart after upgrade. If you check /var/lib/dpkg/info/certbot.postinst, you'll see a systemctl try-restart certbot-timer. This will only restart systemd if it was started. Since the old package stopped certbot timer in prerm, your certbot.timer is not (re)started on upgrades. Regards, Michael -- Why is it that all of the instruments seeking intelligent life in the universe are pointed away from Earth? signature.asc Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
On Sun, Mar 10, 2019 at 12:29 PM Michael Biebl wrote: > Can you provide the output of > systemctl status certbot.timer > journalctl -u certbot.timer The output of `systemctl show certbot.timer` is at https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=922031;filename=systemctl-show-certbot.timer.txt;msg=20 . One of the reporters will have to follow up with the output of journalctl -u certbot.timer, as I can't replicate the problem. > Is certbot.timer restarted as part of the package update? Not unless dh_installsystemd is doing it automagically, no. Sincerely, -- Harlan Lieberman-Berg ~hlieberman
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Am 10.03.19 um 17:05 schrieb Harlan Lieberman-Berg:
> Hello pkg-systemd!
>
> There's a strange bug being reported in the stable-updated version of
> certbot about the systemd timer that I'm having trouble reproducing or
> understanding how it could happen.
>
> According to the users reporting the issue, between the update from
> 0.10~ to 0.28.0.1~deb9u1, the systemd timer stops functioning. In the
> list-timers output, both NEXT and LAST are listed as "n/a". Rebooting
> the machine fixes the problem (potentially temporarily?).
>
> Looking back at the changes that were made to the debian/ directory, I
> don't see anything that could have caused this. We did change the
> timer file during the upgrade, but just to change the value of
> RandomizedDelaySec -- and we didn't change from dh_installsystemd to
> dh_systemd_{enable,start}. I'm not sure how/why this could have
> happened, but I'm clearly out of my depth on the systemd stuff and
> need some help.
>
Can you provide the output of
systemctl status certbot.timer
journalctl -u certbot.timer
Is certbot.timer restarted as part of the package update?
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
signature.asc
Description: OpenPGP digital signature
Bug#922031: Timer Disabling on Package Update? (is: #922031)
Hello pkg-systemd!
There's a strange bug being reported in the stable-updated version of
certbot about the systemd timer that I'm having trouble reproducing or
understanding how it could happen.
According to the users reporting the issue, between the update from
0.10~ to 0.28.0.1~deb9u1, the systemd timer stops functioning. In the
list-timers output, both NEXT and LAST are listed as "n/a". Rebooting
the machine fixes the problem (potentially temporarily?).
Looking back at the changes that were made to the debian/ directory, I
don't see anything that could have caused this. We did change the
timer file during the upgrade, but just to change the value of
RandomizedDelaySec -- and we didn't change from dh_installsystemd to
dh_systemd_{enable,start}. I'm not sure how/why this could have
happened, but I'm clearly out of my depth on the systemd stuff and
need some help.
I'm not subscribed to pkg-systemd-maintainers, so please keep me (and
the bug!) CCed.
Thanks much!
Sincerely,
--
Harlan Lieberman-Berg
~hlieberman
