Bug#922031: certbot: Debian 9 systemd timer inactive after upgrade to 0.28.0-1~deb9u1
Hi. I'm affected by this issue as well. I have a few virtual machines on which certificates stopped renewing. All machines have Debian Stable with the most recent updates installed as soon as possible. It seems to be an issue with systemd. The following output is on "broken" system: # systemctl list-timers --all NEXT LEFT LAST PASSED UNIT ACTIVATES Fri 2019-03-08 13:31:33 CET 17min left Thu 2019-03-07 13:31:33 CET 23h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Fri 2019-03-08 16:59:19 CET 3h 45min left Thu 2019-03-07 21:10:33 CET 16h ago apt-daily.timer apt-daily.service Sat 2019-03-09 06:46:22 CET 17h left Fri 2019-03-08 06:19:33 CET 6h ago apt-daily-upgrade.timer apt-daily-upgrade.service n/a n/a Mon 2019-02-04 12:21:02 CET 1 months 1 days ago certbot.timer certbot.service I tried to run systemctl daemon-reload, but it doesn't help. However, rebooting the machine fixed the issue, at least for some time: # systemctl list-timers --all NEXT LEFT LAST PASSED UNIT ACTIVATES Fri 2019-03-08 14:07:46 CET 52min left Thu 2019-03-07 14:07:46 CET 23h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Fri 2019-03-08 18:27:06 CET 5h 12min left Fri 2019-03-08 04:43:46 CET 8h ago certbot.timer certbot.service Sat 2019-03-09 05:08:12 CET 15h left Fri 2019-03-08 13:09:49 CET 5min ago apt-daily.timer apt-daily.service Sat 2019-03-09 06:38:08 CET 17h left Fri 2019-03-08 06:03:49 CET 7h ago apt-daily-upgrade.timer apt-daily-upgrade.service It is quite a huge problem on production when one cannot rely on correct behavior of systemd. -- Marcel Šebek
Bug#922031: certbot: Debian 9 systemd timer inactive after upgrade to 0.28.0-1~deb9u1
Hi, I am seeing the exact same behavior on my system running raspian 9, certbot 0.28.0-1~deb9u1 with apache. My letsencrypt certificate almost expired because of it. Is there a workaround? "sudo systemctl start certbot.timer" appears to have restarted the timer, but I am not sure it is stable upon reboot. Thanks, David On Tue, 12 Feb 2019 08:56:02 +0100 =?iso-8859-1?Q?V=E1clav_Ovs=EDk?= wrote: > Hello Harlan, > > On Mon, Feb 11, 2019 at 06:45:46PM -0500, Harlan Lieberman-Berg wrote: > > ... > > > > Hello Václav, > > > > Hm, that's strange. I went back and checked, and the timer file was > > present in the 0.10 version as well. The timer used to work, and now > > no longer does, correct? > > Exactly. I speak to colleague and he is also affected on one machine. > > > Can you send the output of `systemctl show certbot.timer`? > > attached... > I left the thing as is on one server (anketa). The timer is still > inactive... > > anketa:~# systemctl list-timers --all > NEXT LEFT LAST PASSED UNIT ACTIVATES > Tue 2019-02-12 08:39:00 CET 4min 18s left Tue 2019-02-12 08:09:01 CET 25min ago phpsessionclean.timer phpsessionclean.service > Tue 2019-02-12 09:10:54 CET 36min left Mon 2019-02-11 20:36:35 CET 11h ago apt-daily.timer apt-daily.service > Tue 2019-02-12 20:30:40 CET 11h left Mon 2019-02-11 20:30:40 CET 12h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service > Wed 2019-02-13 06:11:23 CET 21h left Tue 2019-02-12 06:39:20 CET 1h 55min ago apt-daily-upgrade.timer apt-daily-upgrade.service > n/a n/a Tue 2019-01-29 00:18:30 CET 2 weeks 0 days ago certbot.timer certbot.service > > 5 timers listed. > > anketa:/var/log/letsencrypt# ll -lrt /var/log/letsencrypt/ | tail -15 > -rw-r--r-- 1 root root 642 Jan 26 00:42 letsencrypt.log.12 > -rw-r--r-- 1 root root 642 Jan 26 12:58 letsencrypt.log.11 > -rw-r--r-- 1 root root 642 Jan 27 00:01 letsencrypt.log.10 > -rw-r--r-- 1 root root 642 Jan 27 12:44 letsencrypt.log.9 > -rw-r--r-- 1 root root 642 Jan 28 00:06 letsencrypt.log.8 > -rw-r--r-- 1 root root 973 Jan 28 10:09 letsencrypt.log.7 > -rw-r--r-- 1 root root 17371 Jan 28 10:09 letsencrypt.log.6 > -rw-r--r-- 1 root root 2976 Jan 28 10:13 letsencrypt.log.5 > -rw-r--r-- 1 root root 10862 Jan 28 10:13 letsencrypt.log.4 > -rw-r--r-- 1 root root 24656 Jan 28 10:15 letsencrypt.log.3 > -rw-r--r-- 1 root root 26076 Jan 28 10:16 letsencrypt.log.2 > -rw-r--r-- 1 root root 642 Jan 28 12:26 letsencrypt.log.1 > -rw-r--r-- 1 root root 297 Jan 29 00:18 letsencrypt.log-20190203.gz > -rw-r--r-- 1 root root 20 Feb 3 06:25 letsencrypt.log-20190210.gz > -rw-r--r-- 1 root root 0 Feb 10 06:25 letsencrypt.log > > > As the certbot was not run anymore, logrotate started to rotate > letsencrypt.log file. I have logrotate configured to use dateext to not > fulfill incremental backups with cruft. Renamed logfile series are > backed-up repeatedly... With date extension they are static and go to > backup only one times. > BTW: certbot with its thousand files is a bit inconvenient. Every time > certbot runs, it rotates all thousand files, so they all go to backup > everyday. This is upstream problem probably... > > The times of some of the above logfiles are from Jan 28, because I was solving > a change of the LE authentication method. > > Thanks for your time and effort -- ** David M. Kaplan Charge de Recherche 1 Institut de Recherche pour le Developpement (IRD) UMR MARBEC (IRD/Ifremer/CNRS/UMII) av. Jean Monnet CS 30171 34203 Sete cedex France Email: david.kap...@ird.fr Phone: +33 (0)4 99 57 32 25 Fax: +33 (0)4 99 57 32 95 http://www.umr-marbec.fr/kaplan-david.html http://www.davidmkaplan.fr/ **
Bug#922031: certbot: Debian 9 systemd timer inactive after upgrade to 0.28.0-1~deb9u1
Hello Harlan, On Mon, Feb 11, 2019 at 06:45:46PM -0500, Harlan Lieberman-Berg wrote: > ... > > Hello Václav, > > Hm, that's strange. I went back and checked, and the timer file was > present in the 0.10 version as well. The timer used to work, and now > no longer does, correct? Exactly. I speak to colleague and he is also affected on one machine. > Can you send the output of `systemctl show certbot.timer`? attached... I left the thing as is on one server (anketa). The timer is still inactive... anketa:~# systemctl list-timers --all NEXT LEFT LAST PASSED UNIT ACTIVATES Tue 2019-02-12 08:39:00 CET 4min 18s left Tue 2019-02-12 08:09:01 CET 25min ago phpsessionclean.timerphpsessionclean.service Tue 2019-02-12 09:10:54 CET 36min leftMon 2019-02-11 20:36:35 CET 11h agoapt-daily.timer apt-daily.service Tue 2019-02-12 20:30:40 CET 11h left Mon 2019-02-11 20:30:40 CET 12h agosystemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service Wed 2019-02-13 06:11:23 CET 21h left Tue 2019-02-12 06:39:20 CET 1h 55min ago apt-daily-upgrade.timer apt-daily-upgrade.service n/a n/a Tue 2019-01-29 00:18:30 CET 2 weeks 0 days ago certbot.timercertbot.service 5 timers listed. anketa:/var/log/letsencrypt# ll -lrt /var/log/letsencrypt/ | tail -15 -rw-r--r-- 1 root root 642 Jan 26 00:42 letsencrypt.log.12 -rw-r--r-- 1 root root 642 Jan 26 12:58 letsencrypt.log.11 -rw-r--r-- 1 root root 642 Jan 27 00:01 letsencrypt.log.10 -rw-r--r-- 1 root root 642 Jan 27 12:44 letsencrypt.log.9 -rw-r--r-- 1 root root 642 Jan 28 00:06 letsencrypt.log.8 -rw-r--r-- 1 root root 973 Jan 28 10:09 letsencrypt.log.7 -rw-r--r-- 1 root root 17371 Jan 28 10:09 letsencrypt.log.6 -rw-r--r-- 1 root root 2976 Jan 28 10:13 letsencrypt.log.5 -rw-r--r-- 1 root root 10862 Jan 28 10:13 letsencrypt.log.4 -rw-r--r-- 1 root root 24656 Jan 28 10:15 letsencrypt.log.3 -rw-r--r-- 1 root root 26076 Jan 28 10:16 letsencrypt.log.2 -rw-r--r-- 1 root root 642 Jan 28 12:26 letsencrypt.log.1 -rw-r--r-- 1 root root 297 Jan 29 00:18 letsencrypt.log-20190203.gz -rw-r--r-- 1 root root20 Feb 3 06:25 letsencrypt.log-20190210.gz -rw-r--r-- 1 root root 0 Feb 10 06:25 letsencrypt.log As the certbot was not run anymore, logrotate started to rotate letsencrypt.log file. I have logrotate configured to use dateext to not fulfill incremental backups with cruft. Renamed logfile series are backed-up repeatedly... With date extension they are static and go to backup only one times. BTW: certbot with its thousand files is a bit inconvenient. Every time certbot runs, it rotates all thousand files, so they all go to backup everyday. This is upstream problem probably... The times of some of the above logfiles are from Jan 28, because I was solving a change of the LE authentication method. Thanks for your time and effort -- Zito Unit=certbot.service NextElapseUSecRealtime=infinity NextElapseUSecMonotonic=infinity LastTriggerUSec=49y 3w 6d 17h 18min 30.552506s LastTriggerUSecMonotonic=1w 3d 4h 8min 41.634499s Result=success AccuracyUSec=1min RandomizedDelayUSec=12h Persistent=yes WakeSystem=no RemainAfterElapse=yes Id=certbot.timer Names=certbot.timer Requires=-.mount sysinit.target var.mount WantedBy=timers.target Conflicts=shutdown.target Before=shutdown.target certbot.service timers.target After=-.mount time-sync.target sysinit.target var.mount Triggers=certbot.service RequiresMountsFor=/var/lib/systemd/timers Description=Run certbot twice daily LoadState=loaded ActiveState=inactive SubState=dead FragmentPath=/lib/systemd/system/certbot.timer UnitFileState=enabled UnitFilePreset=enabled StateChangeTimestamp=Tue 2019-01-29 08:14:17 CET StateChangeTimestampMonotonic=907468902955 InactiveExitTimestamp=Fri 2019-01-18 20:10:53 CET InactiveExitTimestampMonotonic=64450831 ActiveEnterTimestamp=Fri 2019-01-18 20:10:53 CET ActiveEnterTimestampMonotonic=64450831 ActiveExitTimestamp=Tue 2019-01-29 08:14:17 CET ActiveExitTimestampMonotonic=907468902955 InactiveEnterTimestamp=Tue 2019-01-29 08:14:17 CET InactiveEnterTimestampMonotonic=907468902955 CanStart=yes CanStop=yes CanReload=no CanIsolate=no StopWhenUnneeded=no RefuseManualStart=no RefuseManualStop=no AllowIsolate=no DefaultDependencies=yes OnFailureJobMode=replace IgnoreOnIsolate=no NeedDaemonReload=no JobTimeoutUSec=infinity JobTimeoutAction=none ConditionResult=yes AssertResult=yes ConditionTimestamp=Fri 2019-01-18 20:10:53 CET ConditionTimestampMonotonic=64239605 AssertTimestamp=Fri 2019-01-18 20:10:53 CET AssertTimestampMonotonic=64239605 Transient=no Perpetual=no StartLimitIntervalSec=1000 StartLimitBurst=5 StartLimitAction=none InvocationID=242ece5a59404f9ea996e16b505f3aa4
Bug#922031: certbot: Debian 9 systemd timer inactive after upgrade to 0.28.0-1~deb9u1
On Mon, Feb 11, 2019 at 8:12 AM Václav Ovsík wrote: > there is no periodic certificate renew task. LE sent me a first email > about a first certificate expiration. Hello Václav, Hm, that's strange. I went back and checked, and the timer file was present in the 0.10 version as well. The timer used to work, and now no longer does, correct? Can you send the output of `systemctl show certbot.timer`? Sincerely, -- Harlan Lieberman-Berg ~hlieberman
Bug#922031: certbot: Debian 9 systemd timer inactive after upgrade to 0.28.0-1~deb9u1
Package: certbot Version: 0.28.0-1~deb9u1 Severity: important Dear Maintainer, after the latest certbot upgrade on Debian Stretch python-certbot-apache:amd64 0.10.2-1 -> 0.28.0-1~deb9u1 there is no periodic certificate renew task. LE sent me a first email about a first certificate expiration. vidle1:~# systemctl list-timers --all NEXT LEFT LAST PASSED UNIT ACTIVATES Mon 2019-02-11 16:27:43 CET 2h 36min left Mon 2019-02-11 05:59:09 CET 7h ago apt-daily.timer apt-daily.se Mon 2019-02-11 20:48:40 CET 6h left Sun 2019-02-10 20:48:40 CET 17h agosystemd-tmpfiles-clean.timer systemd-tmpf Tue 2019-02-12 06:53:13 CET 17h left Mon 2019-02-11 06:58:01 CET 6h ago apt-daily-upgrade.timer apt-daily-up n/a n/a Tue 2019-01-29 00:12:14 CET 1 weeks 6 days ago certbot.timercertbot.serv 4 timers listed. I have this problem on all my hosts with certbot. Starting timer by hand seems ok. systemctl start certbot.timer Regards -- Zito *** Reporter, please consider answering these questions, where appropriate *** * What led up to the situation? * What exactly did you do (or not do) that was effective (or ineffective)? * What was the outcome of this action? * What outcome did you expect instead? *** End of the template - remove these template lines *** -- System Information: Debian Release: 9.7 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-8-amd64 (SMP w/2 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=cs_CZ.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages certbot depends on: ii init-system-helpers 1.48 ii python3 3.5.3-1 ii python3-certbot 0.28.0-1~deb9u1 certbot recommends no packages. Versions of packages certbot suggests: pn python-certbot-doc ii python3-certbot-apache 0.28.0-1~deb9u1 pn python3-certbot-nginx -- no debconf information
