Re: Ghost cronjob
On Sb, 08 mai 21, 10:12:59, l0f...@tuta.io wrote:
> Hi,
>
> 5 mai 2021, 04:48 de deb...@lionunicorn.co.uk:
>
> > After all, it's easy to override it with
> >
> > # ln -s /dev/null /etc/systemd/system/btrbk.service
> >
> You may need the `-f` (force) switch if btrbk.service file already exists.
While this is technically true, package provided service files are (or
at least should be) in /usr/lib/systemd/{system|user}/ only.
The .service files in /etc are there only because the admin put them
there, either to 1) provide a .service file for a service that doesn't
provide its own or 2) to override the /usr/lib version.
In both cases 'ln -sf' would lead to data loss.
For 1), just move the file out of the way or rename it to something that
doesn't end in .service. For 2) move the file out of the way and *then*
create the symlink to /dev/null [1], no '-f' needed.
[1] or just use 'systemctl mask ', which does the same.
Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser
signature.asc
Description: PGP signature
Re: Ghost cronjob
Hi, 5 mai 2021, 04:48 de deb...@lionunicorn.co.uk: > After all, it's easy to override it with > > # ln -s /dev/null /etc/systemd/system/btrbk.service > You may need the `-f` (force) switch if btrbk.service file already exists. Best regards, l0f4r0
Re: Ghost cronjob
On Tue 04 May 2021 at 22:20:14 (+0300), Andrei POPESCU wrote: > On Ma, 04 mai 21, 19:50:27, Mart van de Wege wrote: > > David Wright writes: > > > On Tue 04 May 2021 at 17:06:50 (+0200), Mart van de Wege wrote: > > >> > > >> Surely packaging a system timer file like that should not be done? It > > >> should be at the discretion of the admin to create one if they want to > > >> run btrbk on a schedule. > > > > > > I think this has been discussed in the past, and that installing a > > > package is generally seen as meaning that you want it to run: Debian > > > therefore tries to supply a reasonable and secure configuration. > > > > > But that's not applicable in this case. btrbk can be run standalone or > > from a cronjob/timer, but the exact time someone wants to run their > > backups, or even if they want them run periodically at all, is not a > > decision you can make for them. > > > > Of course an argument can be made that without edits to the default > > btrbk.conf file btrbk won't run anyway, so configuring the timer is part > > of configuring your backup strategy. I still think that supplying a > > system timer is wrong here. > > Feel free to contact the Maintainer about it. > > In any case, it probably makes sense to at least mention this in > NEWS.Debian, especially if the package previously shipped a cron job. AIUI the version in stretch is Debian's first appearance, so it came with systemd rather than cron. Reading README.md, "btrbk is designed to run as a cron job for triggering periodic snapshots and backups, as well as from the command line (e.g. for instantly creating additional snapshots)" I would have thought that btrbk was really designed for frequent and regular backups, eg even hourly (hence all the discussion about timestamps, and how they interact with timezones and with concurrent daily backups). Wouldn't that make a systemd timer a reasonable default to supply. After all, it's easy to override it with # ln -s /dev/null /etc/systemd/system/btrbk.service Cheers, David.
Re: Ghost cronjob
On Ma, 04 mai 21, 19:50:27, Mart van de Wege wrote: > David Wright writes: > > > On Tue 04 May 2021 at 17:06:50 (+0200), Mart van de Wege wrote: > >> > >> Surely packaging a system timer file like that should not be done? It > >> should be at the discretion of the admin to create one if they want to > >> run btrbk on a schedule. > > > > I think this has been discussed in the past, and that installing a > > package is generally seen as meaning that you want it to run: Debian > > therefore tries to supply a reasonable and secure configuration. > > > But that's not applicable in this case. btrbk can be run standalone or > from a cronjob/timer, but the exact time someone wants to run their > backups, or even if they want them run periodically at all, is not a > decision you can make for them. > > Of course an argument can be made that without edits to the default > btrbk.conf file btrbk won't run anyway, so configuring the timer is part > of configuring your backup strategy. I still think that supplying a > system timer is wrong here. Feel free to contact the Maintainer about it. In any case, it probably makes sense to at least mention this in NEWS.Debian, especially if the package previously shipped a cron job. Kind regards, Andrei -- http://wiki.debian.org/FAQsFromDebianUser signature.asc Description: PGP signature
Re: Ghost cronjob
David Wright writes: > On Tue 04 May 2021 at 17:06:50 (+0200), Mart van de Wege wrote: >> Sven Hartge writes: >> > Mart van de Wege wrote: >> > >> >> Nope, not ephemeral at all, it's PID 1. Since I don't have timers >> >> running this job, apparently there's a zombie process somewhere? >> > >> > PID 1 hints at a systemd.timer, even if you have dismissed this >> > previously. Also the start time of just after midnight hints at this. >> > >> > And when I look at the current btrbk package in Sid/Testing I do indeed >> > see it ships a timer in /usr/lib/systtemd/ >> > >> > So, there is your answer then. >> >> Well whaddayaknow. I hadn't checked the system timers under >> /usr/lib/systemd/ >> >> And that is the bastard running the ghost job. >> >> Surely packaging a system timer file like that should not be done? It >> should be at the discretion of the admin to create one if they want to >> run btrbk on a schedule. > > I think this has been discussed in the past, and that installing a > package is generally seen as meaning that you want it to run: Debian > therefore tries to supply a reasonable and secure configuration. > But that's not applicable in this case. btrbk can be run standalone or from a cronjob/timer, but the exact time someone wants to run their backups, or even if they want them run periodically at all, is not a decision you can make for them. Of course an argument can be made that without edits to the default btrbk.conf file btrbk won't run anyway, so configuring the timer is part of configuring your backup strategy. I still think that supplying a system timer is wrong here. > But I can't help wondering what was the result of both systemd *and* > your cron job running btrbk at the same time. Do you have backups of > the logs? > I have backups. What happened is that for about 1.5 months I ran double backups. Thankfully these are incremental snapshots, so not much diskspace is lost on my server. Regards, Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
On Tue 04 May 2021 at 17:06:50 (+0200), Mart van de Wege wrote: > Sven Hartge writes: > > Mart van de Wege wrote: > > > >> Nope, not ephemeral at all, it's PID 1. Since I don't have timers > >> running this job, apparently there's a zombie process somewhere? > > > > PID 1 hints at a systemd.timer, even if you have dismissed this > > previously. Also the start time of just after midnight hints at this. > > > > And when I look at the current btrbk package in Sid/Testing I do indeed > > see it ships a timer in /usr/lib/systtemd/ > > > > So, there is your answer then. > > Well whaddayaknow. I hadn't checked the system timers under > /usr/lib/systemd/ > > And that is the bastard running the ghost job. > > Surely packaging a system timer file like that should not be done? It > should be at the discretion of the admin to create one if they want to > run btrbk on a schedule. I think this has been discussed in the past, and that installing a package is generally seen as meaning that you want it to run: Debian therefore tries to supply a reasonable and secure configuration. But I can't help wondering what was the result of both systemd *and* your cron job running btrbk at the same time. Do you have backups of the logs? Cheers, David.
Re: Ghost cronjob
Sven Hartge writes: > Mart van de Wege wrote: > >> Nope, not ephemeral at all, it's PID 1. Since I don't have timers >> running this job, apparently there's a zombie process somewhere? > > PID 1 hints at a systemd.timer, even if you have dismissed this > previously. Also the start time of just after midnight hints at this. > > And when I look at the current btrbk package in Sid/Testing I do indeed > see it ships a timer in /usr/lib/systtemd/ > > So, there is your answer then. Well whaddayaknow. I hadn't checked the system timers under /usr/lib/systemd/ And that is the bastard running the ghost job. Surely packaging a system timer file like that should not be done? It should be at the discretion of the admin to create one if they want to run btrbk on a schedule. Regards, Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
Mart van de Wege wrote: > Nope, not ephemeral at all, it's PID 1. Since I don't have timers > running this job, apparently there's a zombie process somewhere? PID 1 hints at a systemd.timer, even if you have dismissed this previously. Also the start time of just after midnight hints at this. And when I look at the current btrbk package in Sid/Testing I do indeed see it ships a timer in /usr/lib/systtemd/ So, there is your answer then. Also check via "systemctl list-timers". S° -- Sigmentation fault. Core dumped.
Re: Ghost cronjob
Mart van de Wege writes: > Stefan Monnier writes: > >> Mart van de Wege [2021-05-03 20:11:25] wrote: >>> Stefan Monnier writes: > root@galahad:~# grep btrbk /etc/ -rl Have you `grep`d in `/var/` as well? [ E.g. `/var/spool/crontabs` ] >>> Yep, nothing there, aside from the usual suspects (apt & dpkg files). >>> > And yet I find this in /var/log/btrbk.log: > > 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command > line client, version 0.24.0 Any other mention of activity around that time in some other log file? >>> Not that I can see. I am going to see what patching btrbk to log PPID >>> shows up tonight. >> >> My usual "trick" is to log a full `ps --forest -ef`. > > Hmmm. If the PPID turns out to be ephemeral, then that is a good second > option. > Nope, not ephemeral at all, it's PID 1. Since I don't have timers running this job, apparently there's a zombie process somewhere? Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
Stefan Monnier writes: > Mart van de Wege [2021-05-03 20:11:25] wrote: >> Stefan Monnier writes: root@galahad:~# grep btrbk /etc/ -rl >>> >>> Have you `grep`d in `/var/` as well? >>> [ E.g. `/var/spool/crontabs` ] >>> >> Yep, nothing there, aside from the usual suspects (apt & dpkg files). >> And yet I find this in /var/log/btrbk.log: 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line client, version 0.24.0 >>> >>> Any other mention of activity around that time in some other log file? >>> >>> >> Not that I can see. I am going to see what patching btrbk to log PPID >> shows up tonight. > > My usual "trick" is to log a full `ps --forest -ef`. Hmmm. If the PPID turns out to be ephemeral, then that is a good second option. > [ Instead of patching, you can rename it and replace it with a script > that runs the "real" btrbk. ] > It's a Perl script, I automatically started editing it as I am pretty fluent in Perl. Wrapper scripts are almost always my second thought. Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
On Mon, May 03, 2021 at 02:37:36PM -0400, Stefan Monnier wrote: > Mart van de Wege [2021-05-03 20:11:25] wrote: [...] > > Not that I can see. I am going to see what patching btrbk to log PPID > > shows up tonight. > > My usual "trick" is to log a full `ps --forest -ef`. > [ Instead of patching, you can rename it and replace it with a script > that runs the "real" btrbk. ] A nice trick, indeed. Cheers - t signature.asc Description: Digital signature
Re: Ghost cronjob
Mart van de Wege [2021-05-03 20:11:25] wrote: > Stefan Monnier writes: >>> root@galahad:~# grep btrbk /etc/ -rl >> >> Have you `grep`d in `/var/` as well? >> [ E.g. `/var/spool/crontabs` ] >> > Yep, nothing there, aside from the usual suspects (apt & dpkg files). > >>> And yet I find this in /var/log/btrbk.log: >>> >>> 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line >>> client, version 0.24.0 >> >> Any other mention of activity around that time in some other log file? >> >> > Not that I can see. I am going to see what patching btrbk to log PPID > shows up tonight. My usual "trick" is to log a full `ps --forest -ef`. [ Instead of patching, you can rename it and replace it with a script that runs the "real" btrbk. ] Stefan
Re: Ghost cronjob
Stefan Monnier writes: >> root@galahad:~# grep btrbk /etc/ -rl > > Have you `grep`d in `/var/` as well? > [ E.g. `/var/spool/crontabs` ] > Yep, nothing there, aside from the usual suspects (apt & dpkg files). >> And yet I find this in /var/log/btrbk.log: >> >> 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line >> client, version 0.24.0 > > Any other mention of activity around that time in some other log file? > > Not that I can see. I am going to see what patching btrbk to log PPID shows up tonight. Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
> root@galahad:~# grep btrbk /etc/ -rl Have you `grep`d in `/var/` as well? [ E.g. `/var/spool/crontabs` ] > And yet I find this in /var/log/btrbk.log: > > 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line client, > version 0.24.0 Any other mention of activity around that time in some other log file? Stefan
Re: Ghost cronjob
David Wright writes: > > Just guessing. You set the cron job to initiate a backup at 04:00. > Perhaps there's something configured in your /etc/btrbk/btrbk.conf > that says check for retention by day/week/month/year rather than > 04:00/day/week/month/year. The former check has to made at midnight. > Nope. btrbk is very simple, it just checks what snapshots are already there, and if there is already a snapshot in your snapshot directory of the form . it just adds a snapshot ._. Otherwise it just completes its entire run, creating snapshots, and sending the diffs between snapshots to a backup destination at whatever time you run it. And the logs show a full run at 00:00, when there is nothing configured to kick that off. And if I activate that 04:00 cronjob, I get two full runs in my logs, with the attendant _1 snapshots in my snapshot directory. Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
On Mon 03 May 2021 at 09:07:26 (+0200), Mart van de Wege wrote: > I have the following configured to back up my laptop to my file server: > > root@galahad:~# cat /etc/cron.d/backup > MAILTO=m...@vdwege.eu > #00 04 * * * root /usr/sbin/btrbk --verbose --format=long run > > Note: it is currently disabled. > > The only other places I have anything mentioning btrbk in /etc is in > btrbk's config files: > > root@galahad:~# grep btrbk /etc/ -rl > /etc/btrbk/btrbk.conf.example > /etc/btrbk/btrbk.conf.old > /etc/btrbk/btrbk.conf.local > /etc/btrbk/btrbk.conf~ > /etc/btrbk/btrbk.conf > /etc/cron.d/backup > > And yet I find this in /var/log/btrbk.log: > > 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line client, > version 0.24.0 > > There are no systemd timers related to btrbk either. > > This started about 1.5 months ago when I changed the backup time from > midnight to 04:00; it took me some time to notice in the status mails > that btrbk was creating duplicate snapshots, and checking the logs > showed that at ran both at midnight and at 04:00. Disabling the 04:00 > cronjob left the ghost cronjob at 00:00 intact. > > How do I find out why a periodic job is running with nothing obviously > configured for it? Just guessing. You set the cron job to initiate a backup at 04:00. Perhaps there's something configured in your /etc/btrbk/btrbk.conf that says check for retention by day/week/month/year rather than 04:00/day/week/month/year. The former check has to made at midnight. Cheers, David.
Re: Ghost cronjob
writes: > Now I do :) > > Well, no clue. But it's a script, so you could just insert some > debugging stuff (like, for example, reporting its parent PID > when it's started again)? So you might catch the ghosts parent? > > Cheers > - t > Neat idea. btrbk is pure Perl, in which I happened to be fluent. I added the relevant log messages, let's see what happens tonight. Regards, Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
On Mon, May 03, 2021 at 09:54:24AM +0200, Mart van de Wege wrote: > writes: > > > On Mon, May 03, 2021 at 09:07:26AM +0200, Mart van de Wege wrote: > >> I have the following configured to back up my laptop to my file server: > >> > >> root@galahad:~# cat /etc/cron.d/backup > >> MAILTO=m...@vdwege.eu > >> #00 04 * * * root /usr/sbin/btrbk --verbose --format=long run > >> > >> Note: it is currently disabled. > > > > [...] > > > >> And yet I find this in /var/log/btrbk.log: > >> > >> 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line > >> client, version 0.24.0 > > > > Another place to check is in /var/spool/cron/crontabs. Perhaps the > > entry slipped in via some `crontab -e' or similar. > > > Assume I checked those :) Now I do :) Well, no clue. But it's a script, so you could just insert some debugging stuff (like, for example, reporting its parent PID when it's started again)? So you might catch the ghosts parent? Cheers - t signature.asc Description: Digital signature
Re: Ghost cronjob
writes: > On Mon, May 03, 2021 at 09:07:26AM +0200, Mart van de Wege wrote: >> I have the following configured to back up my laptop to my file server: >> >> root@galahad:~# cat /etc/cron.d/backup >> MAILTO=m...@vdwege.eu >> #00 04 * * * root /usr/sbin/btrbk --verbose --format=long run >> >> Note: it is currently disabled. > > [...] > >> And yet I find this in /var/log/btrbk.log: >> >> 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line >> client, version 0.24.0 > > Another place to check is in /var/spool/cron/crontabs. Perhaps the > entry slipped in via some `crontab -e' or similar. > Assume I checked those :) root@galahad:/var/spool/cron/crontabs# grep btrbk . -r root@galahad:/var/spool/cron/crontabs# Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
Re: Ghost cronjob
On Mon, May 03, 2021 at 09:07:26AM +0200, Mart van de Wege wrote: > I have the following configured to back up my laptop to my file server: > > root@galahad:~# cat /etc/cron.d/backup > MAILTO=m...@vdwege.eu > #00 04 * * * root /usr/sbin/btrbk --verbose --format=long run > > Note: it is currently disabled. [...] > And yet I find this in /var/log/btrbk.log: > > 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line client, > version 0.24.0 Another place to check is in /var/spool/cron/crontabs. Perhaps the entry slipped in via some `crontab -e' or similar. Cheers - t signature.asc Description: Digital signature
Re: Ghost cronjob
Mart van de Wege writes: > And yet I find this in /var/log/btrbk.log: > > 2017-03-12T20:16:28+0100 startup v0.24.0 - - - - # btrbk command line client, > version 0.24.0 > Wrong logline copy/pasted, it should be this one: 2021-05-03T00:00:03+0200 startup v0.27.1 - - - # btrbk command line client, version 0.27.1 Mart -- "We will need a longer wall when the revolution comes." --- AJS, quoting an uncertain source.
