Re: [gentoo-user] Bug in run-crons?
On 13/12/2021 22:03, Frank Steinmetzger wrote: If they are involved multiple times with the default options I think any attempt to scrub something that is already being scrubbed is just a no-op. Obviously if you don't want all that IO during the day you'll have to do something more clever - you can instruct it to pause and resume so you could have a couple of crontab entries and scripts to do just that. Or, since I am the only user of that system, I could go back to my previous way: just run the scrub manually every other month or so before I go to bed. Because then I know that nothing will interfere with it and it won’t interfere with anything else. Yup. I was thinking that. The problem with scrubs et al auto-resuming or firing at boot is that disk i/o is then knackered for ages, and interferes with whatever it is you're actually trying to do :-) Cheers, Wol
Re: [gentoo-user] Bug in run-crons?
Am Mon, Dec 13, 2021 at 04:54:30PM -0500 schrieb Rich Freeman:
> On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky wrote:
> >
> > On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
> > >
> > > Well I *could* disable run-crons altogether and add entries to fcron’s own
> > > crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> > > instead.
> > >
> > > However, I like predictable times at which those jobs will run. Especially
> > > if one of them is a zfs scrub; the NAS is powered down for weeks,
> > > sometimes
> > > months. And when I power it up, it’s for a reason. And that reason usually
> > > is not a scrub, which—at the current zfs fill level—takes 10½ hours.
> > >
> >
> > Why choose fcron then? It sounds like you have the same rationale as I
> > do: "no, I don't want to run the 4am backup job in the middle of the
> > business day just because it wasn't run at 4am."
Fair point.
Fcron has a serialisation feature, so that for instance updatedb and mandb
don’t run at the same time. Me kinda liky.
> fcron is perfectly capable of running jobs at either set times or at
> loosely-defined intervals that are compensated for if the machine is
> off.
Indeed, I can say "@monthly * 0 *" to run a monthly job (like the scrub)
only at night. *pondering*
> Systemd timers can also be set that way, and can also have
> limits put on scheduling (daily nominally at 3AM but allowed between
> 11PM and 5AM or whatever). Obviously the more expressive the crontab
> equivalent is, the more you can tweak it to do what you want it to.
I’ve gotten used to systemd on my Arch-based desktops. But the NAS is still
on good-ol’ openrc. :)
> I'll also note that zfs scrubs checkpoint at shutdown and will
> auto-resume where you leave off.
That’s a nice bit of knowledge.
> If they are involved multiple times
> with the default options I think any attempt to scrub something that
> is already being scrubbed is just a no-op. Obviously if you don't
> want all that IO during the day you'll have to do something more
> clever - you can instruct it to pause and resume so you could have a
> couple of crontab entries and scripts to do just that.
Or, since I am the only user of that system, I could go back to my previous
way: just run the scrub manually every other month or so before I go to bed.
Because then I know that nothing will interfere with it and it won’t
interfere with anything else.
--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.
This message was written using only recycled electrons.
signature.asc
Description: PGP signature
Re: [gentoo-user] Bug in run-crons?
On Mon, Dec 13, 2021 at 4:42 PM Michael Orlitzky wrote:
>
> On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
> >
> > Well I *could* disable run-crons altogether and add entries to fcron’s own
> > crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> > instead.
> >
> > However, I like predictable times at which those jobs will run. Especially
> > if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
> > months. And when I power it up, it’s for a reason. And that reason usually
> > is not a scrub, which—at the current zfs fill level—takes 10½ hours.
> >
>
> Why choose fcron then? It sounds like you have the same rationale as I
> do: "no, I don't want to run the 4am backup job in the middle of the
> business day just because it wasn't run at 4am."
>
> If you pick a dumber cron, the crontab entries are run only at the
> specified times.
There really are no one-size-fits all solutions obviously. You can
either have predictable times and if the machine is down things get
skipped, or you can run opportunistically in which case you lose
predictable times at least until the machine goes back to running
24x7. The machine has no way to know if you'll leave it on overnight
the NEXT night - only if you did so the one before.
fcron is perfectly capable of running jobs at either set times or at
loosely-defined intervals that are compensated for if the machine is
off. Systemd timers can also be set that way, and can also have
limits put on scheduling (daily nominally at 3AM but allowed between
11PM and 5AM or whatever). Obviously the more expressive the crontab
equivalent is, the more you can tweak it to do what you want it to.
The default configs are a one-size-fits all approach that tend to do
the right thing if you leave the machine on 24x7, or if you run it at
random times. If you want to specify exact behavior just edit the
config files and make it do so.
I'll also note that zfs scrubs checkpoint at shutdown and will
auto-resume where you leave off. If they are involved multiple times
with the default options I think any attempt to scrub something that
is already being scrubbed is just a no-op. Obviously if you don't
want all that IO during the day you'll have to do something more
clever - you can instruct it to pause and resume so you could have a
couple of crontab entries and scripts to do just that.
--
Rich
Re: [gentoo-user] Bug in run-crons?
On Mon, 2021-12-13 at 22:38 +0100, Frank Steinmetzger wrote:
>
> Well I *could* disable run-crons altogether and add entries to fcron’s own
> crontab which would run those scripts in /etc/cron.{hourly,daily,...}
> instead.
>
> However, I like predictable times at which those jobs will run. Especially
> if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
> months. And when I power it up, it’s for a reason. And that reason usually
> is not a scrub, which—at the current zfs fill level—takes 10½ hours.
>
Why choose fcron then? It sounds like you have the same rationale as I
do: "no, I don't want to run the 4am backup job in the middle of the
business day just because it wasn't run at 4am."
If you pick a dumber cron, the crontab entries are run only at the
specified times.
Re: [gentoo-user] Bug in run-crons?
Am Mon, Dec 13, 2021 at 04:33:32PM -0500 schrieb Michael Orlitzky:
> On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote:
> >
> > For the record: The checks in run-crons that I referred to earlier are
> > actually more for those cases in which the machine was powered off for a
> > while in order to restore cron completeness as early as possible after boot.
> >
>
> The run-crons quackery has been causing problems since 2004:
>
> https://bugs.gentoo.org/69777
>
> One-liners with run-parts (NOT run-crons) are a lot more predictable.
Well I *could* disable run-crons altogether and add entries to fcron’s own
crontab which would run those scripts in /etc/cron.{hourly,daily,...}
instead.
However, I like predictable times at which those jobs will run. Especially
if one of them is a zfs scrub; the NAS is powered down for weeks, sometimes
months. And when I power it up, it’s for a reason. And that reason usually
is not a scrub, which—at the current zfs fill level—takes 10½ hours.
--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.
A preposition is something you should never end a sentence with.
signature.asc
Description: PGP signature
Re: [gentoo-user] Bug in run-crons?
On Mon, 2021-12-13 at 22:19 +0100, Frank Steinmetzger wrote: > > For the record: The checks in run-crons that I referred to earlier are > actually more for those cases in which the machine was powered off for a > while in order to restore cron completeness as early as possible after boot. > The run-crons quackery has been causing problems since 2004: https://bugs.gentoo.org/69777 One-liners with run-parts (NOT run-crons) are a lot more predictable.
Re: [gentoo-user] Bug in run-crons?
Am Mon, Dec 13, 2021 at 03:18:44PM -0500 schrieb Rich Freeman: > On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger wrote: > > > > Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman: > > > On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger wrote: > > > > > > > > > > I don't use this, but I believe there should be an hourly crontab > > > entry that deletes the cron.hourly file, which would mean it gets run > > > on the next 10min cycle (or maybe sooner - I'm not sure if those jobs > > > are run in parallel or serial). > > > > The check that I mentioned above is actually the deletion which you mention: > > run-crons looks for the state file for the given interval and - if it is old > > enough - deletes it. > > The check I'm talking about isn't in run-crons at all. It is in > /etc/crontab. It doesn't look at the age of the file and > unconditionally deletes it every hour: > 59 * * * *rm -f /var/spool/cron/lastrun/cron.hourly I had a look at files and docs on the net again. Thus I found exactly those rm entries in /etc/crontab, which by itself is not used by fcron. But after I understood all the logic behind it, I added them to fcron to be run serially before run-crons. Now everything is as I wanted it. For the record: The checks in run-crons that I referred to earlier are actually more for those cases in which the machine was powered off for a while in order to restore cron completeness as early as possible after boot. -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. “Selfies are electronic masturbation.” — Karl Lagerfeld signature.asc Description: PGP signature
Re: [gentoo-user] Bug in run-crons?
On Sun, Dec 12, 2021 at 2:07 PM Frank Steinmetzger wrote: > > Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman: > > On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger wrote: > > > > > > > I don't use this, but I believe there should be an hourly crontab > > entry that deletes the cron.hourly file, which would mean it gets run > > on the next 10min cycle (or maybe sooner - I'm not sure if those jobs > > are run in parallel or serial). > > The check that I mentioned above is actually the deletion which you mention: > run-crons looks for the state file for the given interval and - if it is old > enough - deletes it. The check I'm talking about isn't in run-crons at all. It is in /etc/crontab. It doesn't look at the age of the file and unconditionally deletes it every hour: 59 * * * *rm -f /var/spool/cron/lastrun/cron.hourly -- Rich
Re: [gentoo-user] Bug in run-crons?
Am Sun, Dec 12, 2021 at 01:41:33PM -0500 schrieb Rich Freeman: > On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger wrote: > > > > It uses state files in /var/spool/cron/lastrun/ to know when each interval > > was last run, so it only runs once per period. But: the age threshold for > > the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes? > > > > My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at > > the next run, the state file is exactly one hour old. Since this is not old > > enough for the check, run-crons thinks that the last run is too recent and > > ignores this period. As a result, each period is only run on every other > > iteration. > > I don't use this, but I believe there should be an hourly crontab > entry that deletes the cron.hourly file, which would mean it gets run > on the next 10min cycle (or maybe sooner - I'm not sure if those jobs > are run in parallel or serial). The check that I mentioned above is actually the deletion which you mention: run-crons looks for the state file for the given interval and - if it is old enough - deletes it. The part that executes the individual cron scripts is only executed if there is no state file. The first thing it then does is to create a new state file. In pseudo code: 1. look for period state file that is older than interval + 5 mins found one? delete it 2. Look for period state file none found? create state file execute cron scripts for that interval -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. The boss is a human just like everyone else, he just doesn’t know. signature.asc Description: PGP signature
Re: [gentoo-user] Bug in run-crons?
On Sun, Dec 12, 2021 at 1:21 PM Frank Steinmetzger wrote: > > It uses state files in /var/spool/cron/lastrun/ to know when each interval > was last run, so it only runs once per period. But: the age threshold for > the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes? > > My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at > the next run, the state file is exactly one hour old. Since this is not old > enough for the check, run-crons thinks that the last run is too recent and > ignores this period. As a result, each period is only run on every other > iteration. I don't use this, but I believe there should be an hourly crontab entry that deletes the cron.hourly file, which would mean it gets run on the next 10min cycle (or maybe sooner - I'm not sure if those jobs are run in parallel or serial). The overall result is that things will get run at traditional crontab times no matter what, but if those are missed and enough time is accrued then run-crons will catch this and run the job on the next 10min cycle, such as a missed daily overnight slot. So if you don't have your computer on a 3AM then the job will get run a bit more than 24h after the last time it was run. Then if you leave it on overnight that night it will run at 3AM despite it not being 24h, but at that point if you leave it on it will run at 3AM daily since it will no longer be missed. The fixed times serve to anchor the actual execution times if you do leave the computer on. -- Rich
[gentoo-user] Bug in run-crons?
Hi, it’s me again, :)
I’m currently improving the cron setup on my Gentoo-powered NAS. I installed
zfs-auto-snapshot, which includes scripts for hourly, daily, weekly and
monthly creation of snapshots.
The NAS uses fcron for its cron daemon. In order for the scripts to work,
I enabled the run-crons script from sys-process/cronbase, which runs all
scripts in /etc/cron.{hourly,daily,...}. But I have the feeling that there
is a bug in the script:
It uses state files in /var/spool/cron/lastrun/ to know when each interval
was last run, so it only runs once per period. But: the age threshold for
the state file is period + 5 minutes. Shouldn’t that be period - 5 minutes?
My reasoning: assume run-crons is run hourly, at the 0 minute sharp. So at
the next run, the state file is exactly one hour old. Since this is not old
enough for the check, run-crons thinks that the last run is too recent and
ignores this period. As a result, each period is only run on every other
iteration.
Any thoughts on that?
Cheerio.
--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.
“Today I watched my first porn movie.” – “And?” – “I was so young back then.”
signature.asc
Description: PGP signature
