On 11/6/26 15:10, Miroslav Lachman wrote:
On 05/11/2026 16:28, Darren Henderson wrote:
Considering POLA I believe it would be better as an opt-in kind of
thing rather than opt-out. If people need it they can enable it. In
general opt-out changes should be avoided. Just flipping this
around is likely to cause many many unexpected issues for people who
are not paying close attention.
Given the POLA concerns raised, I have an alternative idea:
fixed per-host offset instead of per-run random jitter.
100% agree.
Per-run jitter allows for job timing to drift, which can (will on a big
enough scale) allow for jobs to accidentally run "in phase" at
unpredictable times, which will re-create the thundering herd problem
the jitter is meant to avoid. Machine A and Machine B may randomly
choose the same offset and run their job simultaneously, without any
clear indication (load or otherwise) why.
In the fixed-offset case, Machine A would always run "every minute" jobs
at a fixed offset, where Machine B would run them on a different offset.
- cron derives a constant offset in the range 0-59 seconds from a
hash of kern.hostuuid (plus hostname, to cope with cloned VMs and
jails sharing the same hostuuid). Optionally, the user can override it
with a fixed value via cron_flags in rc.conf
In terms of choosing that offset, things like HostID, Hostname, even MAC
or IP address could be accidentally duplicated in various situations.
Cloud-init data may contain a unique instance ID. Time of last boot,
uptime of last reboot, etc are less likely to align between nodes. The
chance of this causing eventual alignment of offsets would be dependent
on uptime/reboot frequency. It would provide high entropy in cloud
environments where per-host data may be difficult, and insufficient
entropy in physical hosts where reboots are relatively rare.
Possible variant is a per-job offset, hashed from hostuuid and the
crontab entry, which would also spread jobs on a single host. Systemd
does something similar with RandomizedDelaySec= combined with
FixedRandomDelay=, derived from the machine-id.
Per-job offsets feel like a bit more of a POLA violation. If I have
three jobs that are expected to run exactly on the minute of a schedule,
the (maybe desired, maybe not) expected simultaneity of those jobs may
be lost. If per-job offsets are a default, I would recommend tighter
jitter limits across jobs so that admins can configure jobs for expected
behaviors.
Daniel
