Public bug reported:
cron: orphan list grows without bound on every database reload (no
duplicate check); floods NSS and systemd PID 1 with getpwnam() calls
every minute
SOURCE PACKAGE
cron
1) UBUNTU RELEASE (lsb_release -rd)
Description: Ubuntu 22.04.5 LTS
Release: 22.04
2) PACKAGE VERSION (apt-cache policy cron)
cron:
Installed: 3.0pl1-137ubuntu3
Candidate: 3.0pl1-137ubuntu3
Version table:
*** 3.0pl1-137ubuntu3 500
500 http://mirror.hetzner.com/ubuntu/packages jammy/main amd64 Packages
500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
100 /var/lib/dpkg/status
Also affected:
- Ubuntu 24.04.5 LTS, cron 3.0pl1-184ubuntu2 (reproduced, see below)
- By source comparison, same code: 25.04 (3.0pl1-194ubuntu1), 25.10
(3.0pl1-196ubuntu2), Debian git master
3) WHAT I EXPECTED TO HAPPEN
cron looks up each orphaned crontab (crontab file exists, getpwnam() for its
name fails) at most once per minute, and stops looking it up once the crontab
file is gone.
4) WHAT HAPPENED INSTEAD
The number of getpwnam() calls cron makes every minute grows with every
database reload and never shrinks, not even after the orphaned crontab files
are deleted. After some days cron spends about 25 seconds of every minute on
these lookups, delays all jobs by that time and, via nss-systemd, makes PID 1
log "Looping too fast. Throttling execution a little." every minute.
ROOT CAUSE
debian/patches/features/Check-orphaned-crontabs-for-adoption.patch keeps a
linked list of orphaned crontabs. Two problems in that code:
a. add_orphan() never checks whether the crontab is already on the list.
process_crontab() calls it on every database reload, so each reload appends
every orphaned crontab again.
b. Entries are only removed when getpwnam() succeeds. Deleting the orphaned
crontab file does not remove its entries.
check_orphans() runs every time cron wakes up (once per minute) and
calls getpwnam() for every list entry. On a host where the database is
reloaded often, the list grows by (number of orphaned crontabs) per
reload, without limit, until cron is restarted.
IMPACT SEEN IN PRODUCTION
Two hosting servers (Plesk, Ubuntu 22.04.5), each with a few orphaned crontabs
left over from a migration, and with /etc/cron.d or the spool directory
changing every few minutes.
Server A, 13 orphaned crontabs, cron running for 9 days 19 hours:
- cron main process: 24 h 40 min CPU time, 29 MB RSS.
- 52,091 "ORPHAN (no passwd entry)" lines in the current /var/log/syslog.
- strace on the cron main process over one minute boundary: 11,955 connect()
calls to /run/systemd/userdb/io.systemd.DynamicUser within 35 s, spread evenly
over the 13 orphan names (919 or 920 each). The orphaned files had already been
deleted hours earlier.
- With "passwd: files systemd" in nsswitch.conf every failed lookup reaches PID
1 via nss-systemd. PID 1 debug log: 3,125 GetUserRecord requests in 15 s, all
answered with NoRecordFound.
- PID 1 logged "Looping too fast. Throttling execution a little." 8 to 10 times
per minute, every minute, from second :02 to about :24.
- All cron jobs started late: median start second :26 instead of :01, because
check_orphans() runs before the jobs are started.
Server B showed the same pattern with more than 30,000 lookups in 35 s.
"systemctl restart cron" (after removing the orphaned crontabs) stopped
it at once on both servers: "Looping too fast" messages in 3 minutes
went from 10 to 0 and from 28 to 0.
After a reboot the problem was absent for hours to days and then came
back, which matches a list that starts empty and grows.
STEPS TO REPRODUCE (tested on Ubuntu 24.04.5, cron 3.0pl1-184ubuntu2)
1. Create five orphaned crontabs:
cd /var/spool/cron/crontabs
for i in 1 2 3 4 5; do echo '* * * * * true' > ghost$i; chown root:crontab
ghost$i; chmod 600 ghost$i; done
2. Run cron under strace:
strace -tt -e trace=openat -o /tmp/cron.trace /usr/sbin/cron -f &
3. Force one reload per minute, five times (run at second :30 of each minute):
touch /var/spool/cron/crontabs
4. Delete the orphaned crontabs:
rm /var/spool/cron/crontabs/ghost*
5. Wait two more minutes, then count passwd lookups per minute:
grep '"/etc/passwd"' /tmp/cron.trace | cut -c1-5 | sort | uniq -c
Result, lookups by the cron main process per minute after the start-up minute:
10, 15, 20, 25, 30 (grows by 5 with every reload)
30, 30 (after the files were deleted: stays at 30)
SUGGESTED FIX
cronie, where this patch was taken from, received the same report in 2015 and
added a duplicate check: add_orphan() first calls a new find_orphan() and
returns if the entry already exists (cronie-1.4.12-refresh-users.patch,
changelog "do not add already existing orphan on reload"). Porting that check
fixes problem a. For problem b, clearing the orphan list before each full
database load (the reload re-adds every orphan that still exists) would cover
both problems.
WORKAROUND
Remove or fix the orphaned crontabs in /var/spool/cron/crontabs, then restart
cron. Removing the files alone is not enough.
REFERENCES
- cronie-devel, "Cronie bug (handling of orphans list)", Feb 2015:
https://lists.fedorahosted.org/archives/list/[email protected]/thread/TD4DKVVZEOCKYJUF7KEOS5OJDCXQMZ7X/
- cronie fix as shipped in the RHEL/AlmaLinux package:
https://git.almalinux.org/rpms/cronie/src/commit/847f386cc7baa45e9eec46fe2c294a838922a9de/cronie-1.4.12-refresh-users.patch
- Bugs that introduced the orphan list: Debian #634926, LP: #27520
HOW THIS REPORT WAS PRODUCED
This bug was tracked down together with an AI model (Claude, by Anthropic). All
production measurements were taken by me on my own servers. The assistant did
the analysis of the cron source, wrote the diagnostic script and the
reproducer, ran the reproducer on Ubuntu 24.04 in its own test environment and
drafted this report. I have reviewed the text; the fix result on both servers
is my own observation.
** Affects: cron (Ubuntu)
Importance: Undecided
Status: New
** Tags: cron
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174449
Title:
cron: orphan list grows without bound on every database reload (no
duplicate check)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cron/+bug/2174449/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs