Public bug reported:

cron: orphan list grows without bound on every database reload (no
duplicate check); floods NSS and systemd PID 1 with getpwnam() calls
every minute

SOURCE PACKAGE
cron

1) UBUNTU RELEASE (lsb_release -rd)
Description:    Ubuntu 22.04.5 LTS
Release:        22.04

2) PACKAGE VERSION (apt-cache policy cron)
cron:
  Installed: 3.0pl1-137ubuntu3
  Candidate: 3.0pl1-137ubuntu3
  Version table:
 *** 3.0pl1-137ubuntu3 500
        500 http://mirror.hetzner.com/ubuntu/packages jammy/main amd64 Packages
        500 http://de.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status

Also affected:
- Ubuntu 24.04.5 LTS, cron 3.0pl1-184ubuntu2 (reproduced, see below)
- By source comparison, same code: 25.04 (3.0pl1-194ubuntu1), 25.10 
(3.0pl1-196ubuntu2), Debian git master

3) WHAT I EXPECTED TO HAPPEN
cron looks up each orphaned crontab (crontab file exists, getpwnam() for its 
name fails) at most once per minute, and stops looking it up once the crontab 
file is gone.

4) WHAT HAPPENED INSTEAD
The number of getpwnam() calls cron makes every minute grows with every 
database reload and never shrinks, not even after the orphaned crontab files 
are deleted. After some days cron spends about 25 seconds of every minute on 
these lookups, delays all jobs by that time and, via nss-systemd, makes PID 1 
log "Looping too fast. Throttling execution a little." every minute.


ROOT CAUSE
debian/patches/features/Check-orphaned-crontabs-for-adoption.patch keeps a 
linked list of orphaned crontabs. Two problems in that code:

a. add_orphan() never checks whether the crontab is already on the list. 
process_crontab() calls it on every database reload, so each reload appends 
every orphaned crontab again.
b. Entries are only removed when getpwnam() succeeds. Deleting the orphaned 
crontab file does not remove its entries.

check_orphans() runs every time cron wakes up (once per minute) and
calls getpwnam() for every list entry. On a host where the database is
reloaded often, the list grows by (number of orphaned crontabs) per
reload, without limit, until cron is restarted.

IMPACT SEEN IN PRODUCTION
Two hosting servers (Plesk, Ubuntu 22.04.5), each with a few orphaned crontabs 
left over from a migration, and with /etc/cron.d or the spool directory 
changing every few minutes.

Server A, 13 orphaned crontabs, cron running for 9 days 19 hours:
- cron main process: 24 h 40 min CPU time, 29 MB RSS.
- 52,091 "ORPHAN (no passwd entry)" lines in the current /var/log/syslog.
- strace on the cron main process over one minute boundary: 11,955 connect() 
calls to /run/systemd/userdb/io.systemd.DynamicUser within 35 s, spread evenly 
over the 13 orphan names (919 or 920 each). The orphaned files had already been 
deleted hours earlier.
- With "passwd: files systemd" in nsswitch.conf every failed lookup reaches PID 
1 via nss-systemd. PID 1 debug log: 3,125 GetUserRecord requests in 15 s, all 
answered with NoRecordFound.
- PID 1 logged "Looping too fast. Throttling execution a little." 8 to 10 times 
per minute, every minute, from second :02 to about :24.
- All cron jobs started late: median start second :26 instead of :01, because 
check_orphans() runs before the jobs are started.

Server B showed the same pattern with more than 30,000 lookups in 35 s.

"systemctl restart cron" (after removing the orphaned crontabs) stopped
it at once on both servers: "Looping too fast" messages in 3 minutes
went from 10 to 0 and from 28 to 0.

After a reboot the problem was absent for hours to days and then came
back, which matches a list that starts empty and grows.

STEPS TO REPRODUCE (tested on Ubuntu 24.04.5, cron 3.0pl1-184ubuntu2)
1. Create five orphaned crontabs:
     cd /var/spool/cron/crontabs
     for i in 1 2 3 4 5; do echo '* * * * * true' > ghost$i; chown root:crontab 
ghost$i; chmod 600 ghost$i; done
2. Run cron under strace:
     strace -tt -e trace=openat -o /tmp/cron.trace /usr/sbin/cron -f &
3. Force one reload per minute, five times (run at second :30 of each minute):
     touch /var/spool/cron/crontabs
4. Delete the orphaned crontabs:
     rm /var/spool/cron/crontabs/ghost*
5. Wait two more minutes, then count passwd lookups per minute:
     grep '"/etc/passwd"' /tmp/cron.trace | cut -c1-5 | sort | uniq -c

Result, lookups by the cron main process per minute after the start-up minute:
     10, 15, 20, 25, 30    (grows by 5 with every reload)
     30, 30                (after the files were deleted: stays at 30)

SUGGESTED FIX
cronie, where this patch was taken from, received the same report in 2015 and 
added a duplicate check: add_orphan() first calls a new find_orphan() and 
returns if the entry already exists (cronie-1.4.12-refresh-users.patch, 
changelog "do not add already existing orphan on reload"). Porting that check 
fixes problem a. For problem b, clearing the orphan list before each full 
database load (the reload re-adds every orphan that still exists) would cover 
both problems.

WORKAROUND
Remove or fix the orphaned crontabs in /var/spool/cron/crontabs, then restart 
cron. Removing the files alone is not enough.

REFERENCES
- cronie-devel, "Cronie bug (handling of orphans list)", Feb 2015:
  
https://lists.fedorahosted.org/archives/list/[email protected]/thread/TD4DKVVZEOCKYJUF7KEOS5OJDCXQMZ7X/
- cronie fix as shipped in the RHEL/AlmaLinux package:
  
https://git.almalinux.org/rpms/cronie/src/commit/847f386cc7baa45e9eec46fe2c294a838922a9de/cronie-1.4.12-refresh-users.patch
- Bugs that introduced the orphan list: Debian #634926, LP: #27520

HOW THIS REPORT WAS PRODUCED
This bug was tracked down together with an AI model (Claude, by Anthropic). All 
production measurements were taken by me on my own servers. The assistant did 
the analysis of the cron source, wrote the diagnostic script and the 
reproducer, ran the reproducer on Ubuntu 24.04 in its own test environment and 
drafted this report. I have reviewed the text; the fix result on both servers 
is my own observation.

** Affects: cron (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: cron

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174449

Title:
  cron: orphan list grows without bound on every database reload (no
  duplicate check)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cron/+bug/2174449/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to