** Description changed:
+ [ Impact ]
+
+ On Ubuntu 26.04 (glibc 2.43), slurmstepd never collects job usage data.
+ sacct shows empty MaxRSS, AveRSS and TRESUsageInMax for every job, sstat
+ shows no memory and a nonsense AveCPU, and the slurmdbd database stores
+ no measured usage. Sites can't see how much memory a job used, which
+ breaks memory accounting and reporting.
+
+ The cause is sscanf(sbuf, "%d (%39c", ...) in
+ src/plugins/jobacct_gather/common/common_jag.c. glibc 2.43 changed %Nc
+ to fail when fewer than N characters are left, and process names are at
+ most 15 characters, so the call always fails and the PID is never
+ stored. Both jobacct_gather/cgroup and jobacct_gather/linux use this
+ code.
+
+ Upstream fixed it in Slurm 25.11.7 with commit 0d323a1ed8cc
+ ("jobacct_gather - Remove unused 'cmd' string", upstream issue 50879).
+ The development release already ships Slurm 26.05.4, which contains the
+ fix.
+
+ [ Test Plan ]
+
+ 1. Install slurm-wlm (slurmctld, slurmd, slurmdbd) on Ubuntu 26.04 with
JobAcctGatherType=jobacct_gather/cgroup and accounting_storage/slurmdbd.
+ 2. Submit a job that holds 300 MB for 90 seconds:
+ sbatch --wrap 'python3 -c "import time; b=bytearray(300*2**20);
time.sleep(90)"'
+ 3. While it runs: sstat -j <id>.batch -o JobID,MaxRSS,AveRSS. Expected: about
300M.
+ 4. After it ends: sacct -j <id> -o JobID,MaxRSS,AveRSS,TRESUsageInMax.
Expected: about 300M, not empty.
+ 5. Repeat with JobAcctGatherType=jobacct_gather/linux. Expected: same result.
+ 6. Check that TotalCPU, sreport and memory limits (ConstrainRAMSpace) still
work.
+
+ With the current package, steps 3 to 5 show empty values.
+
+ [ Where problems could occur ]
+
+ The patch only removes an unused field from one sscanf() call that
+ parses /proc/<pid>/stat. A mistake would show up as wrong or missing
+ per-process data in jobacct_gather (MaxRSS, CPU, disk), the same symptom
+ as today. Job scheduling, limits and slurmctld/slurmdbd don't use this
+ code. The change is the upstream fix, already released in Slurm 25.11.7
+ and 26.05.
+
+ [ Other Info ]
+
+ Debdiff for 25.11.2-1ubuntu2.1 follows in the next comment.
+
+
Package: slurm-wlm / slurmd 25.11.2-1ubuntu2 (universe, resolute release
pocket)
Ubuntu 26.04.1 LTS, glibc 2.43-2ubuntu2.4, kernel 7.0.0-34-generic, systemd
259, cgroup v2
PROBLEM
sacct and sstat never report memory usage: MaxRSS, AveRSS and TRESUsageInMax
are always empty,
for every job and every step (including long completed jobs). TotalCPU is
populated (it comes
from rusage, not from the gather plugin). sstat on a running step is empty
too, and AveCPU shows
a garbage value (213503982334-14:25:51), so slurmstepd is not delivering any
gathered data.
It happens with both JobAcctGatherType=jobacct_gather/cgroup (our production
setting) and
jobacct_gather/linux. It is not a cgroup problem: the step cgroup has
memory.current and
memory.peak, and task/cgroup enforces the limits correctly.
Relevant slurm.conf on the production nodes:
JobAcctGatherType=jobacct_gather/cgroup,
JobAcctGatherFrequency=30, ProctrackType=proctrack/cgroup,
TaskPlugin=task/affinity,task/cgroup,
AccountingStorageType=accounting_storage/slurmdbd.
EVIDENCE
With SlurmdDebug=debug and DebugFlags=JobAccountGather, slurmd logs a single
poll attempt and
then nothing for the whole step, even with --acctg-freq=task=10:
- jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded
- jobacct_gather/cgroup: jag_common_poll_data: JAG: cont_id hasn't been set
yet not running poll
- ... (no further poll until the step ends) ...
- jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded
+ jobacct_gather/cgroup: init: Job accounting gather cgroup plugin loaded
+ jobacct_gather/cgroup: jag_common_poll_data: JAG: cont_id hasn't been set
yet not running poll
+ ... (no further poll until the step ends) ...
+ jobacct_gather/cgroup: fini: Job accounting gather cgroup plugin unloaded
CAUSE
Upstream SchedMD fixed this in 25.11.7 (also in 26.05.0). The changelog says:
"Fix JobAccountGather failing on glibc 2.43+ due to a sscanf() %Nc behavior
change."
Upstream commit on branch slurm-25.11: 0d323a1ed8 "jobacct_gather - Remove
unused 'cmd' string"
(cherry-picked from master 6e2c994986, upstream issue 50879). Its message
explains that the
sscanf(sbuf, "%d (%39c", ...) in _get_process_data_line()
(src/plugins/jobacct_gather/common/common_jag.c) was broken by glibc 2.43
(glibc commit
2b16c76609): %Nc now succeeds only if N characters are available, so process
names shorter than
39 characters made it fail, leaving prec->pid unset. That produces exactly
the log line above.
The file is shared by jobacct_gather/linux and jobacct_gather/cgroup, so both
plugins are
affected (I checked this by reading the code; only the linux plugin was
tested end to end).
Resolute ships glibc 2.43 together with Slurm 25.11.2, so every Resolute
system is affected.
CONFIRMATION
Reproduced in two containers from the same ubuntu:26.04 image (glibc
2.43-2ubuntu2.4), single
node Slurm with jobacct_gather/linux, JobAcctGatherFrequency=5,
accounting_storage/none, one job
allocating 400 MB, and "sstat -j <id>.batch" while the job runs:
- - Slurm 25.11.2 (slurm-wlm from the Ubuntu archive): MaxRSS and AveRSS
empty,
- AveCPU=213503982334-14:25:51; slurmd log says "JAG: no pids in this
container".
- - Slurm 25.11.7 (upstream tarball built on the same image): MaxRSS=420708K,
AveRSS=420708K;
- the log shows the per-task "mem size/max" values.
+ - Slurm 25.11.2 (slurm-wlm from the Ubuntu archive): MaxRSS and AveRSS
empty,
+ AveCPU=213503982334-14:25:51; slurmd log says "JAG: no pids in this
container".
+ - Slurm 25.11.7 (upstream tarball built on the same image): MaxRSS=420708K,
AveRSS=420708K;
+ the log shows the per-task "mem size/max" values.
The attached archive slurm-repro-docker.tar.gz contains the reproduction
files (Dockerfile.pkg,
Dockerfile.src, slurm.conf, cgroup.conf, run-test.sh, compose.yml) and the
upstream patch
slurm-25.11-jobacct-glibc-2.43-sscanf.patch. Usage after extracting: "docker
compose run --rm pkg"
and "docker compose run --rm src".
WHAT IS NOT AFFECTED
CPU and memory limits still work: a 2-CPU job running 30 busy loops used 2.01
cores, and a
512 MB job allocating 3 GB ended OUT_OF_MEMORY (exit 137, oom_kill in
slurmd.log).
HOW TO REPRODUCE
- sbatch --mem=1G -c1 --wrap 'python3 -c "x=bytearray(400*2**20);
x[::4096]=b\"a\"*len(x[::4096]); import time; time.sleep(70)"'
- sacct -j <jobid> -o JobID,MaxRSS,AveRSS,TRESUsageInMax # all empty
- sstat -j <jobid>.batch -o JobID,MaxRSS,AveCPU # while running:
empty / garbage
+ sbatch --mem=1G -c1 --wrap 'python3 -c "x=bytearray(400*2**20);
x[::4096]=b\"a\"*len(x[::4096]); import time; time.sleep(70)"'
+ sacct -j <jobid> -o JobID,MaxRSS,AveRSS,TRESUsageInMax # all empty
+ sstat -j <jobid>.batch -o JobID,MaxRSS,AveCPU # while running:
empty / garbage
Expected: MaxRSS around 400M.
REQUEST
Please update slurm-wlm in resolute (SRU) to a release containing the
upstream fix (>= 25.11.7),
or backport the one-file upstream patch to 25.11.2 (it only drops an unused
%39c field from one
sscanf, so the regression risk is low). The development series already has
26.05.4-1ubuntu1,
which includes the fix.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2169842
Title:
slurmd 25.11.2 on Resolute: job accounting gather never polls (empty
MaxRSS/sstat); upstream 25.11.7 fixes JobAccountGather on glibc 2.43
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/slurm-wlm/+bug/2169842/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
