Public bug reported:
[Impact]
When one cgroup saturates a CPU, per-CPU kernel threads on that CPU
(ksoftirqd, kworker) can wait seconds to minutes to run. Seen on GCP A4X
GB200 (arm64, linux-nvidia-6.17) during multi-node training with all MPI
ranks on CPU 0: ksoftirqd/0 and kworker/0:* waited up to several hundred
seconds, causing khugepaged hung-task warnings.
Commit 6d71a9c61604 ("sched/fair: Fix EEVDF entity placement bug causing
scheduling lag") makes reweight_entity() re-place an on_rq entity through
place_entity() using avg_vruntime() after the weight change. avg_vruntime()
already includes the current entity, so the running entity's virtual
deadline is pulled backwards on every reweight. update_cfs_group() reweights
the saturating group entity thousands of times per second, so kernel threads
in the root cgroup stay eligible but keep losing the deadline pick.
[Fix]
Upstream reverted 6d71a9c61604 in 101f3498b4bd (v7.1; linux-6.18.y as
62fefb817bb3). On this tree the revert is not safe alone: it computes the
relative deadline against the raw avg_vruntime(), which 79f3f9bedd14 (already
here) leaves unbounded for a single running task until b3d99f43c72b and
1319ea57529e are applied. Without them, (deadline - avg_vruntime) * weight can
overflow s64. linux-6.18.y applied both before the revert; this series keeps
that order:
1/3 b3d99f43c72b sched/fair: Fix zero_vruntime tracking
2/3 1319ea57529e sched/fair: Fix zero_vruntime tracking fix
3/3 101f3498b4bd sched/fair: Revert 6d71a9c61604
Adaptations are noted in each patch.
[Test Plan]
GCP a4x-highgpu-4g (GB200, arm64), stock 6.17.0-1032 vs the same tree plus
this series, built with the official nvidia-64k config (only
kernel/sched/fair.c differs). Reproducer: a root-cgroup task pinned to CPU 0
sleeping 1 ms and recording its wakeup latency; a child cgroup with 64 busy
threads pinned to CPU 0; the child's cpu.weight swung between 1 and 10000.
15 runs of 20 s per kernel; a run fails if the task or ksoftirqd/0 waits
1 s or more.
stock 6.17.0-1032: 10/15 runs failed, worst wait 186 s
with this series: 0/15 runs failed, worst wait 34 ms
No WARN, BUG, RCU stall or hung-task message on either kernel.
[Where problems could occur]
The series changes EEVDF placement in reweight_entity() and how
avg_vruntime() maintains zero_vruntime, so a regression would show as
scheduling unfairness or latency, most likely around nice changes, cgroup
weight changes, yield, or a CPU with a single runnable task. Patches 1-2
match what linux-6.18.y carries; patch 3 restores the reweight behaviour from
before 6d71a9c61604.
[Other Info]
Slice protection (vprot) across a reweight is not preserved on 6.17, because
vprot shares a union with vlag. That predates this series and is not changed
by it.
** Affects: linux-nvidia-6.17 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2171398
Title:
sched/fair: kernel threads starve when one cgroup saturates a CPU
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-6.17/+bug/2171398/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
