Following up on my previous two comments, because comment #39 changes how I
read my own numbers and suggests a test I do not think anyone here has run.
Eivind reported that locking to 600MHz fixes it. 600MHz is the *minimum*, so
the thing that helped may not be running slowly - it may be not changing
frequency at all. If that is right, the variable is the DVFS transition rate,
not the clock speed and not the TX load.
That would also explain my report of stalls at essentially idle TX. A box
under sustained iperf pins at max and mostly stays there. An idle box that
still has periodic small work - swarm gossip every few seconds, in my case -
ramps up and down constantly, so it sees far more frequency transitions than
a loaded one. My three nodes were all on "ondemand" swinging 1500-2400MHz.
So "stalls when idle" and "stalls under bulk transfer" may not be two
behaviours; idle may simply be the higher-transition case.
The test that seems to be missing is a constant *maximum* rather than a
constant minimum. Eivind locked to the floor and reasonably objected that it
makes the hardware pointless. Sam tried flipping governors (comment #26) but
reported the frequency did not actually move. Pinning to the stock maximum
would separate "constant" from "slow", and if it works it costs nothing in
performance and needs no overclocking.
I have that running now, split so I can tell the two workarounds apart:
swarm01 governor=performance (pinned 2400MHz, stock max, no overclock)
offloads left at default: tso on, sg on
swarm02 ethtool -K eth0 tso off sg off, governor left at ondemand
swarm03 ethtool -K eth0 tso off sg off, governor left at ondemand
Started 2026-09-02 19:09 UTC. Verified swarm01 holds 2400MHz across repeated
samples, at 49C with the throttle point well clear, so thermal capping should
not reintroduce transitions.
Baseline to beat, from my earlier comment: 24 stalls in ~344 node-hours,
about one per 14 node-hours. swarm01 alone accounted for 10 of those 24, so
it should produce a signal within a few days either way. Detection is a 30s
poll of tx_packets with an automatic interface bounce, so I get a timestamped
record of every stall even when the workaround is in place.
I will report back with node-hours to first stall on each arm, including if
the governor arm fails - a negative result there is worth having too, since
it would point back at the TX path rather than DVFS.
If anyone with a larger fleet wants to try the same split, the governor side
is a one-liner and does not sacrifice performance the way the 600MHz lock
does:
for g in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
echo performance > "$g"
done
AI DISCLOSURE: I have been heavily assisted by Claude Fable when
diagnosing this issue
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2133877
Title:
Complete network hang on Raspberry Pi 5 with kernel 6.17 under load -
possibly related to CPU frequency scaling
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2133877/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
