# 6.8 soak result: 14 days, five hosts, zero panics
This is the test promised at the end of the original report, which said the
hosts would be moved
to the `linux-azure-6.8` GA track to run **current userspace against the older
kernel series**,
and that we would post the outcome either way. It has now completed.
## What was changed
On 2026-09-30 all five QAT hosts were moved from `6.17.0-1022-azure` to
`6.8.0-1068-azure`
(`linux-azure-6.8`, the 24.04 GA track). Nothing else changed: same VMs, same
Azure SKU, same
workload, same userspace, same container images. One node (qat1) was migrated a
day earlier,
2026-09-29.
## Result
```
host kernel booted exposure
qat0 6.8.0-1068-azure 2026-09-30 15:52 13.65 node-days
qat1 6.8.0-1068-azure 2026-09-29 16:22 14.63
qat2 6.8.0-1068-azure 2026-09-30 15:11 13.68
qat3 6.8.0-1068-azure 2026-09-30 15:15 13.67
qat4 6.8.0-1068-azure 2026-09-30 15:25 13.67
TOTAL 69.29 node-days (2.28
node-months)
kernel panics 0
oops / BUG 0
Bad rss-counter state 0
non-zero pgtables_bytes on freeing mm 0
hung_task / soft lockup 0
```
No host rebooted during the soak, so each node's ring buffer covers the entire
window — the zeros
are full coverage, not a truncated log.
## Against the prior rate
These same hosts produced **45 panics in the nine months** between 2025-12-26
and the migration,
roughly 1 panic per node-month. Over 2.28 node-months we therefore expected
about **2.3 panics**
and saw none.
Treating panics as Poisson with λ = 2.28, a clean run of this length would
occur by chance about
**10%** of the time if the rate were unchanged — so this alone is roughly 90%
confidence, not
proof. It is considerably stronger in combination with the pre-regression
record on the same
machines: 379 and 397 consecutive days without a panic on `6.8.0-1018-azure`.
## Why we think this isolates the kernel
The original report flagged one weakness explicitly: the 2025-12-26 reboot
changed kernel **and**
userspace together, because a manual `apt upgrade` ran minutes beforehand. That
left the
before/after boundary unable to separate the two.
This soak addresses that directly. The hosts are running **today's** userspace
— every package
updated since December, including everything in that original `apt upgrade` —
against the older
kernel series. The panics stopped. Userspace is therefore not a plausible
cause, and the remaining
difference is the kernel.
## Contemporaneous observation, not evidence
Our production fleet has **not** been migrated and remains on `6.17.0-1022` and
`7.0.0-1014`.
During the same 14-day window one of those hosts (6.17.0-1022, 4 weeks uptime)
logged:
```
2026-10-09 15:30:33 BUG: Bad rss-counter state mm:00000000741ffa1b
type:MM_ANONPAGES val:6 Comm:runc
```
That is the precursor signature described in the original report, on an
unmigrated host, while the
migrated hosts logged none. We offer this as a concurrent observation only —
one event against
one fleet is not a controlled comparison and we are not claiming it as a
statistical result.
## Limits
- **QAT only.** Production has not yet been migrated. The plan is to move it
now that this soak
has completed.
- **QAT's workload is lighter than production's.** The report identified query
load against an
mmap-heavy workload as the apparent trigger, so a clean QAT run is not
automatically predictive
of a clean production run.
- **Precursor counts are not decisive here.** The historical precursor rate
(0.019–0.082 per
node-day) predicts 1.3–5.7 events over this exposure. Zero is consistent with
the rate having
dropped, but also with the low end of the prior range. The panic count
carries the argument;
the precursor count merely does not contradict it.
- **This does not identify the defect.** It brackets it to the 6.8 → 6.14
window and shows the GA
series is unaffected; it says nothing about the underlying cause.
- **We will shortly have no host that reproduces this.** Production migrates to
the GA series next,
after which no machine in our estate runs an affected kernel. Flagging it
because it bears on
triage: our ability to supply further data from a reproducing host ends with
that migration.
## Related
A separate failure mode observed on 6.17.0-1022 during this period — a
`kcompactd0` NULL
dereference in the page-migration path, which hung the host without panicking —
is described in
a companion comment.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2168861
Title:
linux-azure: kernel panic in rcu_do_batch (NULL rcu_head on freed VMAP
stack) — regression between 6.8 and 6.14
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2168861/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
