# Second mm failure mode on 6.17.0-1022: kcompactd0 NULL deref in the page-migration path
Reporting a new and distinct failure on the same fleet described in the original report. This one is **not a panic** — a kernel thread oopsed and died, leaving the system running but permanently broken. That makes it more informative than our vmcore, because the faulting function and address are named. **Host:** `vm-prd-4` — Azure `Standard_E2as_v5` (2 vCPU, 15.6 GB, AMD EPYC 7763), eastus; Ubuntu 24.04.3 LTS. One of the four production VMs in the original report, and one of the hosts that ran 396 consecutive days without incident on 6.8.0-1018-azure. **Kernel:** 6.17.0-1022-azure #22-Ubuntu **Date:** 2026-10-08 18:21:04 UTC **Uptime at failure:** 30 days (booted 2026-09-07 20:00:58 UTC) ## The oops ``` CPU: 1 UID: 0 PID: 37 Comm: kcompactd0 Not tainted 6.17.0-1022-azure #22-Ubuntu VOLUNTARY try_to_migrate+0x8f/0x140 ? __pfx_try_to_migrate_one+0x10/0x10 ? __pfx_invalid_migration_vma+0x10/0x10 migrate_folio_unmap+0x245/0x2f0 migrate_pages_batch+0x162/0x820 ? __pfx_compaction_alloc+0x10/0x10 ? __pfx_compaction_free+0x10/0x10 migrate_pages_sync+0x83/0x1d0 migrate_pages+0x37c/0x480 compact_zone+0x40f/0x6a0 compact_node+0xaf/0x130 kcompactd+0x369/0x4d0 ? __pfx_kcompactd+0x10/0x10 ret_from_fork+0xf2/0x100 ret_from_fork_asm+0x1a/0x30 RIP: 0010:down_read_trylock+0x13/0x60 RDI: 0000000000000008 CR2: 0000000000000008 ---[ end trace 0000000000000000 ]--- sched: DL replenish lagged too much note: kcompactd0[37] exited with irqs disabled ``` `RDI` is the `struct rw_semaphore *` argument to `down_read_trylock`, and it is `0x8` — a NULL structure plus a member offset. `CR2` confirms the fault address. The register dump repeats ~19 s after the first trace (kernel timestamps 2672390 and 2672409) before the thread exits, which may be a second fault during oops teardown with IRQs already disabled; we have not tried to separate those. ## Consequence: unkillable tasks, then the node leaves the cluster Page migration installs migration entries in the page tables and clears them on completion. kcompactd0 died **mid-batch**, so those entries were never cleared. Every subsequent fault on an affected page blocks permanently. Eight seconds later the Docker daemon lost its Swarm raft leadership and stopped logging entirely — it never emitted another line. Inspecting it live, 2 of its 28 threads were parked here indefinitely: ``` migration_entry_wait_on_locked+0x251/0x2c0 migration_entry_wait+0xc6/0xf0 do_swap_page+0xe07/0x11f0 handle_pte_fault+0x190/0x1d0 __handle_mm_fault+0x2a4/0x6e0 handle_mm_fault+0x10d/0x2d0 do_user_addr_fault+0x1b9/0x860 exc_page_fault+0x78/0x150 ``` Thread states across the process: 12 `wait_for_partner`, 7 `futex_do_wait`, 6 `do_wait`, **2 `migration_entry_wait_on_locked`**, 1 `hrtimer_nanosleep`. The futex and pipe waits are downstream — threads queued behind the two that could not complete their faults. Recovery was not possible from userspace: - `SIGTERM` could not be serviced; systemd escalated to `SIGKILL` at its 90 s timeout - `SIGKILL` cannot terminate a task in uninterruptible sleep, so the two threads survived - the thread-group leader became a zombie that `init` could never reap - the stale pidfile then blocked every daemon restart attempt - a clean `reboot` reached `reboot.target` and then **failed to reset**, with `CIFS: VFS: ... has not responded in 180 seconds` and `CIFS: Serverclose failed 4 times, giving up` - the VM only recovered via a hypervisor-level deallocate/start ## Why we think this is worth adding here - Same fleet and same kernel series as the original report, on hosts that previously ran 396 consecutive days without incident on 6.8.0-1018-azure. - Both failures are in memory management. The panic was RCU invoking a callback on a freed VMAP'd kernel stack with maple-tree VMA teardown frames; this is compaction/migration. We are **not** claiming they are the same defect — the mechanisms differ and we cannot bridge them from this evidence. We offer it as a second data point in the same subsystem. - An oops with a named function and faulting address is stronger evidence than our vmcore, which only yielded stale stack remnants. ## Limits of this evidence - **Single occurrence.** We have one instance of this signature, not a pattern. - **The link between the oops and the stuck threads is inferred.** Timing (8 s) and mechanism fit, but proving it needs a PTE dump from the live hung system, and that system is gone. - **The host was under memory pressure.** The same console log shows three prior OOM kills of a container on this node at 11.8–13.6 GB anon-rss. Compaction activity is expected under fragmentation, so pressure plausibly raised exposure. We have not measured compaction rate against that workload, so we cannot say whether pressure is required to trigger the bug or merely makes it more likely. - **No 6.8 comparison for this signature.** We never observed kcompactd oopsing on 6.8, but we were not watching for it, so its absence there is not evidence. ## Status of the 6.8 test The `linux-azure-6.8` migration described at the end of the original report has now completed its soak: five hosts, 69.29 node-days, zero panics and zero precursors. Full numbers are in the companion comment posted alongside this one. Note that the host in this report is now running `7.0.0-1014-azure` following the forced recovery described above — that was not a deliberate choice, and production has not yet been migrated to the GA series. ## Attachments available on request - full per-thread kernel stacks from the live hang (28 threads) - serial console log covering the oops, the subsequent hang, and the failed reboot -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2168861 Title: linux-azure: kernel panic in rcu_do_batch (NULL rcu_head on freed VMAP stack) — regression between 6.8 and 6.14 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2168861/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
