Public bug reported:
Ubuntu 26.04, linux-image-7.0.0-30-generic 7.0.0-30.30, on a LENOVO 82VG
(IdeaPad 1, BIOS KSCN31WW 05/09/2024), amd64.
The machine hard-locked in the middle of an unattended-upgrades run. The
crash reproduced a second time during manual recovery. Both crashes happened
in snapd, in the epoll teardown path, and both were tied to the
snapd 2.76 -> 2.76.3 package transition -- not to steady-state operation.
TIMELINE
Aug 20 14:25 kernel 7.0.0-30 installed. Ran 8 days, zero panics.
Aug 28 05:45 unattended-upgrades starts a ~120 package transaction.
Aug 28 05:47 term.log ends at "Setting up snapd (2.76.3+ubuntu26.04)".
Machine dies here. history.log has no End-Date for this
transaction. dpkg left 51 packages in iU and snapd in iF.
Aug 28 11:03 Manual recovery: `dpkg --configure -a` reconfigures snapd.
-> list_del corruption WARNING, kernel tainted G W.
Aug 28 11:14 Reboot requested. snapd receives SIGTERM.
-> fatal Oops, kdump captured a 252 MB vmcore.
Aug 28 11:20+ Three subsequent clean shutdowns, plus a multi-hour
memtest86+ run. No further crashes. /proc/sys/kernel/tainted
back to 0.
RAM was ruled out: memtest86+ passed clean, and the same kernel had already
run 8 days without incident before snapd 2.76.3 arrived. The failure is
deterministic and tied to the package transition, not random.
FIRST EVENT (warning, kernel still alive)
list_del corruption. next->prev should be ffff8f518099a328,
but was 0000000000000000. (next=ffff8f4fa4969b50)
WARNING: lib/list_debug.c:65 at __list_del_entry_valid_or_report+0xc0/0x10b,
CPU#4: snapd/284305
Not tainted 7.0.0-30-generic #30-Ubuntu PREEMPT(lazy)
Call Trace:
remove_wait_queue.cold+0x9/0x12
ep_remove_safe+0x3c/0xe0
do_epoll_ctl+0x599/0x840
__x64_sys_epoll_ctl+0x6c/0xb0
x64_sys_call+0x1be6/0x2390
do_syscall_64+0x105/0x5a0
A second, symmetric warning followed immediately (lib/list_debug.c:62,
prev->next should be ffff8f4f2004b068, but was 0000000000000000), i.e. both
directions of the list entry had already been zeroed.
SECOND EVENT (fatal, ~11 minutes later, on snapd termination)
BUG: unable to handle page fault for address: ffffffff3226cb80
Oops: Oops: 0002 [#1] SMP NOPTI
CPU: 3 UID: 0 PID: 284675 Comm: snapd Kdump: loaded
Tainted: G W 7.0.0-30-generic #30-Ubuntu PREEMPT(lazy)
Hardware name: LENOVO 82VG/LNVNB161216, BIOS KSCN31WW 05/09/2024
RIP: 0010:native_queued_spin_lock_slowpath+0x2f5/0x370
Call Trace:
__raw_spin_lock_irqsave+0x57/0x80
_raw_spin_lock_irqsave+0xe/0x20
remove_wait_queue+0x1b/0x80
ep_remove_safe+0x3c/0xe0
do_epoll_ctl+0x599/0x840
__x64_sys_epoll_ctl+0x6c/0xb0
x64_sys_call+0x1be6/0x2390
do_syscall_64+0x105/0x5a0
? __memcg_slab_free_hook+0x113/0x180
? kmem_cache_free+0x266/0x3f0
? __fput+0x1a2/0x2d0
? fput_close_sync+0x40/0xc0
? __x64_sys_close+0x3e/0x90
ANALYSIS
The fatal trace shows epoll_ctl(EPOLL_CTL_DEL) racing a concurrent close()
on the same descriptor: the kmem_cache_free / __fput / fput_close_sync /
__x64_sys_close frames sit alongside the ep_remove_safe path. This matches
the known eventpoll use-after-free shape, where ep_remove() drops
file->f_ep under the lock but keeps using the file object, while a
concurrent __fput() frees the struct eventpoll underneath it. The
subsequent list operation then writes into freed memory -- which is exactly
what the two list_debug warnings reported (both list pointers zeroed), and
what the page fault at ffffffff3226cb80 in the spinlock slowpath is the
consequence of.
The zeroed pointers in the warning, and the fact that the machine survived
11 more minutes in a tainted state before dying on the next snapd
termination, are consistent with memory that was freed and then reused.
IMPACT
Total loss of the machine mid-upgrade, with dpkg left in a broken state
(51 packages unconfigured). Recovering required a second boot and a manual
`dpkg --configure -a`, which itself re-triggered the bug.
ATTACHED / AVAILABLE
/var/crash/linux-image-7.0.0-30-generic-202608281115.crash (48 KB apport
report,
contains the full VmCoreDmesg, 1675 lines)
/var/crash/202608281115/dump.202608281115 (252 MB vmcore, preserved,
available on request)
/var/crash/202608281115/dmesg.202608281115 (165 KB)
WORKAROUND
None applied to the kernel itself. 7.0.0-29-generic is still installed and
available as a fallback from the GRUB advanced menu.
unattended-upgrades was disabled locally right after the crash, then
re-enabled the same day, once it was established that snapd cannot in fact
be upgraded unattended on this system: the installed snapd
(2.76.3+ubuntu26.04) comes from resolute-updates, which is not among
Unattended-Upgrade::Allowed-Origins, and resolute-security only carries the
older 2.76+ubuntu26.04.3.
ADDITIONAL EVIDENCE (same day, after recovery)
Steady-state snapd operation does NOT reproduce this bug. On the same
kernel, after recovery, all of the following completed with no warning and
no Oops:
- 8x `snap remove --purge <snap>`
- 23x `snap remove <snap> --revision=<rev>`
- 2x `snap set system <option>` (daemon reconfiguration)
- 1x `snap forget <snapshot set>`
- 1 full clean reboot, i.e. snapd receiving SIGTERM at shutdown -- which
is precisely the operation that produced the fatal Oops on Aug 28 11:14.
/proc/sys/kernel/tainted stayed at 0 throughout, and the current boot logs
contain no list_del, BUG: or Oops entries.
This narrows the trigger considerably: the fault appears to require the
package transition itself -- dpkg reconfiguring and restarting snapd across
the 2.76 -> 2.76.3 version change -- rather than epoll teardown during
ordinary snapd shutdown, which by itself is not sufficient to reproduce it.
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "apport report with full VmCoreDmesg, kernel 7.0.0-30
panic"
https://bugs.launchpad.net/bugs/2165791/+attachment/5996187/+files/linux-image-7.0.0-30-generic-202608281115.crash
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165791
Title:
linux 7.0.0-30-generic: list_del corruption + fatal Oops in
ep_remove_safe/remove_wait_queue, triggered by the snapd 2.76 ->
2.76.3 upgrade
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2165791/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
