Public bug reported: [Impact]
On Ubuntu 26.04, zpool online with the pool GUID as the device name dereferences NULL in vdev_reopen(). The command dies with "Killed". The dead thread still holds SCL_STATE_ALL as writer, so txg_sync and later zpool commands block in D state until a hard reset. The pool imported normally after the reset, and a later scrub was clean. Admins can reach this by copying what zpool status prints. After zpool import -d /dev/disk/by-uuid/<uuid>, zpool status shows the disk under the pool GUID, because blkid's UUID for a zfs_member is the pool GUID. Passing that name to zpool online -e triggers the crash. The other names we tried for the disk (sdb, /dev/sdb, /dev/sdb1, the full by-uuid path, the leaf GUID) fail with ordinary errors. ZED reached the same oops once, as Comm: agents, with autoexpand=on and no zpool online running, right after a re-import through by-uuid. The trigger has not been isolated. Upstream report: https://github.com/openzfs/zfs/issues/19659 Regression from upstream commit 5caeef02fa53 (RAID-Z expansion), first in 2.3.0, which removed the non-leaf check from vdev_online(). [Affected versions] resolute, 2.4.1-1ubuntu5.1: reproduced on kernel 7.0.0-38.38. The zfs.ko in linux-main-modules-zfs-7.0.0-42-generic has the same version and srcversion (0BF563C747C51B92C058D78), so the current kernel carries the same code. stonking and tenacious, 2.4.4-1ubuntu4: upstream 2.4.4 has the same vdev_online() code (read, not run). noble, 2.2.2-0ubuntu9.5: not affected. Tested; it returns "operation not supported on this type of pool" with no oops. The loaded module ships in linux-main-modules-zfs-* (source linux-main- signed). Its srcversion matches zfs-dkms 2.4.1-1ubuntu5.1 built against the same kernel, so the fix probably needs a kernel build as well as zfs-linux. [Test Plan] zpool create -f -o ashift=12 ncdata /dev/sdX partprobe -s U=$(blkid -o value -s UUID /dev/sdX1) zpool get -H -o value guid ncdata # same value as $U zpool export ncdata zpool import -d /dev/disk/by-uuid/"$U" ncdata zpool status ncdata # the disk is shown as $U zpool online -e ncdata "$U" Before the fix: "Killed", the oops below, and zpool list hangs. After the fix: "cannot expand <guid>: operation not supported on this type of pool", exit 1, and the pool stays responsive. In our run the disk had been grown to 3 TB with autoexpand=off, and the pool had non-default properties. Neither is used on this code path. RAID-Z check, because the removed check was dropped for this feature: zpool create rz raidz1 <f1> <f2> <f3> zpool attach rz raidz1-0 <f4> zpool wait -t raidz_expand rz zpool online -e rz raidz1-0 # must return 0 [Proposed fix] Reject only the root vdev. The old non-leaf check can't simply go back, because spa_raidz_expand_thread() calls vdev_online() with a top-level RAID-Z vdev's GUID. Upstream has not reviewed this yet. --- a/module/zfs/vdev.c +++ b/module/zfs/vdev.c @@ -4447,6 +4447,14 @@ if ((vd = spa_lookup_by_guid(spa, guid, B_TRUE)) == NULL) return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENODEV))); + /* + * The root vdev has no vdev_top to reopen. This is reachable from + * userland, since a numeric device name is looked up as a GUID and the + * root vdev's GUID is the pool GUID. + */ + if (vd == rvd) + return (spa_vdev_state_exit(spa, NULL, SET_ERROR(ENOTSUP))); + wasoffline = (vd->vdev_offline || vd->vdev_tmpoffline); oldstate = vd->vdev_state; Tested with zfs-dkms 2.4.1-1ubuntu5.1 plus this patch on kernel 7.0.0-38. The test plan's crash becomes the error above, and the RAID-Z check passes. [Where problems could occur] The patch only changes the root vdev case, which currently oopses. Any mistake would show up in zpool online or at the end of a RAID-Z expansion. [Other Info] First reported from VMware guests after growing the data disk. Reproduced in a QEMU guest (TCG, PVSCSI, VMware SMBIOS strings, no disk serial or VPD IDs), not on ESXi. BUG: unable to handle page fault for address: 00000000000033e0 #PF: supervisor read access in kernel mode Oops: Oops: 0000 [#1] SMP NOPTI CPU: 0 UID: 0 PID: 2352 Comm: zpool Tainted: P O 7.0.0-38-generic #38-Ubuntu PREEMPT(lazy) RIP: 0010:vdev_reopen+0x15/0x1a0 [zfs] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 CR2: 00000000000033e0 Call Trace: vdev_online+0xc8/0x540 [zfs] zfs_ioc_vdev_set_state+0x86/0x1b0 [zfs] zfsdev_ioctl_common+0x8e3/0x9d0 [zfs] zfsdev_ioctl+0x54/0xe0 [zfs] __x64_sys_ioctl+0xa3/0x100 ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Affects: zfs-linux (Ubuntu) Importance: Undecided Status: New ** Tags: regression-release resolute ** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags added: regression-release ** Tags added: resolute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2172684 Title: zpool online with the pool GUID oopses in vdev_reopen and hangs ZFS until reboot To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2172684/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
