This bug is awaiting verification that the linux- gcp-5.4/5.4.0-1140.149~18.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic-linux- gcp-5.4' to 'verification-done-bionic-linux-gcp-5.4'. If the problem still exists, change the tag 'verification-needed-bionic-linux-gcp-5.4' to 'verification-failed-bionic-linux-gcp-5.4'.
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: kernel-spammed-bionic-linux-gcp-5.4-v2 verification-needed-bionic-linux-gcp-5.4 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2081085 Title: wbt:wbt_* trace event NULL pointer dereference with GENHD_FL_HIDDEN disks Status in linux package in Ubuntu: Invalid Status in linux source package in Focal: Fix Released Bug description: [Impact] Systems with storage devices that utilize the GENHD_FL_HIDDEN flag, such as NVMe disks declaring support for multiple controllers (aka native multipathing), will have a request queue with backing_dev_info->dev set to NULL. When tracing is enabled with any of the wbt:wbt_* events enabled, a NULL pointer dereference will occur in the corresponding trace function called from wb_timer_fn. This occurs when the trace function attempts to access the device's name with dev_name. On a DGXA100 system, this can be reproduced by running the following, where /dev/nvme0n1 is one of the 4 NVMe disks in the system that support native multipathing: $ echo 1 | sudo tee /sys/kernel/tracing/events/wbt/enable $ echo 1 | sudo tee /sys/kernel/tracing/tracing_on $ sudo dd if=/dev/zero of=/dev/nvme0n1 A NULL pointer dereference will occur and the system will become unresponsive. [Fix] The upstream commit d51cfc53ade318 ("bdi: use bdi_dev_name() to get device name") resolves this by changing the wbt:wbt_* trace functions to use the bdi_dev_name function instead of dev_name. The bdi_dev_name function safely handles the case where the supplied device is NULL. [Test Case] Verified that the commit d51cfc53ade318 ("bdi: use bdi_dev_name() to get device name") resolves the issue on DGXA100 when applied to the "Ubuntu-5.4.0-196.216" tag. The reproducer no longer causes a NULL pointer dereference or otherwise crash the system. [Regression Potential] There is a low risk of a regression: * In the focal K5.4 kernel, the bdi_dev_name function is used in other trace event functions for the same purpose of catching the case where bdi->dev is NULL. * This change is already present in kernel versions 5.7 and newer. [Other] The patch d51cfc53ade318 ("bdi: use bdi_dev_name() to get device name") is already present in Jammy K5.15 and newer. ----- Originally found via kernel regression testing and reported here by @cypressyew: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2072972. [ 1073.837085] BUG: kernel NULL pointer dereference, address: 0000000000000050 [ 1073.844858] #PF: supervisor read access in kernel mode [ 1073.850589] #PF: error_code(0x0000) - not-present page [ 1073.856318] PGD 0 P4D 0 [ 1073.859141] Oops: 0000 [#1] SMP NOPTI [ 1073.863226] CPU: 9 PID: 0 Comm: swapper/9 Tainted: P OE 5.4.0-196-generic #216-Ubuntu [ 1073.873319] Hardware name: NVIDIA DGXA100 920-23687-2530-000/DGXA100, BIOS 1.25 08/31/2023 [ 1073.882547] RIP: 0010:trace_event_raw_event_wbt_timer+0x6f/0x100 [ 1073.889248] Code: 59 80 e5 02 0f 85 8f 00 00 00 4c 89 e6 ba 34 00 00 00 48 8d 7d a0 e8 00 aa c9 ff 49 89 c4 48 85 c0 74 37 49 8b 87 b8 03 00 00 <48> 8b 70 50 48 85 f6 74 45 49 8d 7c 24 08 ba 20 00 00 00 e8 c9 18 [ 1073.910200] RSP: 0018:ffffaf08598a8da0 EFLAGS: 00010282 [ 1073.916029] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000080000100 [ 1073.923988] RDX: ffff970f0ba9501c RSI: 0000000000000100 RDI: ffff970f0ba95018 [ 1073.931947] RBP: ffffaf08598a8e08 R08: ffff970f0ba95010 R09: 0000000000000100 [ 1073.939906] R10: ffffcf083fdc9a58 R11: 0000000000000386 R12: ffff970f0ba9501c [ 1073.947864] R13: 0000000000000000 R14: 0000000000000001 R15: ffff976efdfba000 [ 1073.955824] FS: 0000000000000000(0000) GS:ffff970f0f640000(0000) knlGS:0000000000000000 [ 1073.964850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1073.971259] CR2: 0000000000000050 CR3: 0000001f7de80000 CR4: 0000000000340ee0 [ 1073.979217] Call Trace: [ 1073.981942] <IRQ> [ 1073.984188] ? show_regs.cold+0x1a/0x1f [ 1073.988456] ? __die+0x90/0xd9 [ 1073.991862] ? no_context.isra.0+0x12c/0x320 [ 1073.996626] ? update_group_capacity+0x2c/0x1d0 [ 1074.001679] ? __bad_area_nosemaphore+0x45/0x1a0 [ 1074.006829] ? bad_area_nosemaphore+0x16/0x20 [ 1074.011687] ? do_user_addr_fault+0x267/0x440 [ 1074.016547] ? __enqueue_entity+0x96/0xa0 [ 1074.021018] ? enqueue_entity+0x139/0x670 [ 1074.025490] ? __do_page_fault+0x58/0x90 [ 1074.029861] ? do_page_fault+0x2c/0xe0 [ 1074.034044] ? page_fault+0x34/0x40 [ 1074.037933] ? trace_event_raw_event_wbt_timer+0x6f/0x100 [ 1074.043954] ? enqueue_entity+0x139/0x670 [ 1074.048426] wb_timer_fn+0x1d6/0x3c0 [ 1074.052413] ? blk_mq_tag_update_depth+0x100/0x100 [ 1074.057755] blk_stat_timer_fn+0x13a/0x140 [ 1074.062326] call_timer_fn+0x32/0x130 [ 1074.066409] __run_timers.part.0+0x180/0x280 [ 1074.071174] ? tick_sched_handle+0x33/0x60 [ 1074.075740] ? tick_sched_timer+0x3d/0x80 [ 1074.080211] ? recalibrate_cpu_khz+0x10/0x10 [ 1074.084971] ? ktime_get+0x3e/0xa0 [ 1074.088765] ? native_apic_msr_write+0x2b/0x30 [ 1074.093719] run_timer_softirq+0x2a/0x50 [ 1074.098091] __do_softirq+0xd1/0x2c1 [ 1074.102078] irq_exit+0xae/0xb0 [ 1074.105580] smp_apic_timer_interrupt+0x7b/0x140 [ 1074.110728] apic_timer_interrupt+0xf/0x20 [ 1074.115295] </IRQ> [ 1074.117634] RIP: 0010:native_safe_halt+0xe/0x10 [ 1074.122685] Code: 7b ff ff ff eb bd 90 90 90 90 90 90 e9 07 00 00 00 0f 00 2d e6 0c 50 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d d6 0c 50 00 fb f4 <c3> 90 0f 1f 44 00 00 55 48 89 e5 41 55 41 54 53 e8 dd 3e 61 ff 65 [ 1074.143638] RSP: 0018:ffffaf0840387e70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 1074.152083] RAX: ffffffff90d0ab70 RBX: 0000000000000009 RCX: 0000000000000001 [ 1074.160042] RDX: 0000000000065c72 RSI: 0000000000000083 RDI: 0000000000000009 [ 1074.168002] RBP: ffffaf0840387e90 R08: 0000000000000001 R09: 0000000000022f80 [ 1074.175962] R10: ffff980f3fed5328 R11: 0000000000000000 R12: 0000000000000009 [ 1074.183920] R13: ffff970f0bff5e00 R14: 0000000000000000 R15: 0000000000000000 [ 1074.191882] ? __cpuidle_text_start+0x8/0x8 [ 1074.196546] ? default_idle+0x20/0x140 [ 1074.200726] arch_cpu_idle+0x15/0x20 [ 1074.204711] default_idle_call+0x23/0x30 [ 1074.209085] do_idle+0x1fb/0x270 [ 1074.212683] ? complete+0x49/0x50 [ 1074.216376] cpu_startup_entry+0x20/0x30 [ 1074.220750] start_secondary+0x178/0x1d0 [ 1074.225126] secondary_startup_64+0xa4/0xb0 [ 1074.229789] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua amd64_edac_mod edac_mce_amd kvm_amd kvm nvidia_uvm(O) nvidia_drm(PO) nvidia_modeset(PO) ipmi_ssif nvidia(PO) input_leds mlx5_ib(OE) ib_uverbs(OE) binfmt_misc ib_core(OE) ccp k10temp ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel msr ramoops reed_solomon efi_pstore ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel mlx5_core(OE) ast crypto_simd drm_vram_helper cryptd glue_helper ttm drm_kms_helper pci_hyperv_intf syscopyarea mlxdevm(OE) auxiliary(OE) sysfillrect mpt3sas hid_generic igb uas tls sysimgblt usbhid raid_class mlxfw(OE) psample scsi_transport_sas usb_storage dca hid fb_sys_fops i2c_algo_bit nvme mlx_compat(OE) drm nvme_core i2c_piix4 [ 1074.323033] CR2: 0000000000000050 [ 1074.326858] ---[ end trace f3c3dea0291e7e7a ]--- [ 1074.472012] RIP: 0010:trace_event_raw_event_wbt_timer+0x6f/0x100 [ 1074.478714] Code: 59 80 e5 02 0f 85 8f 00 00 00 4c 89 e6 ba 34 00 00 00 48 8d 7d a0 e8 00 aa c9 ff 49 89 c4 48 85 c0 74 37 49 8b 87 b8 03 00 00 <48> 8b 70 50 48 85 f6 74 45 49 8d 7c 24 08 ba 20 00 00 00 e8 c9 18 [ 1074.499668] RSP: 0018:ffffaf08598a8da0 EFLAGS: 00010282 [ 1074.505497] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000080000100 [ 1074.513459] RDX: ffff970f0ba9501c RSI: 0000000000000100 RDI: ffff970f0ba95018 [ 1074.521418] RBP: ffffaf08598a8e08 R08: ffff970f0ba95010 R09: 0000000000000100 [ 1074.529378] R10: ffffcf083fdc9a58 R11: 0000000000000386 R12: ffff970f0ba9501c [ 1074.537336] R13: 0000000000000000 R14: 0000000000000001 R15: ffff976efdfba000 [ 1074.545297] FS: 0000000000000000(0000) GS:ffff970f0f640000(0000) knlGS:0000000000000000 [ 1074.554325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1074.560733] CR2: 0000000000000050 CR3: 0000001f7de80000 CR4: 0000000000340ee0 [ 1074.568695] Kernel panic - not syncing: Fatal exception in interrupt [ 1074.578317] Kernel Offset: 0xf200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 1075.126325] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2081085/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp