[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** Changed in: linux (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: Fix Released Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
This bug was fixed in the package linux - 4.13.0-36.40 --- linux (4.13.0-36.40) artful; urgency=medium * linux: 4.13.0-36.40 -proposed tracker (LP: #1750010) * Rebuild without "CVE-2017-5754 ARM64 KPTI fixes" patch set linux (4.13.0-35.39) artful; urgency=medium * linux: 4.13.0-35.39 -proposed tracker (LP: #1748743) * CVE-2017-5715 (Spectre v2 Intel) - Revert "UBUNTU: SAUCE: turn off IBPB when full retpoline is present" - SAUCE: turn off IBRS when full retpoline is present - [Packaging] retpoline files must be sorted - [Packaging] pull in retpoline files linux (4.13.0-34.37) artful; urgency=medium * linux: 4.13.0-34.37 -proposed tracker (LP: #1748475) * libata: apply MAX_SEC_1024 to all LITEON EP1 series devices (LP: #1743053) - libata: apply MAX_SEC_1024 to all LITEON EP1 series devices * KVM patches for s390x to provide facility bits 81 (ppa15) and 82 (bpb) (LP: #1747090) - KVM: s390: wire up bpb feature * artful 4.13 i386 kernels crash after memory hotplug remove (LP: #1747069) - Revert "mm, memory_hotplug: do not associate hotadded memory to zones until online" * CVE-2017-5715 (Spectre v2 Intel) - x86/feature: Enable the x86 feature to control Speculation - x86/feature: Report presence of IBPB and IBRS control - x86/enter: MACROS to set/clear IBRS and set IBPB - x86/enter: Use IBRS on syscall and interrupts - x86/idle: Disable IBRS entering idle and enable it on wakeup - x86/idle: Disable IBRS when offlining cpu and re-enable on wakeup - x86/mm: Set IBPB upon context switch - x86/mm: Only set IBPB when the new thread cannot ptrace current thread - x86/entry: Stuff RSB for entry to kernel for non-SMEP platform - x86/kvm: add MSR_IA32_SPEC_CTRL and MSR_IA32_PRED_CMD to kvm - x86/kvm: Set IBPB when switching VM - x86/kvm: Toggle IBRS on VM entry and exit - x86/spec_ctrl: Add sysctl knobs to enable/disable SPEC_CTRL feature - x86/spec_ctrl: Add lock to serialize changes to ibrs and ibpb control - x86/cpu/AMD: Add speculative control support for AMD - x86/microcode: Extend post microcode reload to support IBPB feature - KVM: SVM: Do not intercept new speculative control MSRs - x86/svm: Set IBRS value on VM entry and exit - x86/svm: Set IBPB when running a different VCPU - KVM: x86: Add speculative control CPUID support for guests - SAUCE: turn off IBPB when full retpoline is present * Artful 4.13 fixes for tun (LP: #1748846) - tun: call dev_get_valid_name() before register_netdevice() - tun: allow positive return values on dev_get_valid_name() call - tun/tap: sanitize TUNSETSNDBUF input * boot failure on AMD Raven + WestonXT (LP: #1742759) - SAUCE: drm/amdgpu: add atpx quirk handling (v2) linux (4.13.0-33.36) artful; urgency=low * linux: 4.13.0-33.36 -proposed tracker (LP: #1746903) [ Stefan Bader ] * starting VMs causing retpoline4 to reboot (LP: #1747507) // CVE-2017-5715 (Spectre v2 retpoline) - x86/retpoline: Fill RSB on context switch for affected CPUs - x86/retpoline: Add LFENCE to the retpoline/RSB filling RSB macros - x86/retpoline: Optimize inline assembler for vmexit_fill_RSB - x86/retpoline: Remove the esp/rsp thunk - x86/retpoline: Simplify vmexit_fill_RSB() * Missing install-time driver for QLogic QED 25/40/100Gb Ethernet NIC (LP: #1743638) - [d-i] Add qede to nic-modules udeb * hisi_sas: driver robustness fixes (LP: #1739807) - scsi: hisi_sas: fix reset and port ID refresh issues - scsi: hisi_sas: avoid potential v2 hw interrupt issue - scsi: hisi_sas: fix v2 hw underflow residual value - scsi: hisi_sas: add v2 hw DFX feature - scsi: hisi_sas: add irq and tasklet cleanup in v2 hw - scsi: hisi_sas: service interrupt ITCT_CLR interrupt in v2 hw - scsi: hisi_sas: fix internal abort slot timeout bug - scsi: hisi_sas: us start_phy in PHY_FUNC_LINK_RESET - scsi: hisi_sas: fix NULL check in SMP abort task path - scsi: hisi_sas: fix the risk of freeing slot twice - scsi: hisi_sas: kill tasklet when destroying irq in v3 hw - scsi: hisi_sas: complete all tasklets prior to host reset * [Artful/Zesty] ACPI APEI error handling bug fixes (LP: #1732990) - ACPI: APEI: fix the wrong iteration of generic error status block - ACPI / APEI: clear error status before acknowledging the error * [Zesty/Artful] On ARM64 PCIE physical function passthrough guest fails to boot (LP: #1732804) - vfio/pci: Virtualize Maximum Payload Size - vfio/pci: Virtualize Maximum Read Request Size * hisi_sas: Add ATA command support for SMR disks (LP: #1739891) - scsi: hisi_sas: support zone management commands * thunderx2: i2c driver PEC and ACPI clock fixes (LP: #1738073) - ACPI / APD: Add clock frequency for ThunderX2 I2C controller - i2c: xlp9xx: Get clock frequency with clk API - i2c: xlp9xx: Handle I2C_M_RE
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** Tags removed: verification-needed-artful ** Tags added: verification-done-artful -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: Fix Committed Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed- artful'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-artful -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: Fix Committed Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** Changed in: linux (Ubuntu Artful) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: Fix Committed Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** No longer affects: linux (Ubuntu Bionic) ** Changed in: linux (Ubuntu Xenial) Status: New => Fix Released ** Changed in: linux (Ubuntu Artful) Status: New => In Progress -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: In Progress Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** Also affects: linux (Ubuntu Artful) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: Incomplete ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Status in linux source package in Xenial: Fix Released Status in linux source package in Artful: In Progress Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1744300] Re: bt_iter() crash due to NULL pointer
** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** No longer affects: linux-gcp (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1744300 Title: bt_iter() crash due to NULL pointer Status in linux package in Ubuntu: Incomplete Bug description: SRU Justification: [Impact] The following crash was observed in Ubuntu 16.04 running linux-gcp kernel version 4.13 (specifically 4.13.0-1006.9): [ 10.972644] BUG: unable to handle kernel NULL pointer dereference at 0030 [ 10.980708] IP: bt_iter+0x31/0x50 [ 10.984310] PGD 0 [ 10.984310] P4D 0 [ 10.986439] [ 10.990190] Oops: [#1] SMP PTI [ 11.016282] Workqueue: kblockd blk_mq_timeout_work [ 11.021196] task: 8e7c2e70 task.stack: b8d4c67a8000 [ 11.027234] RIP: 0010:bt_iter+0x31/0x50 [ 11.031187] RSP: 0018:b8d4c67abda0 EFLAGS: 00010206 [ 11.037730] RAX: b8d4c67abdd0 RBX: 0180 RCX: [ 11.045172] RDX: 8e7c34c8d280 RSI: RDI: 8e7c32dd8000 [ 11.053321] RBP: b8d4c67abe20 R08: R09: 2100 [ 11.060582] R10: 0130 R11: fffee5bf R12: 8e7c3572c790 [ 11.068094] R13: 8e7c3572c780 R14: 0008 R15: 8e7c35e7c180 [ 11.075522] FS: () GS:8e7c3a4c() knlGS: [ 11.083721] CS: 0010 DS: ES: CR0: 80050033 [ 11.089593] CR2: 0030 CR3: 9e20a003 CR4: 001606e0 [ 11.096871] Call Trace: [ 11.099468] ? blk_mq_queue_tag_busy_iter+0xe2/0x1f0 [ 11.104558] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.109130] ? blk_mq_rq_timed_out+0x70/0x70 [ 11.114933] blk_mq_timeout_work+0xbb/0x170 [ 11.119408] process_one_work+0x156/0x410 [ 11.123641] worker_thread+0x4b/0x460 [ 11.127827] kthread+0x109/0x140 [ 11.131186] ? process_one_work+0x410/0x410 [ 11.135499] ? kthread_create_on_node+0x70/0x70 [ 11.140408] ret_from_fork+0x1f/0x30 [ 11.144110] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 30 01 00 00 84 c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 c0 00 00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff [ 11.167573] RIP: bt_iter+0x31/0x50 RSP: b8d4c67abda0 [ 11.173028] CR2: 0030 [ 11.176515] ---[ end trace 2f8e5b1cf4139fec ]--- [ 11.182589] Kernel panic - not syncing: Fatal exception Basically, we have a NULL pointer dereference while in bt_iter() function - this is caused because after the merge of blk-mq scheduler capability on Linux kernel , tags->rqs[] array has been dinamically assigned and there's a small window of time in which the bit is set but tags->rqs[] array wasn't allocated yet. This was reported to happen in about 5% of test runs (more details on test section). [Fix] The fix is small and simple, and it's upstream already. Basically, it adds a NULL pointer check on bt_iter() and bt_tags_iter() functions. The fix is: 7f5562d5ecc4 ("blk-mq-tag: check for NULL rq when iterating tags"), by Jens Axboe. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7f5562d5ecc4) [Testcase] Since the problem manifests in a small non-deterministic time window, there's no easy test to reproduce this. In our case, it was observed while testing a large number of CPU's and attached disks (>200 disks, >150 cores), trying to exercise all CPUs and disks (the disks with quick dd commands). In this test scenario, as already mentioned, issue occured in about 5% of the runs. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744300/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp