[Kernel-packages] [Bug 2020607] Re: ftracetest from selftests in linux ADT test failure with jammy/linux-intel-iotg (kernel NULL pointer dereference)
After splitting ubuntu_kselftests_ftrace out and run test cases one-by- one, we can see it's failing with the second test case, ftrace:test.d-- 00basic--basic2.tc, on J-intel-iotg-5.15.0-1048.54 with node rizzo. However I was unable to reproduce this manually on rizzo: * Passed with running just the ftrace:test.d--00basic--basic2.tc, with "./ftracetest -vvv test.d/00basic/basic2.tc" * Passed with running basic2.tc multiple times. * Passed with running the 1st test case and the offending basic2.tc test case. * Passed with running the whole test suite. But if you try to run this remotely from out build server: SRU_CYCLE="2024.01.08-1" INSTANCE_TYPE="rizzo" timeout 180m $KT/sut-test --nc --region kernel $DEBUG metal $SUT jammy ubuntu_kselftests_ftrace $HOME It will panic right away when hitting the second test case. It looks like it has something to do with CPU hotplug: [ 5990.967618] mmiotrace: Disabling non-boot CPUs... [ 5991.032796] smpboot: CPU 1 is now offline [ 5991.052877] mmiotrace: CPU1 is down. [ 5991.124833] smpboot: CPU 2 is now offline [ 5991.140709] mmiotrace: CPU2 is down. [ 5991.196717] smpboot: CPU 3 is now offline [ 5991.216486] mmiotrace: CPU3 is down. [ 5991.233400] smpboot: CPU 4 is now offline [ 5991.272507] mmiotrace: CPU4 is down. [ 5991.313356] smpboot: CPU 5 is now offline [ 5991.328204] mmiotrace: CPU5 is down. [ 5991.353591] smpboot: CPU 6 is now offline [ 5991.376155] mmiotrace: CPU6 is down. [ 5991.393484] smpboot: CPU 7 is now offline [ 5991.394580] mmiotrace: CPU7 is down. [ 5991.394586] mmiotrace: enabled. [ 5991.394693] mmiotrace: Re-enabling CPUs... [ 5991.394761] x86: Booting SMP configuration: [ 5991.394763] smpboot: Booting Node 0 Processor 1 APIC 0x2 [ 5991.432595] mmiotrace: enabled CPU1. [ 5991.479537] smpboot: Booting Node 0 Processor 2 APIC 0x4 [ 5991.508524] mmiotrace: enabled CPU2. [ 5991.547586] smpboot: Booting Node 0 Processor 3 APIC 0x6 [ 5991.576690] mmiotrace: enabled CPU3. [ 5991.619582] smpboot: Booting Node 0 Processor 4 APIC 0x1 [ 5991.639516] BUG: kernel NULL pointer dereference, address: [ 5991.646618] #PF: supervisor instruction fetch in kernel mode [ 5991.652336] #PF: error_code(0x0010) - not-present page [ 5991.657530] PGD 0 P4D 0 [ 5991.660096] Oops: 0010 [#1] SMP PTI [ 5991.663626] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.0-1048-intel-iotg #54-Ubuntu [ 5991.671709] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.12.0 09/06/2013 [ 5991.679350] RIP: 0010:0x0 [ 5991.682010] Code: Unable to access opcode bytes at RIP 0xffd6. [ 5991.688955] RSP: 0018:b92a40003e90 EFLAGS: 00010097 [ 5991.694233] RAX: RBX: 000231f0 RCX: 0004 [ 5991.701435] RDX: RSI: 0020 RDI: 9c20c007b990 [ 5991.708639] RBP: b92a40003eb8 R08: 9c20c007b990 R09: 0001 [ 5991.715842] R10: 0020 R11: R12: 9c20c007b990 [ 5991.723050] R13: 0572e77e8500 R14: 0004 R15: [ 5991.730252] FS: () GS:9c21f760() knlGS: [ 5991.738417] CS: 0010 DS: ES: CR0: 80050033 [ 5991.744218] CR2: ffd6 CR3: 10c1 CR4: 06f0 [ 5991.751425] Call Trace: [ 5991.753903] [ 5991.755945] ? show_trace_log_lvl+0x1d6/0x2ea [ 5991.760362] ? show_trace_log_lvl+0x1d6/0x2ea [ 5991.764773] ? tick_do_broadcast+0xa1/0xd0 [ 5991.768922] ? show_regs.part.0+0x23/0x29 [ 5991.773026] ? __die_body.cold+0x8/0xd [ 5991.776821] ? __die+0x2b/0x37 [ 5991.779918] ? page_fault_oops+0x13b/0x170 [ 5991.784063] ? do_user_addr_fault+0x321/0x670 [ 5991.788476] ? obj_cgroup_uncharge_pages+0x68/0xf0 [ 5991.793324] ? exc_page_fault+0x77/0x170 [ 5991.797293] ? asm_exc_page_fault+0x27/0x30 [ 5991.801529] tick_do_broadcast+0xa1/0xd0 [ 5991.805501] tick_handle_oneshot_broadcast+0x14d/0x200 [ 5991.810694] timer_interrupt+0x18/0x30 [ 5991.814495] __handle_irq_event_percpu+0x42/0x170 [ 5991.819255] handle_irq_event+0x59/0xb0 [ 5991.823136] handle_edge_irq+0x8c/0x230 [ 5991.827019] __common_interrupt+0x52/0xe0 [ 5991.831078] common_interrupt+0x89/0xa0 [ 5991.834966] [ 5991.837098] [ 5991.839247] asm_common_interrupt+0x27/0x40 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-intel-iotg in Ubuntu. https://bugs.launchpad.net/bugs/2020607 Title: ftracetest from selftests in linux ADT test failure with jammy/linux- intel-iotg (kernel NULL pointer dereference) Status in ubuntu-kernel-tests: New Status in linux-intel-iotg package in Ubuntu: Invalid Status in linux-intel-iotg source package in Jammy: New Bug description: the failure only is seen on the machine rizzo. how to reproduce: 1. run net selftest in the kernel. 2. run ftracetest in the kernel, and then there is a highly chance that causes the kernel oops. issue could be seen on
[Kernel-packages] [Bug 2020607] Re: ftracetest from selftests in linux ADT test failure with jammy/linux-intel-iotg (kernel NULL pointer dereference)
Found on J-realtime 5.15.0-1040.45 with node rizzo ** Attachment added: "j-realtime-ftrace-rizzo.log" https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2020607/+attachment/5677329/+files/j-realtime-ftrace-rizzo.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-intel-iotg in Ubuntu. https://bugs.launchpad.net/bugs/2020607 Title: ftracetest from selftests in linux ADT test failure with jammy/linux- intel-iotg (kernel NULL pointer dereference) Status in ubuntu-kernel-tests: New Status in linux-intel-iotg package in Ubuntu: Invalid Status in linux-intel-iotg source package in Jammy: New Bug description: the failure only is seen on the machine rizzo. how to reproduce: 1. run net selftest in the kernel. 2. run ftracetest in the kernel, and then there is a highly chance that causes the kernel oops. issue could be seen on kernel 5.15.112-0515112 (mainline), 5.15.0-1030-intel-iotg, and 5.15.0-74-generic on the same machine. issue could not be reproduced on kernel 5.19.0-42-generic, 5.17.15-051715 (mainline), and 5.16.20-051620 (mainline). [13279.176639] BUG: kernel NULL pointer dereference, address: [13279.183712] #PF: supervisor instruction fetch in kernel mode [13279.189446] #PF: error_code(0x0010) - not-present page [13279.194654] PGD 0 P4D 0 [13279.197230] Oops: 0010 [#1] SMP PTI [13279.200767] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.0-74-generic #81-Ubuntu [13279.208431] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.12.0 09/06/2013 [13279.216100] RIP: 0010:0x0 [13279.218767] Code: Unable to access opcode bytes at RIP 0xffd6. [13279.225721] RSP: 0018:9a1f80003e90 EFLAGS: 00010097 [13279.231013] RAX: RBX: 000231f0 RCX: 0004 [13279.238229] RDX: RSI: 0020 RDI: 8bfdc0074280 [13279.245449] RBP: 9a1f80003eb8 R08: 8bfdc0074280 R09: 0001 [13279.252673] R10: 0020 R11: R12: 8bfdc0074280 [13279.259900] R13: 0c13a1cab100 R14: 0004 R15: [13279.267124] FS: () GS:8bfef760() knlGS: [13279.275317] CS: 0010 DS: ES: CR0: 80050033 [13279.281140] CR2: ffd6 CR3: 4a01 CR4: 06f0 [13279.288366] Call Trace: [13279.290851] [13279.292902] tick_do_broadcast+0xa1/0xd0 [13279.296894] tick_handle_oneshot_broadcast+0x14d/0x200 [13279.302107] timer_interrupt+0x18/0x30 [13279.305914] __handle_irq_event_percpu+0x42/0x170 [13279.310689] ? timekeeping_advance+0x32a/0x470 [13279.315194] handle_irq_event+0x59/0xb0 [13279.319086] handle_edge_irq+0x8c/0x230 [13279.322976] __common_interrupt+0x52/0xe0 [13279.327045] common_interrupt+0x89/0xa0 [13279.330941] [13279.333079] [13279.335211] asm_common_interrupt+0x27/0x40 [13279.339461] RIP: 0010:cpuidle_enter_state+0xd9/0x620 [13279.344501] Code: 3d e4 e1 d8 4a e8 77 cb 67 ff 49 89 c7 0f 1f 44 00 00 31 ff e8 b8 d8 67 ff 80 7d d0 00 0f 85 61 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6d 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e7 03 00 00 [13279.363476] RSP: 0018:b6603db8 EFLAGS: 0246 [13279.368768] RAX: RBX: 8bfef763b900 RCX: [13279.375984] RDX: 8bfdc01863c0 RSI: 0002 RDI: [13279.383203] RBP: b6603e08 R08: 0c13cc9b2925 R09: 000c3500 [13279.390423] R10: 0005 R11: 071c71c71c71c71c R12: b68d4b20 [13279.397646] R13: 0004 R14: 0004 R15: 0c13cc9b2925 [13279.404883] ? cpuidle_enter_state+0x24a/0x620 [13279.409389] cpuidle_enter+0x2e/0x50 [13279.413019] cpuidle_idle_call+0x142/0x1e0 [13279.417176] do_idle+0x83/0xf0 [13279.422843] cpu_startup_entry+0x20/0x30 [13279.429360] rest_init+0xd3/0x100 [13279.435396] ? acpi_enable_subsystem+0x21d/0x229 [13279.442594] arch_call_rest_init+0xe/0x23 [13279.449114] start_kernel+0x4a9/0x4ca [13279.455389] x86_64_start_reservations+0x24/0x2a [13279.462619] x86_64_start_kernel+0xfb/0x106 [13279.469328] secondary_startup_64_no_verify+0xc2/0xcb [13279.476844] [13279.481553] Modules linked in: br_netfilter tls act_mirred cls_matchall ip6_gre gre ip6_tunnel tunnel6 sch_ingress dummy ip6t_rpfilter mpls_gso mpls_iptunnel mpls_router ip_tunnel esp6 esp4 xfrm_user xfrm_algo l2tp_ip6 l2tp_eth l2tp_ip l2tp_netlink l2tp_core 8021q garp mrp ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp sch_etf sch_fq dccp_ipv6 dccp_ipv4 dccp vxlan ip6_udp_tunnel udp_tunnel bridge stp llc vrf nft_counter nft_chain_nat xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink algif_hash af_alg veth ipmi_ssif intel_powerclamp coretemp kvm_intel kvm
[Kernel-packages] [Bug 2020607] Re: ftracetest from selftests in linux ADT test failure with jammy/linux-intel-iotg (kernel NULL pointer dereference)
This issue can be found on J-ibm 5.15.0-1032.35 as well. ** Attachment added: "j-ibm-console.log" https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2020607/+attachment/5676265/+files/j-ibm-console.log ** Tags added: sru-20230515 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-intel-iotg in Ubuntu. https://bugs.launchpad.net/bugs/2020607 Title: ftracetest from selftests in linux ADT test failure with jammy/linux- intel-iotg (kernel NULL pointer dereference) Status in ubuntu-kernel-tests: New Status in linux-intel-iotg package in Ubuntu: Invalid Status in linux-intel-iotg source package in Jammy: New Bug description: the failure only is seen on the machine rizzo. how to reproduce: 1. run net selftest in the kernel. 2. run ftracetest in the kernel, and then there is a highly chance that causes the kernel oops. issue could be seen on kernel 5.15.112-0515112 (mainline), 5.15.0-1030-intel-iotg, and 5.15.0-74-generic on the same machine. issue could not be reproduced on kernel 5.19.0-42-generic, 5.17.15-051715 (mainline), and 5.16.20-051620 (mainline). [13279.176639] BUG: kernel NULL pointer dereference, address: [13279.183712] #PF: supervisor instruction fetch in kernel mode [13279.189446] #PF: error_code(0x0010) - not-present page [13279.194654] PGD 0 P4D 0 [13279.197230] Oops: 0010 [#1] SMP PTI [13279.200767] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.0-74-generic #81-Ubuntu [13279.208431] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.12.0 09/06/2013 [13279.216100] RIP: 0010:0x0 [13279.218767] Code: Unable to access opcode bytes at RIP 0xffd6. [13279.225721] RSP: 0018:9a1f80003e90 EFLAGS: 00010097 [13279.231013] RAX: RBX: 000231f0 RCX: 0004 [13279.238229] RDX: RSI: 0020 RDI: 8bfdc0074280 [13279.245449] RBP: 9a1f80003eb8 R08: 8bfdc0074280 R09: 0001 [13279.252673] R10: 0020 R11: R12: 8bfdc0074280 [13279.259900] R13: 0c13a1cab100 R14: 0004 R15: [13279.267124] FS: () GS:8bfef760() knlGS: [13279.275317] CS: 0010 DS: ES: CR0: 80050033 [13279.281140] CR2: ffd6 CR3: 4a01 CR4: 06f0 [13279.288366] Call Trace: [13279.290851] [13279.292902] tick_do_broadcast+0xa1/0xd0 [13279.296894] tick_handle_oneshot_broadcast+0x14d/0x200 [13279.302107] timer_interrupt+0x18/0x30 [13279.305914] __handle_irq_event_percpu+0x42/0x170 [13279.310689] ? timekeeping_advance+0x32a/0x470 [13279.315194] handle_irq_event+0x59/0xb0 [13279.319086] handle_edge_irq+0x8c/0x230 [13279.322976] __common_interrupt+0x52/0xe0 [13279.327045] common_interrupt+0x89/0xa0 [13279.330941] [13279.333079] [13279.335211] asm_common_interrupt+0x27/0x40 [13279.339461] RIP: 0010:cpuidle_enter_state+0xd9/0x620 [13279.344501] Code: 3d e4 e1 d8 4a e8 77 cb 67 ff 49 89 c7 0f 1f 44 00 00 31 ff e8 b8 d8 67 ff 80 7d d0 00 0f 85 61 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6d 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e7 03 00 00 [13279.363476] RSP: 0018:b6603db8 EFLAGS: 0246 [13279.368768] RAX: RBX: 8bfef763b900 RCX: [13279.375984] RDX: 8bfdc01863c0 RSI: 0002 RDI: [13279.383203] RBP: b6603e08 R08: 0c13cc9b2925 R09: 000c3500 [13279.390423] R10: 0005 R11: 071c71c71c71c71c R12: b68d4b20 [13279.397646] R13: 0004 R14: 0004 R15: 0c13cc9b2925 [13279.404883] ? cpuidle_enter_state+0x24a/0x620 [13279.409389] cpuidle_enter+0x2e/0x50 [13279.413019] cpuidle_idle_call+0x142/0x1e0 [13279.417176] do_idle+0x83/0xf0 [13279.422843] cpu_startup_entry+0x20/0x30 [13279.429360] rest_init+0xd3/0x100 [13279.435396] ? acpi_enable_subsystem+0x21d/0x229 [13279.442594] arch_call_rest_init+0xe/0x23 [13279.449114] start_kernel+0x4a9/0x4ca [13279.455389] x86_64_start_reservations+0x24/0x2a [13279.462619] x86_64_start_kernel+0xfb/0x106 [13279.469328] secondary_startup_64_no_verify+0xc2/0xcb [13279.476844] [13279.481553] Modules linked in: br_netfilter tls act_mirred cls_matchall ip6_gre gre ip6_tunnel tunnel6 sch_ingress dummy ip6t_rpfilter mpls_gso mpls_iptunnel mpls_router ip_tunnel esp6 esp4 xfrm_user xfrm_algo l2tp_ip6 l2tp_eth l2tp_ip l2tp_netlink l2tp_core 8021q garp mrp ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp sch_etf sch_fq dccp_ipv6 dccp_ipv4 dccp vxlan ip6_udp_tunnel udp_tunnel bridge stp llc vrf nft_counter nft_chain_nat xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink algif_hash af_alg veth ipmi_ssif intel_powerclamp coretemp
[Kernel-packages] [Bug 2020607] Re: ftracetest from selftests in linux ADT test failure with jammy/linux-intel-iotg (kernel NULL pointer dereference)
** Tags added: 5.15 jammy sru-20230417 ubuntu-kernel-selftests -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-intel-iotg in Ubuntu. https://bugs.launchpad.net/bugs/2020607 Title: ftracetest from selftests in linux ADT test failure with jammy/linux- intel-iotg (kernel NULL pointer dereference) Status in ubuntu-kernel-tests: New Status in linux-intel-iotg package in Ubuntu: Invalid Status in linux-intel-iotg source package in Jammy: New Bug description: the failure only is seen on the machine rizzo. how to reproduce: 1. run net selftest in the kernel. 2. run ftracetest in the kernel, and then there is a highly chance that causes the kernel oops. issue could be seen on kernel 5.15.112-0515112 (mainline), 5.15.0-1030-intel-iotg, and 5.15.0-74-generic on the same machine. issue could not be reproduced on kernel 5.19.0-42-generic, 5.17.15-051715 (mainline), and 5.16.20-051620 (mainline). [13279.176639] BUG: kernel NULL pointer dereference, address: [13279.183712] #PF: supervisor instruction fetch in kernel mode [13279.189446] #PF: error_code(0x0010) - not-present page [13279.194654] PGD 0 P4D 0 [13279.197230] Oops: 0010 [#1] SMP PTI [13279.200767] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.15.0-74-generic #81-Ubuntu [13279.208431] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.12.0 09/06/2013 [13279.216100] RIP: 0010:0x0 [13279.218767] Code: Unable to access opcode bytes at RIP 0xffd6. [13279.225721] RSP: 0018:9a1f80003e90 EFLAGS: 00010097 [13279.231013] RAX: RBX: 000231f0 RCX: 0004 [13279.238229] RDX: RSI: 0020 RDI: 8bfdc0074280 [13279.245449] RBP: 9a1f80003eb8 R08: 8bfdc0074280 R09: 0001 [13279.252673] R10: 0020 R11: R12: 8bfdc0074280 [13279.259900] R13: 0c13a1cab100 R14: 0004 R15: [13279.267124] FS: () GS:8bfef760() knlGS: [13279.275317] CS: 0010 DS: ES: CR0: 80050033 [13279.281140] CR2: ffd6 CR3: 4a01 CR4: 06f0 [13279.288366] Call Trace: [13279.290851] [13279.292902] tick_do_broadcast+0xa1/0xd0 [13279.296894] tick_handle_oneshot_broadcast+0x14d/0x200 [13279.302107] timer_interrupt+0x18/0x30 [13279.305914] __handle_irq_event_percpu+0x42/0x170 [13279.310689] ? timekeeping_advance+0x32a/0x470 [13279.315194] handle_irq_event+0x59/0xb0 [13279.319086] handle_edge_irq+0x8c/0x230 [13279.322976] __common_interrupt+0x52/0xe0 [13279.327045] common_interrupt+0x89/0xa0 [13279.330941] [13279.333079] [13279.335211] asm_common_interrupt+0x27/0x40 [13279.339461] RIP: 0010:cpuidle_enter_state+0xd9/0x620 [13279.344501] Code: 3d e4 e1 d8 4a e8 77 cb 67 ff 49 89 c7 0f 1f 44 00 00 31 ff e8 b8 d8 67 ff 80 7d d0 00 0f 85 61 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 f6 0f 88 6d 01 00 00 4d 63 ee 49 83 fd 09 0f 87 e7 03 00 00 [13279.363476] RSP: 0018:b6603db8 EFLAGS: 0246 [13279.368768] RAX: RBX: 8bfef763b900 RCX: [13279.375984] RDX: 8bfdc01863c0 RSI: 0002 RDI: [13279.383203] RBP: b6603e08 R08: 0c13cc9b2925 R09: 000c3500 [13279.390423] R10: 0005 R11: 071c71c71c71c71c R12: b68d4b20 [13279.397646] R13: 0004 R14: 0004 R15: 0c13cc9b2925 [13279.404883] ? cpuidle_enter_state+0x24a/0x620 [13279.409389] cpuidle_enter+0x2e/0x50 [13279.413019] cpuidle_idle_call+0x142/0x1e0 [13279.417176] do_idle+0x83/0xf0 [13279.422843] cpu_startup_entry+0x20/0x30 [13279.429360] rest_init+0xd3/0x100 [13279.435396] ? acpi_enable_subsystem+0x21d/0x229 [13279.442594] arch_call_rest_init+0xe/0x23 [13279.449114] start_kernel+0x4a9/0x4ca [13279.455389] x86_64_start_reservations+0x24/0x2a [13279.462619] x86_64_start_kernel+0xfb/0x106 [13279.469328] secondary_startup_64_no_verify+0xc2/0xcb [13279.476844] [13279.481553] Modules linked in: br_netfilter tls act_mirred cls_matchall ip6_gre gre ip6_tunnel tunnel6 sch_ingress dummy ip6t_rpfilter mpls_gso mpls_iptunnel mpls_router ip_tunnel esp6 esp4 xfrm_user xfrm_algo l2tp_ip6 l2tp_eth l2tp_ip l2tp_netlink l2tp_core 8021q garp mrp ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp sch_etf sch_fq dccp_ipv6 dccp_ipv4 dccp vxlan ip6_udp_tunnel udp_tunnel bridge stp llc vrf nft_counter nft_chain_nat xt_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink algif_hash af_alg veth ipmi_ssif intel_powerclamp coretemp kvm_intel kvm ipmi_si ipmi_devintf binfmt_misc intel_cstate ipmi_msghandler dcdbas acpi_power_meter i7core_edac mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc
