Public bug reported: Description: Ubuntu 20.04.6 LTS Release: 20.04
linux-image-generic: Installed: 5.4.0.169.167 Candidate: 5.4.0.169.167 A server has started to crash with logs that suggest the igb module is the reason. Investigating the logs of the server showed the issue started with kernel linux-image-5.4.0-166-generic Because of the timing and the log messages this patch is suspected as the cause https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=Ubuntu-5.4.0-166.183&id=64285e660b7ab4614b44b19d062944be1c1971de The crashes are inconsistent. They appear to be triggered by a hardware error which is unexpected. The logs show the hardware error has been happening for a while but only results in a crash with newer kernels. The hardware is a quad-port NIC Ethernet controller: Intel Corporation I350 Gigabit Network Connection ######## Up through kernel 5.4.0-165-generic the kernel logs include the following error and does *not* result in a crash Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3 Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Device recovery successful Nov 01 07:13:41 ksq-a-mon-p13 kernel: igb 0000:04:00.2 enp4s0f2: igb: enp4s0f2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Nov 01 07:13:41 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837221.513:31): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net/enp4s0f2/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Nov 01 07:13:42 ksq-a-mon-p13 kernel: igb 0000:04:00.3 enp4s0f3: igb: enp4s0f3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Nov 01 07:13:42 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837222.301:32): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net/enp4s0f3/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 ######## Starting with kernel 5.4.0-166-generic the kernel logs include the following and the server crashes Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Uncorrected (Non-Fatal) error received: 0000:04:00.3 Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: ------------[ cut here ]------------ Dec 27 09:26:55 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525! Dec 27 09:26:55 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI Dec 27 09:26:55 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-169-generic #187-Ubuntu Dec 27 09:26:55 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: Call Trace: Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20 Dec 27 09:26:55 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280 Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1 Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60 Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: kthread+0x104/0x140 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40 Dec 27 09:26:55 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm ast crct10dif_pclmul ghash_clmulni_intel aesni_intel drm_vram_helper ttm crypto_simd cryptd glue_helper intel_cstate joydev input_leds drm_kms_helper fb_sys_fops syscopyarea ipmi_si sysfillrect sysimgblt ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich crc32_pclmul igb ahci lpc_ich i2c_i801 libahci i2c_ismt i2c_algo_bit dca Dec 27 09:26:55 ksq-a-mon-p13 kernel: ---[ end trace 26c0760dce96415f ]--- Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 0000000042cc7117 (stack is 0000000046d88908..0000000062c2f40a) ######## The behavior is not always consistent. For example, running kernel 5.4.0-166-generic the following was logged and the server did *not* crash Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3 Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: ------------[ cut here ]------------ Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525! Nov 13 18:30:22 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-166-generic #183-Ubuntu Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace: Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20 Nov 13 18:30:22 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280 Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1 Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60 Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: kthread+0x104/0x140 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fo ps cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2 c_algo_bit i2c_ismt dca Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c4 ]--- Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 00000000844f184b (stack is 00000000f8764a81..00000000b1472243) Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel stack overflow (page fault): 0000 [#2] SMP PTI Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Tainted: G D 5.4.0-166-generic #183-Ubuntu Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:mutex_lock+0x1e/0x40 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 0d e8 ff ff 31 c0 65 48 8b 14 25 c0 bb 01 00 <f0> 49 0f b1 14 24 74 08 4c 89 e7 e8 b2 ff ff ff 41 5c 5d c3 66 66 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7df8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: 0000000000000000 RBX: ffff9ad4402d7f38 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: ffff89d9b5d02c80 RSI: 0000000000000000 RDI: ffff9ad4402d8c80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7e00 R08: ffffffffbc05f080 R09: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b625f380 R11: 0000000000000001 R12: ffff9ad4402d8c80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9b5d037ec R14: ffff89d9b5d037b0 R15: ffff9ad4402d7e80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace: Nov 13 18:30:22 ksq-a-mon-p13 kernel: <#DF> Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? handle_stack_overflow+0x43/0x52 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? no_context+0x23b/0x380 Nov 13 18:30:22 ksq-a-mon-p13 kernel: </#DF> Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fops cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2c_algo_bit i2c_ismt dca Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c5 ]--- Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Fixing recursive fault but reboot is needed! ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-169-generic 5.4.0-169.187 ProcVersionSignature: Ubuntu 5.4.0-169.187-generic 5.4.257 Uname: Linux 5.4.0-169-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jan 13 09:56 seq crw-rw---- 1 root audio 116, 33 Jan 13 09:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CasperMD5CheckResult: skip Date: Thu Jan 18 16:51:14 2024 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: Supermicro SYS-5018A-FTN4 PciMultimedia: ProcEnviron: LANGUAGE=en_US: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 astdrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-169-generic root=/dev/mapper/systemvg-rootlv ro crashkernel=512M-:192M RelatedPackageVersions: linux-restricted-modules-5.4.0-169-generic N/A linux-backports-modules-5.4.0-169-generic N/A linux-firmware 1.187.39 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/18/2018 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.1 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: A1SRi dmi.board.vendor: Supermicro dmi.board.version: 123456789 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 1 dmi.chassis.vendor: Supermicro dmi.chassis.version: To Be Filled By O.E.M. dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.1:bd01/18/2018:svnSupermicro:pnSYS-5018A-FTN4:pvr123456789:rvnSupermicro:rnA1SRi:rvr123456789:cvnSupermicro:ct1:cvrToBeFilledByO.E.M.: dmi.product.family: SMC X10 dmi.product.name: SYS-5018A-FTN4 dmi.product.sku: 081315D9 dmi.product.version: 123456789 dmi.sys.vendor: Supermicro ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2049787 Title: igb module crashing server Status in linux package in Ubuntu: New Bug description: Description: Ubuntu 20.04.6 LTS Release: 20.04 linux-image-generic: Installed: 5.4.0.169.167 Candidate: 5.4.0.169.167 A server has started to crash with logs that suggest the igb module is the reason. Investigating the logs of the server showed the issue started with kernel linux-image-5.4.0-166-generic Because of the timing and the log messages this patch is suspected as the cause https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?h=Ubuntu-5.4.0-166.183&id=64285e660b7ab4614b44b19d062944be1c1971de The crashes are inconsistent. They appear to be triggered by a hardware error which is unexpected. The logs show the hardware error has been happening for a while but only results in a crash with newer kernels. The hardware is a quad-port NIC Ethernet controller: Intel Corporation I350 Gigabit Network Connection ######## Up through kernel 5.4.0-165-generic the kernel logs include the following error and does *not* result in a crash Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3 Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Nov 01 07:13:38 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Nov 01 07:13:38 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Device recovery successful Nov 01 07:13:41 ksq-a-mon-p13 kernel: igb 0000:04:00.2 enp4s0f2: igb: enp4s0f2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Nov 01 07:13:41 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837221.513:31): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.2/net/enp4s0f2/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 Nov 01 07:13:42 ksq-a-mon-p13 kernel: igb 0000:04:00.3 enp4s0f3: igb: enp4s0f3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX Nov 01 07:13:42 ksq-a-mon-p13 kernel: audit: type=1400 audit(1698837222.301:32): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/sys/devices/pci0000:00/0000:00:03.0/0000:04:00.3/net/enp4s0f3/type" pid=514 comm="sssd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0 ######## Starting with kernel 5.4.0-166-generic the kernel logs include the following and the server crashes Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Uncorrected (Non-Fatal) error received: 0000:04:00.3 Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported. Dec 27 09:26:55 ksq-a-mon-p13 kernel: ------------[ cut here ]------------ Dec 27 09:26:55 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525! Dec 27 09:26:55 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI Dec 27 09:26:55 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-169-generic #187-Ubuntu Dec 27 09:26:55 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: Call Trace: Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20 Dec 27 09:26:55 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280 Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1 Dec 27 09:26:55 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60 Dec 27 09:26:55 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: kthread+0x104/0x140 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90 Dec 27 09:26:55 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40 Dec 27 09:26:55 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm ast crct10dif_pclmul ghash_clmulni_intel aesni_intel drm_vram_helper ttm crypto_simd cryptd glue_helper intel_cstate joydev input_leds drm_kms_helper fb_sys_fops syscopyarea ipmi_si sysfillrect sysimgblt ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich crc32_pclmul igb ahci lpc_ich i2c_i801 libahci i2c_ismt i2c_algo_bit dca Dec 27 09:26:55 ksq-a-mon-p13 kernel: ---[ end trace 26c0760dce96415f ]--- Dec 27 09:26:55 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Dec 27 09:26:55 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RSP: 0018:ffffab48c02d7ce8 EFLAGS: 00010246 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RAX: ffff9489f40be000 RBX: ffff9489e9d7c8c0 RCX: 0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff9489e94b0b00 Dec 27 09:26:55 ksq-a-mon-p13 kernel: RBP: ffffab48c02d7cf8 R08: 00000000000003d1 R09: ffffab48c08f5000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R10: 0000000000000168 R11: ffff9489f7b2eba0 R12: ffff9489e9d7c000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: R13: ffff9489e9d7c8c0 R14: ffff9489f5e1d000 R15: ffff9489f5e81000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff9489f7b00000(0000) knlGS:0000000000000000 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Dec 27 09:26:55 ksq-a-mon-p13 kernel: CR2: 0000561bc783e140 CR3: 00000001fb00a000 CR4: 00000000001006e0 Dec 27 09:26:55 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 0000000042cc7117 (stack is 0000000046d88908..0000000062c2f40a) ######## The behavior is not always consistent. For example, running kernel 5.4.0-166-generic the following was logged and the server did *not* crash Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcieport 0000:00:03.0: AER: Multiple Uncorrected (Non-Fatal) error received: 0000:04:00.3 Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: device [8086:1521] error status/mask=00004000/00000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: AER: [14] CmpltTO Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.0: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.1: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.2: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb 0000:04:00.3: Non-correctable non-fatal error reported. Nov 13 18:30:22 ksq-a-mon-p13 kernel: ------------[ cut here ]------------ Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel BUG at include/linux/netdevice.h:525! Nov 13 18:30:22 ksq-a-mon-p13 kernel: invalid opcode: 0000 [#1] SMP PTI Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Not tainted 5.4.0-166-generic #183-Ubuntu Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace: Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_trap+0x85/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_error_trap+0x7c/0xc0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? do_invalid_op+0x3c/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? invalid_op+0x1e/0x30 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? igb_up+0x14/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: igb_io_resume+0x31/0x50 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: report_resume+0x5c/0x80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? report_frozen_detected+0x20/0x20 Nov 13 18:30:22 ksq-a-mon-p13 kernel: pci_walk_bus+0x75/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: pcie_do_recovery+0x163/0x280 Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_process_err_devices+0xa2/0xd1 Nov 13 18:30:22 ksq-a-mon-p13 kernel: aer_isr.cold+0x52/0xa1 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_finalize_oneshot.part.0+0xf0/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread_fn+0x28/0x60 Nov 13 18:30:22 ksq-a-mon-p13 kernel: irq_thread+0xf8/0x190 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_forced_thread_fn+0x90/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: kthread+0x104/0x140 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? irq_thread_check_affinity+0xf0/0xf0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? kthread_park+0x90/0x90 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ret_from_fork+0x35/0x40 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fo ps cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2 c_algo_bit i2c_ismt dca Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c4 ]--- Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: 00007fc0857570a0 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: BUG: stack guard page was hit at 00000000844f184b (stack is 00000000f8764a81..00000000b1472243) Nov 13 18:30:22 ksq-a-mon-p13 kernel: kernel stack overflow (page fault): 0000 [#2] SMP PTI Nov 13 18:30:22 ksq-a-mon-p13 kernel: CPU: 4 PID: 131 Comm: irq/26-aerdrv Tainted: G D 5.4.0-166-generic #183-Ubuntu Nov 13 18:30:22 ksq-a-mon-p13 kernel: Hardware name: Supermicro SYS-5018A-FTN4/A1SRi, BIOS 2.1 01/18/2018 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:mutex_lock+0x1e/0x40 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc e8 0d e8 ff ff 31 c0 65 48 8b 14 25 c0 bb 01 00 <f0> 49 0f b1 14 24 74 08 4c 89 e7 e8 b2 ff ff ff 41 5c 5d c3 66 66 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7df8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: 0000000000000000 RBX: ffff9ad4402d7f38 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: ffff89d9b5d02c80 RSI: 0000000000000000 RDI: ffff9ad4402d8c80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7e00 R08: ffffffffbc05f080 R09: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b625f380 R11: 0000000000000001 R12: ffff9ad4402d8c80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9b5d037ec R14: ffff89d9b5d037b0 R15: ffff9ad4402d7e80 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Call Trace: Nov 13 18:30:22 ksq-a-mon-p13 kernel: <#DF> Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? show_regs.cold+0x1a/0x1f Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? __die+0x90/0xd9 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? die+0x30/0x50 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? handle_stack_overflow+0x43/0x52 Nov 13 18:30:22 ksq-a-mon-p13 kernel: ? no_context+0x23b/0x380 Nov 13 18:30:22 ksq-a-mon-p13 kernel: </#DF> Nov 13 18:30:22 ksq-a-mon-p13 kernel: Modules linked in: nfnetlink bluetooth ecdh_generic ecc ipmi_ssif intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel ast drm_vram_helper ttm input_leds joydev aesni_intel drm_kms_helper crypto_simd fb_sys_fops cryptd glue_helper syscopyarea sysfillrect intel_cstate sysimgblt ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ramoops reed_solomon drm efi_pstore ip_tables x_tables autofs4 hid_generic usbhid hid gpio_ich ahci crc32_pclmul i2c_i801 lpc_ich libahci igb i2c_algo_bit i2c_ismt dca Nov 13 18:30:22 ksq-a-mon-p13 kernel: ---[ end trace 9ec172007066a0c5 ]--- Nov 13 18:30:22 ksq-a-mon-p13 kernel: RIP: 0010:igb_up+0x51/0x160 [igb] Nov 13 18:30:22 ksq-a-mon-p13 kernel: Code: d2 eb 16 f0 80 60 60 fe f0 80 60 60 f7 48 83 c2 01 39 93 14 02 00 00 76 13 48 8b 84 d3 08 0f 00 00 48 8b 48 60 83 e1 01 75 d9 <0f> 0b f6 83 11 02 00 00 20 0f 85 c0 00 00 00 48 8b bb 08 0f 00 00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RSP: 0018:ffff9ad4402d7ce8 EFLAGS: 00010246 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RAX: ffff89d9aa57b800 RBX: ffff89d9a9f9c8c0 RCX: 0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff89d9aa57cf00 Nov 13 18:30:22 ksq-a-mon-p13 kernel: RBP: ffff9ad4402d7cf8 R08: 0000000000000001 R09: ffff9ad44088d000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R10: ffff89d9b7b2ebc0 R11: ffff89d9b7b2eba0 R12: ffff89d9a9f9c000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: R13: ffff89d9a9f9c8c0 R14: ffff89d9b5e19800 R15: ffff89d9b5e80000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: FS: 0000000000000000(0000) GS:ffff89d9b7b00000(0000) knlGS:0000000000000000 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 13 18:30:22 ksq-a-mon-p13 kernel: CR2: ffff9ad4402d8c80 CR3: 00000001e3c0a000 CR4: 00000000001006e0 Nov 13 18:30:22 ksq-a-mon-p13 kernel: Fixing recursive fault but reboot is needed! ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-169-generic 5.4.0-169.187 ProcVersionSignature: Ubuntu 5.4.0-169.187-generic 5.4.257 Uname: Linux 5.4.0-169-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jan 13 09:56 seq crw-rw---- 1 root audio 116, 33 Jan 13 09:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CasperMD5CheckResult: skip Date: Thu Jan 18 16:51:14 2024 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: Supermicro SYS-5018A-FTN4 PciMultimedia: ProcEnviron: LANGUAGE=en_US: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 astdrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-169-generic root=/dev/mapper/systemvg-rootlv ro crashkernel=512M-:192M RelatedPackageVersions: linux-restricted-modules-5.4.0-169-generic N/A linux-backports-modules-5.4.0-169-generic N/A linux-firmware 1.187.39 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/18/2018 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.1 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: A1SRi dmi.board.vendor: Supermicro dmi.board.version: 123456789 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 1 dmi.chassis.vendor: Supermicro dmi.chassis.version: To Be Filled By O.E.M. dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.1:bd01/18/2018:svnSupermicro:pnSYS-5018A-FTN4:pvr123456789:rvnSupermicro:rnA1SRi:rvr123456789:cvnSupermicro:ct1:cvrToBeFilledByO.E.M.: dmi.product.family: SMC X10 dmi.product.name: SYS-5018A-FTN4 dmi.product.sku: 081315D9 dmi.product.version: 123456789 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2049787/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
