On 2024-09-28 14:40:05 +0000, Pucha, HimasekharX Reddy wrote:
> >-----Original Message-----
> >From: Intel-wired-lan <[email protected]> On Behalf Of
> >Mohamed Khalfella
> > Sent: Wednesday, September 25, 2024 2:36 AM
> > To: Nguyen, Anthony L <[email protected]>; Kitszel, Przemyslaw
> > <[email protected]>; David S. Miller <[email protected]>; Eric
> > Dumazet <[email protected]>; Jakub Kicinski <[email protected]>; Paolo
> > Abeni <[email protected]>; Auke Kok <[email protected]>; Zhong,
> > YuanYuan <[email protected]>; Jeff Garzik <[email protected]>;
> > Mohamed Khalfella <[email protected]>; Ying Hsu
> > <[email protected]>; Simon Horman <[email protected]>
> > Cc: [email protected]; [email protected];
> > [email protected]
> > Subject: [Intel-wired-lan] [PATCH v2 1/1] igb: Do not bring the device up
> > after non-fatal error
> >
> >Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed
> >igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid
> >hung task that can happen when igb_down() is called multiple times. This
> >caused an issue when processing transient non-fatal errors.
> > igb_io_resume(), which is called after igb_io_error_detected(), assumes
> > that device is brought down by igb_io_error_detected() if the interface is
> > up. This resulted in panic with stacktrace below.
> >
> > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292]
> > pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received:
> > 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error:
> > severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
> > [ T292] igb 0000:09:00.0: device [8086:1537] error
> > status/mask=00004000/00000000
> > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb
> > 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000
> > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [
> > T292] igb 0000:09:00.0: Non-correctable non-fatal error reported.
> > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [
> > T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292]
> > ------------[ cut here ]------------ [ T292] kernel BUG at
> > net/core/dev.c:6539!
> > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP:
> > 0010:napi_enable+0x37/0x40 [ T292] Call Trace:
> > [ T292] <TASK>
> > [ T292] ? die+0x33/0x90
> > [ T292] ? do_trap+0xdc/0x110
> > [ T292] ? napi_enable+0x37/0x40
> > [ T292] ? do_error_trap+0x70/0xb0
> > [ T292] ? napi_enable+0x37/0x40
> > [ T292] ? napi_enable+0x37/0x40
> > [ T292] ? exc_invalid_op+0x4e/0x70
> > [ T292] ? napi_enable+0x37/0x40
> > [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40
> > [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292]
> > report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [
> > T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [
> > T292] pcie_do_recovery+0x22f/0x380 [ T292]
> > aer_process_err_devices+0x110/0x160
> > [ T292] aer_isr+0x1c1/0x1e0
> > [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60
> > [ T292] irq_thread+0xe3/0x1a0 [ T292] ?
> > irq_set_affinity_notifier+0x120/0x120
> > [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [
> > T292] ? kthread_complete_and_exit+0x20/0x20
> > [ T292] ret_from_fork+0x2d/0x50
> > [ T292] ? kthread_complete_and_exit+0x20/0x20
> > [ T292] ret_from_fork_asm+0x11/0x20
> > [ T292] </TASK>
> >
> > To fix this issue igb_io_resume() checks if the interface is running and
> > the device is not down this means igb_io_error_detected() did not bring the
> > device down and there is no need to bring it up.
> >
> > Signed-off-by: Mohamed Khalfella <[email protected]>
> > Reviewed-by: Yuanyuan Zhong<[email protected]>
> > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal")
> > ---
> > drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++
> > 1 file changed, 4 insertions(+)
>
> Any reproductions steps for reproduction of these issue?
>
I know of two way to reproduce this kernel panic on a kernel that does
not have the patch above.
1- Using aer-inject:
This works on both physical machine and vm. Here are the steps on a
virtual machine.
root@(none):~# lspci -t -v -s 03:00.0
-[0000:02]---00.0-[03]----00.0 Intel Corporation 82576 Gigabit Network
Connection
root@(none):~# cat > /tmp/uncor << EOF
> AER
> UNCOR_STATUS COMP_ABORT
> HEADER_LOG 0 1 2 3
> EOF
root@(none):~# modprobe aer_inject
root@(none):~# /var/tmp/aer-inject --id=0000:03:00.0 /tmp/uncor
This is the qemu command used to start the vm. You probably do not
need all the options related to numa settings and iommu. Only the part
related to pci setup should be enough.
/usr/bin/qemu-system-x86_64 \
-kernel $SRCDIR/arch/x86/boot/bzImage \
-initrd $INITRAMFSIMG \
-append "rdinit=/startup.sh console=ttyS0,115200n8" \
-machine q35,accel=kvm,kernel-irqchip=split \
-nographic \
-chardev
socket,id=gdb0,host=0.0.0.0,port=22004,telnet=on,server=on,wait=off \
-gdb chardev:gdb0 \
$GDB_WAIT \
-serial telnet:127.0.0.1:22003,server=on,wait=off \
-device pxb-pcie,id=pcie.1,bus_nr=2,bus=pcie.0 \
-device ioh3420,id=pcie_port1,bus=pcie.1,chassis=1 \
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:22002-:22 \
-device igb,netdev=net0,id=net0,mac=52:54:00:b8:9c:58,bus=pcie_port1 \
-cpu host \
-smp 4 \
-m 8G \
-object memory-backend-ram,size=7G,id=m0 \
-object memory-backend-ram,size=1G,id=m1 \
-numa node,nodeid=0,memdev=m0,cpus=0-1 \
-numa node,nodeid=1,memdev=m1,cpus=2-3 \
-chardev
socket,id=charmonitor,host=0.0.0.0,port=10001,telnet=on,server=on,wait=off \
-mon chardev=charmonitor,id=monitor \
-boot order=c \
-device intel-iommu,intremap=on
2- Using pcie_aer_inject_error
Injecting pcie aer error from qemu monitor should be enough to trigger
the kernel panic. This is using the qemu command above.
(qemu) pcie_aer_inject_error pcie_port1 0x00004000
