[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
It is verified SEV works for the v5.15 kernel where we wants SEV to be supported. Closing this one for v5.11 now. ** Changed in: linux-oracle-5.11 (Ubuntu) Status: New => Invalid -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-oracle-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1980884 Title: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used Status in linux-oracle-5.11 package in Ubuntu: Invalid Bug description: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch libiscsi : libiscsi-1.19.0-1.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91.084591] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-1028-oracle #31~20.04.1-Ubuntu [ 91.085393] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.2 06/01/2022 [ 91.086205] RIP: 0010:memcpy_erms+0x6/0x10 [ 91.086640] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe [ 91.088559] RSP: 0018:a9c1408e4b60 EFLAGS: 00010282 [ 91.089105] RAX: 938cd8e48000 RBX: 1000 RCX: 1000 [ 91.089843] RDX: 1000 RSI: bb6
[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
Thanks for the info, Fabio. I didn't realize the 5.11 kernel is past EoL already. I'll ask the team to check the result on v5.4 and v5.15 kernel. Stay tuned. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-oracle-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1980884 Title: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used Status in linux-oracle-5.11 package in Ubuntu: New Bug description: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch libiscsi : libiscsi-1.19.0-1.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91.084591] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-1028-oracle #31~20.04.1-Ubuntu [ 91.085393] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.2 06/01/2022 [ 91.086205] RIP: 0010:memcpy_erms+0x6/0x10 [ 91.086640] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe [ 91.088559] RSP: 0018:a9c1408e4b60 EFLAGS: 00010282 [ 91.089105] RAX: 938cd8e48000 RBX: 1000 RCX: 1000 [ 91.089843] RDX: 1000 RSI: bb62fcf4fd5bf3d6 RDI: 938cd8e4
[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
Hi Si-Wei, the 5.11 kernel has reached EOL in Feb 2022. Kernel 5.15 is the one currently being used for linux-oracle kernel on Focal (20.04) and Jammy (22.04), and it has the commit that you mentioned above: $ git log --oneline | grep -i "Fix page DMA map/unmap attributes" a865fe280b96 net/mlx5e: Fix page DMA map/unmap attributes $ git tag --contains a865fe280b96 Ubuntu-oracle-5.15.0-1001.1 Ubuntu-oracle-5.15.0-1001.2 Ubuntu-oracle-5.15.0-1001.3 Ubuntu-oracle-5.15.0-1002.4 Ubuntu-oracle-5.15.0-1003.5 Ubuntu-oracle-5.15.0-1004.6 Ubuntu-oracle-5.15.0-1005.7 Ubuntu-oracle-5.15.0-1006.8 Ubuntu-oracle-5.15.0-1007.9 Ubuntu-oracle-5.15.0-1009.12 Ubuntu-oracle-5.15.0-1011.15 Ubuntu-oracle-5.15.0-1012.16 Ubuntu-oracle-5.15.0-1013.17 Can you test a guest running 5.15 to see if this addresses the problem? Regards, Fabio Martins -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-oracle-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1980884 Title: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used Status in linux-oracle-5.11 package in Ubuntu: New Bug description: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch libiscsi : libiscsi-1.19.0-1.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91
[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
That is also available in the 5.4 kernel, so that also covers Bionic (18.04) guests if needed: $ git log --oneline | grep -i "Fix page DMA map/unmap attributes" 53176ef0d809 net/mlx5e: Fix page DMA map/unmap attributes $ git tag --contains 53176ef0d809 Ubuntu-oracle-5.4.0-1071.77 Ubuntu-oracle-5.4.0-1072.78 Ubuntu-oracle-5.4.0-1073.79 Ubuntu-oracle-5.4.0-1074.80 Ubuntu-oracle-5.4.0-1076.83 Ubuntu-oracle-5.4.0-1078.86 Ubuntu-oracle-5.4.0-1079.87 Ubuntu-oracle-5.4.0-1080.88 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-oracle-5.11 in Ubuntu. https://bugs.launchpad.net/bugs/1980884 Title: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used Status in linux-oracle-5.11 package in Ubuntu: New Bug description: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch libiscsi : libiscsi-1.19.0-1.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91.084591] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-1028-oracle #31~20.04.1-Ubuntu [ 91.085393] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.2 06/01/2022 [ 91.086205] RIP: 0010:memcpy_erms+0x6/0x10 [ 91.086640] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
** Description changed: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 - Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.oci.el7) + Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch - libiscsi : libiscsi-1.19.0-1.oci.el7.x86_64 + libiscsi : libiscsi-1.19.0-1.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91.084591] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-1028-oracle #31~20.04.1-Ubuntu [ 91.085393] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.2 06/01/2022 [ 91.086205] RIP: 0010:memcpy_erms+0x6/0x10 [ 91.086640] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe [ 91.088559] RSP: 0018:a9c1408e4b60 EFLAGS: 00010282 [ 91.089105] RAX: 938cd8e48000 RBX: 1000 RCX: 1000 [ 91.089843] RDX: 1000 RSI: bb62fcf4fd5bf3d6 RDI: 938cd8e48000 [ 91.090578] RBP: a9c1408e4c00 R08: ef2745639200 R09: [ 91.091309] R10: ef27456399c8 R11: 4209 R12: 1000 [ 91.092043] R13: ef2745639200 R14: 1000 R15: 0d558380 [ 91.092782] FS: () GS:938df430
[Kernel-packages] [Bug 1980884] Re: ubuntu guest kernel panics when a sev guest with passthrough mlx5 VF is used
** Description changed: Guest kernel panic can be observed when Ubuntu SEV guest with mlx5 vfio-pci is started as iperf3 server using "iperf3 -s" and as soon as the client tries to connect with it. Steps to reproduce: HOST INFO Host type : OCI (Oracle Cloud) Bare-Metal Server Server/Machine: ORACLE SERVER E4-2c CPU model : AMD EPYC 7J13 64-Core Processor Architecture : x86_64 Host OS : Oracle Linux Server release 7.9 Host Kernel : 5.4.17-2136.309.3.el7uek.x86_64 #2 SMP Tue Jun 14 21:58:29 PDT 2022 Hypervisor: QEMU emulator version 4.2.1 (qemu-4.2.1-17.1.oci.el7) OVMF/AAVMF: OVMF-1.6.2-2.el7.noarch libiscsi : libiscsi-1.19.0-1.oci.el7.x86_64 Guest Kernel : 5.11.0-1028-ORACLE - 1) Start Ubuntu 20.04/18.04 SEV guest with vfio-pci: /usr/bin/qemu-system-x86_64 -machine q35 -name OL20.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol20.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true - -fw_cfg name=opt/ovmf/OCI-ENABLE,string=0x1 -drive + -drive file=/systest/atanveer/scripts/Ubuntu-20.04-2022.02.15-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.1 -qmp tcp:127.0.0.1:3334,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL20.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 2) Start a client guest OL/Ubuntu: /usr/bin/qemu-system-x86_64 -machine q35 -name OL18.04-uefi -enable-kvm -nodefaults -cpu host,+host-phys-bits -m 8G -smp 8,maxcpus=240 -monitor stdio -vnc 0.0.0.0:0,to=999 -vga std -drive file=/usr/share/OVMF/OVMF_CODE.pure-efi.fd,index=0,if=pflash,format=raw,readonly -drive file=OVMF_VARS.pure-efi.fd.ol18.04,index=1,if=pflash,format=raw -device virtio-scsi-pci,id=virtio-scsi-pci0,disable-legacy=on,iommu_platform=true - -fw_cfg name=opt/ovmf/OCI-ENABLE,string=0x1 -drive + -drive file=/systest/atanveer/scripts/Ubuntu-18.04-2022.02.13-0-uefi-x86_64.qcow2,if=none,id=local_disk0,format=qcow2,media=disk -device ide-hd,drive=local_disk0,id=local_disk1,bootindex=0 -net none -device vfio-pci,host=:21:10.2 -qmp tcp:127.0.0.1:,server,nowait -serial telnet:127.0.0.1:,server,nowait -D ./OL18.04-uefi.log -device virtio-rng-pci,disable-legacy=on,iommu_platform=true -object sev-guest,id=sev0,cbitpos=51,reduced-phys-bits=1 -machine memory-encryption=sev0 3) Flush iptables on both the VMs using "iptables -F" 4) Start the iperf3 server on the first VM using "iperf3 -s" 5) Start the iperf3 client on the second VM using "iperf3 -c -4 -f M -i 0 -t 70 -O 10 -P 64" The kernel panic is seen on the first VM i.e. Ubuntu 20.04 with iperf3 also showing "Bad Address" error. - Console logs: root@ubuntu-20-04:~# iperf3 -s --- Server listening on 5201 --- Accepted connection from 10.196.246.104, port 33732 [ 5] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33734 [ 8] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33736 [ 10] local 10.196.247.88 port 5201 connected to 10.196.246.104 port 33738 iperf3: error - unable to read from stream socket: Bad address --- Server listening on 5201 --- [ 91.083856] general protection fault: [#1] SMP NOPTI [ 91.084591] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.11.0-1028-oracle #31~20.04.1-Ubuntu [ 91.085393] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.6.2 06/01/2022 [ 91.086205] RIP: 0010:memcpy_erms+0x6/0x10 [ 91.086640] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe [ 91.088559] RSP: 0018:a9c1408e4b60 EFLAGS: 00010282 [ 91.089105] RAX: 938cd8e48000 RBX: 1000 RCX: 1000 [ 91.089843] RDX: 1000 RSI: bb62fcf4fd5bf3d6 RDI: 938cd8e48000 [ 91.090578] RBP: a9c1408e4c00 R08: ef2745639200 R09: [ 91.091309] R10: ef27456399c8 R11: 4209 R12: 1000 [ 91.092043] R13: ef2745639200 R14: 1000 R15: 0d558380 [ 91.092782] FS: () GS:938df430(
