Hi We have a chassis with a peripheral PCI device installed. We have a hypervisor running on the chassis where we have deployed a VM which can use the PCI device once it is attached. When the PCI device is powered on or off we need to do a hotplug in/out using virsh commands. Even though the virsh commands for hotplug is executed successfully, the VM sees the PCI inside the VM for one or 2 secs. What do you think is the issue? [root ~]# uname -a Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Mar 15 05:32:38 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux [root ~]# virsh --version 8.0.0 [root ~]# rpm -qa | grep libvirt libvirt-daemon-driver-nwfilter-8.0.0-5.el8.x86_64 python3-libvirt-7.8.0-1.el8.x86_64 libvirt-daemon-driver-storage-logical-8.0.0-5.el8.x86_64 libvirt-libs-8.0.0-5.el8.x86_64 libvirt-daemon-config-nwfilter-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-gluster-8.0.0-5.el8.x86_64 libvirt-8.0.0-5.el8.x86_64 libvirt-daemon-8.0.0-5.el8.x86_64 libvirt-daemon-driver-nodedev-8.0.0-5.el8.x86_64 libvirt-daemon-config-network-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-iscsi-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-rbd-8.0.0-5.el8.x86_64 libvirt-daemon-driver-network-8.0.0-5.el8.x86_64 libvirt-daemon-driver-secret-8.0.0-5.el8.x86_64 python2-libvirt-python-5.10.0-1.el8.x86_64 libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-core-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-iscsi-direct-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-scsi-8.0.0-5.el8.x86_64 libvirt-client-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-disk-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-8.0.0-5.el8.x86_64 libvirt-daemon-driver-interface-8.0.0-5.el8.x86_64 libvirt-daemon-driver-storage-mpath-8.0.0-5.el8.x86_64 libvirt-daemon-kvm-8.0.0-5.el8.x86_64 [root ~]# rpm -qa | grep qemu qemu-kvm-ui-opengl-6.2.0-11.el8.x86_64 qemu-kvm-6.2.0-11.el8.x86_64 qemu-img-6.2.0-11.el8.x86_64 qemu-kvm-block-iscsi-6.2.0-11.el8.x86_64 ipxe-roms-qemu-20200823-7.git4bd064de.el8.noarch qemu-kvm-block-gluster-6.2.0-11.el8.x86_64 qemu-kvm-block-rbd-6.2.0-11.el8.x86_64 qemu-kvm-block-curl-6.2.0-11.el8.x86_64 qemu-kvm-core-6.2.0-11.el8.x86_64 qemu-kvm-hw-usbredir-6.2.0-11.el8.x86_64 libvirt-daemon-driver-qemu-8.0.0-5.el8.x86_64 qemu-kvm-ui-spice-6.2.0-11.el8.x86_64 qemu-kvm-docs-6.2.0-11.el8.x86_64 qemu-kvm-block-ssh-6.2.0-11.el8.x86_64 qemu-kvm-common-6.2.0-11.el8.x86_64
[root ~]# virsh nodedev-dettach pci_0000_04_00_0 [12013.987821] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12014.063669] pci_slot_reset (printk info): reset hotplug slot. [12014.134164] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [Mar19 21:09] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.075848] pci_slot_reset (printk info): reset hotplug slot. [ +0.070495] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) Device pci_0000_04_00_0 detached [root ~]# virsh attach-device ROUTER8 /opt/us/bin/mrvl.xml [12024.217540] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +10.083376] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.293548] pci_slot_reset (printk info): reset hotplug slot. [12024.434828] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.076008] pci_slot_reset (printk info): reset hotplug slot. [ +0.141280] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)4.526807] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12024.749367] pci_slot_reset (printk info): reset hotplug slot. [12024.821327] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.091979] pci_probe_reset_slot: call pci_slot_reset with probe=1[12024.911065] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.222560] pci_slot_reset (printk info): reset hotplug slot. [ +0.071960] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) 5.057584] pci_slot_reset (printk info): reset hotplug slot. [ +0.089738] pci_probe_reset_slot: call pci_slot_reset with probe=1 [12025.278585] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.146519] pci_slot_reset (printk info): reset hotplug slot. [ +0.221001] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) 5.441574] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.162989] pci_probe_reset_slot: call pci_slot_reset with probe=1[12025.665968] pci_slot_reset (printk info): reset hotplug slot. [12025.806751] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12025.898060] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. [ +0.224394] pci_slot_reset (printk info): reset hotplug slot. [ +0.140783] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)5.988477] pci_slot_reset (printk info): reset hotplug slot. [ +0.091309] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. 12026.205654] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12026.383424] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot. [ +0.090417] pci_slot_reset (printk info): reset hotplug slot. [ +0.217177] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)6.490406] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0) [ +0.177770] __pci_reset_slot (printk info): pci_slot_trylock is non-zero, so reset hotplug slot. 2026.726579] pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [ +0.106982] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 0) [ +0.236167] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 0 [12027.744480] pending interrupts 0x0108 from Slot Status [ +1.017890] pcieport 0000:00:0e.0: pciehp: pending interrupts 0x0108 from Slot Status [ +0.000011] pending interrupts 0x0108 from Slot Status [12027.936469] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008 [ +0.191983] pcieport 0000:00:0e.0: pciehp: pciehp_reset_slot: SLOTCTRL 58 write cmd 100812028.027182] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down [12028.190357] Slot(4): Link Down [ +0.000006] pciehp_reset_slot: SLOTCTRL 58 write cmd 1008[12028.230605] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present [12028.369819] Slot(4): Card not present Device attached successfully [12028.415302] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.090713] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Down[12028.524356] vfio-pci 0000:04:00.0: Relaying device request to user (#0) [ +0.163175] Slot(4): Link Down [ +0.040248] pcieport 0000:00:0e.0: pciehp: Slot(4): Card not present [ +0.139214] Slot(4): Card not present [ +0.045479] pcieport 0000:00:0e.0: pciehp: pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.000004] pciehp_unconfigure_device: domain:bus:dev = 0000:04:00 [ +0.109054] vfio-pci 0000:04:00.0: Relaying device request to user (#0) 9119] pci_probe_reset_slot: call pci_slot_reset with probe=1 [root@nfvis ~]# [12029.151422] pci_slot_reset (printk info): reset hotplug slot. [12029.237806] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.334763] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.327572] pci_probe_reset_slot: call pci_slot_reset with probe=1 [ +0.292303] pci_slot_reset (printk info): reset hotplug slot. [ +0.086384] pci_r12029.474447] pci_slot_reset (printk info): reset hotplug slot. eset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12029.632533] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.089766] pci_probe_reset_slot: call pci_slot_reset with probe=1[12029.788423] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. [ +0.146875] pci_slot_reset (printk info): reset hotplug slot. [12029.950782] pci_slot_reset (printk info): reset hotplug slot. [ +0.158086] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [ +0.155890] __pci_reset_slot (printk info): Reset slot (not hotplug), probe = 1. .089787] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1) [12030.346708] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot. [ +0.162359] pci_slot_reset (printk info): reset hotplug slot.[12030.451555] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible) [ +0.139005] pci_reset_hotplug_slot(printk INFO): calling reset_slot (probe = 1)12030.632974] pci 0000:04:00.0: Removing from iommu group 58 [ +0.256921] __pci_reset_slot (printk info): pci_slot_trylock = 0, so don't reset hotplug slot. [ +0.104847] vfio-pci 0000:04:00.0: can't change power state from D0 to D3hot (config space inaccessible) 5279] pciehp_check_link_active: lnk_status = 7011 [ +0.181419] pci 0000:04:00.0: Removing from iommu group 58 [12031.056747] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present [12031.194871] Slot(4): Card present [ +0.152301] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_active: lnk_status = 701112031.236173] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up [12031.395181] Slot(4): Link Up [ +0.000004] pciehp_check_link_active: lnk_status = 7011 [ +0.271468] pcieport 0000:00:0e.0: pciehp: Slot(4): Card present [ +0.138124] Slot(4): Card present [ +0.041302] pcieport 0000:00:0e.0: pciehp: Slot(4): Link Up [ +0.159008] Slot(4): Link Up 31.560448] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB [12031.803909] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [12031.909784] pciehp_check_link_status: lnk_status = 7011 [ +0.165267] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB2031.974018] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB [12032.176762] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [ +0.243461] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB [ +0.105871] pcieport 0000:00:0e.0: pciehp: pciehp_check_link_status: lnk_status = 7011282681] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000 [ +0.000004] pciehp_check_link_status: lnk_status = 7011 [12032.545840] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref] [12032.701716] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.064234] read_dev_vendor_id: read config_dword worked. bus = 4, dev_vendor id = 0xE61E11AB2032.794062] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref] [12032.984416] pci 0000:04:00.0: supports D1 D2 [ +0.202744] read_dev_vendor_id: device found, return true. bus = 4, dev_vendor id = 0xE61E11AB2033.039350] pci 0000:04:00.0: Adding to iommu group 58 [12033.201376] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref] [12033.299989] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.105919] pci 0000:04:00.0: [11ab:e61e] type 00 class 0x020000[12033.400664] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref] [ +0.263159] pci 0000:04:00.0: reg 0x10: [mem 0x7b44000000-0x7b440fffff 64bit pref]12033.568019] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06] [12033.725967] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff] [ +0.155876] pci 0000:04:00.0: reg 0x18: [mem 0x7b40000000-0x7b43ffffff 64bit pref]12033.805813] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff] [12033.982533] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref] [ +0.092346] pci 0000:04:00.0: reg 0x20: [mem 0x7b30000000-0x7b3fffffff 64bit pref]12034.086495] pci-stub 0000:04:00.0: claimed by stub [ +0.190354] pci 0000:04:00.0: supports D1 D2 [ +0.054934] pci 0000:04:00.0: Adding to iommu group 58 [ +0.162026] pci 0000:04:00.0: BAR 4: assigned [mem 0x7b30000000-0x7b3fffffff 64bit pref] [ +0.098613] pci 0000:04:00.0: BAR 2: assigned [mem 0x7b40000000-0x7b43ffffff 64bit pref] [ +0.100675] pci 0000:04:00.0: BAR 0: assigned [mem 0x7b44000000-0x7b440fffff 64bit pref] [ +0.167355] pcieport 0000:00:0e.0: PCI bridge to [bus 04-06] [ +0.157948] pcieport 0000:00:0e.0: bridge window [io 0xd000-0xdfff] [ +0.079846] pcieport 0000:00:0e.0: bridge window [mem 0xdc000000-0xdcffffff] [ +0.176720] pcieport 0000:00:0e.0: bridge window [mem 0x7b30000000-0x7b6fffffff 64bit pref] [ +0.103962] pci-stub 0000:04:00.0: claimed by stub Thank you for replying in advance, -Chanda
