Public bug reported: Description =========== After successfully attaching a disk to a SEV enabled instance the request to detach the disk never completes with the following trace eventually logged regarding the initial attach:
[ 7.773877] pcieport 0000:00:02.5: Slot(0-5): Attention button pressed [ 7.774743] pcieport 0000:00:02.5: Slot(0-5) Powering on due to button press [ 7.775714] pcieport 0000:00:02.5: Slot(0-5): Card present [ 7.776403] pcieport 0000:00:02.5: Slot(0-5): Link Up [ 7.903183] pci 0000:06:00.0: [1af4:1042] type 00 class 0x010000 [ 7.904095] pci 0000:06:00.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 7.905024] pci 0000:06:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] [ 7.906977] pcieport 0000:00:02.5: bridge window [io 0x1000-0x0fff] to [bus 06] add_size 1000 [ 7.908069] pcieport 0000:00:02.5: BAR 13: no space for [io size 0x1000] [ 7.908917] pcieport 0000:00:02.5: BAR 13: failed to assign [io size 0x1000] [ 7.909832] pcieport 0000:00:02.5: BAR 13: no space for [io size 0x1000] [ 7.910667] pcieport 0000:00:02.5: BAR 13: failed to assign [io size 0x1000] [ 7.911586] pci 0000:06:00.0: BAR 4: assigned [mem 0x800600000-0x800603fff 64bit pref] [ 7.912616] pci 0000:06:00.0: BAR 1: assigned [mem 0x80400000-0x80400fff] [ 7.913472] pcieport 0000:00:02.5: PCI bridge to [bus 06] [ 7.915762] pcieport 0000:00:02.5: bridge window [mem 0x80400000-0x805fffff] [ 7.917525] pcieport 0000:00:02.5: bridge window [mem 0x800600000-0x8007fffff 64bit pref] [ 7.920252] virtio-pci 0000:06:00.0: enabling device (0000 -> 0002) [ 7.924487] virtio_blk virtio4: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.926616] vdb: detected capacity change from 0 to 1073741824 [ .. ] [ 246.751028] INFO: task irq/29-pciehp:173 blocked for more than 120 seconds. [ 246.752801] Not tainted 4.18.0-305.el8.x86_64 #1 [ 246.753902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 246.755457] irq/29-pciehp D 0 173 2 0x80004000 [ 246.756616] Call Trace: [ 246.757328] __schedule+0x2c4/0x700 [ 246.758185] schedule+0x38/0xa0 [ 246.758966] io_schedule+0x12/0x40 [ 246.759801] do_read_cache_page+0x513/0x770 [ 246.760761] ? blkdev_writepages+0x10/0x10 [ 246.761692] ? file_fdatawait_range+0x20/0x20 [ 246.762659] read_part_sector+0x38/0xda [ 246.763554] read_lba+0x10f/0x220 [ 246.764367] efi_partition+0x1e4/0x6de [ 246.765245] ? snprintf+0x49/0x60 [ 246.766046] ? is_gpt_valid.part.5+0x430/0x430 [ 246.766991] blk_add_partitions+0x164/0x3f0 [ 246.767915] ? blk_drop_partitions+0x91/0xc0 [ 246.768863] bdev_disk_changed+0x65/0xd0 [ 246.769748] __blkdev_get+0x3c4/0x510 [ 246.770595] blkdev_get+0xaf/0x180 [ 246.771394] __device_add_disk+0x3de/0x4b0 [ 246.772302] virtblk_probe+0x4ba/0x8a0 [virtio_blk] [ 246.773313] virtio_dev_probe+0x158/0x1f0 [ 246.774208] really_probe+0x255/0x4a0 [ 246.775046] ? __driver_attach_async_helper+0x90/0x90 [ 246.776091] driver_probe_device+0x49/0xc0 [ 246.776965] bus_for_each_drv+0x79/0xc0 [ 246.777813] __device_attach+0xdc/0x160 [ 246.778669] bus_probe_device+0x9d/0xb0 [ 246.779523] device_add+0x418/0x780 [ 246.780321] register_virtio_device+0x9e/0xe0 [ 246.781254] virtio_pci_probe+0xb3/0x140 [ 246.782124] local_pci_probe+0x41/0x90 [ 246.782937] pci_device_probe+0x105/0x1c0 [ 246.783807] really_probe+0x255/0x4a0 [ 246.784623] ? __driver_attach_async_helper+0x90/0x90 [ 246.785647] driver_probe_device+0x49/0xc0 [ 246.786526] bus_for_each_drv+0x79/0xc0 [ 246.787364] __device_attach+0xdc/0x160 [ 246.788205] pci_bus_add_device+0x4a/0x90 [ 246.789063] pci_bus_add_devices+0x2c/0x70 [ 246.789916] pciehp_configure_device+0x91/0x130 [ 246.790855] pciehp_handle_presence_or_link_change+0x334/0x460 [ 246.791985] pciehp_ist+0x1a2/0x1b0 [ 246.792768] ? irq_finalize_oneshot.part.47+0xf0/0xf0 [ 246.793768] irq_thread_fn+0x1f/0x50 [ 246.794550] irq_thread+0xe7/0x170 [ 246.795299] ? irq_forced_thread_fn+0x70/0x70 [ 246.796190] ? irq_thread_check_affinity+0xe0/0xe0 [ 246.797147] kthread+0x116/0x130 [ 246.797841] ? kthread_flush_work_fn+0x10/0x10 [ 246.798735] ret_from_fork+0x22/0x40 [ 246.799523] INFO: task sfdisk:1129 blocked for more than 120 seconds. [ 246.800717] Not tainted 4.18.0-305.el8.x86_64 #1 [ 246.801733] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 246.803155] sfdisk D 0 1129 1107 0x00004080 [ 246.804225] Call Trace: [ 246.804827] __schedule+0x2c4/0x700 [ 246.805590] ? submit_bio+0x3c/0x160 [ 246.806373] schedule+0x38/0xa0 [ 246.807089] schedule_preempt_disabled+0xa/0x10 [ 246.807990] __mutex_lock.isra.6+0x2d0/0x4a0 [ 246.808876] ? wake_up_q+0x80/0x80 [ 246.809636] ? fdatawait_one_bdev+0x20/0x20 [ 246.810508] iterate_bdevs+0x98/0x142 [ 246.811304] ksys_sync+0x6e/0xb0 [ 246.812041] __ia32_sys_sync+0xa/0x10 [ 246.812820] do_syscall_64+0x5b/0x1a0 [ 246.813613] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 246.814652] RIP: 0033:0x7fa9c04924fb [ 246.815431] Code: Unable to access opcode bytes at RIP 0x7fa9c04924d1. [ 246.816655] RSP: 002b:00007fff47661478 EFLAGS: 00000246 ORIG_RAX: 00000000000000a2 [ 246.818047] RAX: ffffffffffffffda RBX: 000055d79fc512f0 RCX: 00007fa9c04924fb [ 246.824526] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d79fc512f0 [ 246.825714] RBP: 0000000000000000 R08: 000055d79fc51012 R09: 0000000000000006 [ 246.826941] R10: 000000000000000a R11: 0000000000000246 R12: 00007fa9c075e6e0 [ 246.828169] R13: 000055d79fc58c80 R14: 0000000000000001 R15: 00007fff47661590 This is caused by the device XML supplied to libvirt missing the driver iommu attribute: <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source dev="/dev/sdc"/> <target bus="virtio" dev="vdb"/> <serial>b11ce83a-723a-49a2-a5cc-025cb8985b0d</serial> </disk> As called out in the original SEV spec this is required: https://specs.openstack.org/openstack/nova-specs/specs/train/implemented /amd-sev-libvirt-support > The iommu attribute is on for all virtio devices. > Despite the name, this does not require the guest > or host to have an IOMMU device, but merely enables > the virtio flag which indicates that virtualized DMA > should be used. This ties into the SEV code to handle > memory encryption/decryption, and prevents IO buffers > being shared between host and guest. > > The DMA will go through bounce buffers, so some > overhead is expected compared to non-SEV guests. > > (Note: virtio-net device queues are not encrypted.) Steps to reproduce ================== 1. Hot plug a PCIe device into a SEV enabled instance. Expected result =============== Hot plug succeeds and the device is visible within the instance. Actual result ============= Hot plug appears to succeed but the device is never present within the instance and a trace is later logged. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== [OSP 16.2] Volumes and vNICs are being hot plugged into SEV based instances without iommu='on' causing failures to attach and later detach within the guest OS https://bugzilla.redhat.com/show_bug.cgi?id=1967293 ** Affects: nova Importance: Undecided Assignee: Lee Yarwood (lyarwood) Status: New ** Tags: libvirt -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1930734 Title: Volumes and vNICs are being hot plugged into SEV based instances without iommu='on' causing failures to attach and later detach within the guest OS Status in OpenStack Compute (nova): New Bug description: Description =========== After successfully attaching a disk to a SEV enabled instance the request to detach the disk never completes with the following trace eventually logged regarding the initial attach: [ 7.773877] pcieport 0000:00:02.5: Slot(0-5): Attention button pressed [ 7.774743] pcieport 0000:00:02.5: Slot(0-5) Powering on due to button press [ 7.775714] pcieport 0000:00:02.5: Slot(0-5): Card present [ 7.776403] pcieport 0000:00:02.5: Slot(0-5): Link Up [ 7.903183] pci 0000:06:00.0: [1af4:1042] type 00 class 0x010000 [ 7.904095] pci 0000:06:00.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 7.905024] pci 0000:06:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] [ 7.906977] pcieport 0000:00:02.5: bridge window [io 0x1000-0x0fff] to [bus 06] add_size 1000 [ 7.908069] pcieport 0000:00:02.5: BAR 13: no space for [io size 0x1000] [ 7.908917] pcieport 0000:00:02.5: BAR 13: failed to assign [io size 0x1000] [ 7.909832] pcieport 0000:00:02.5: BAR 13: no space for [io size 0x1000] [ 7.910667] pcieport 0000:00:02.5: BAR 13: failed to assign [io size 0x1000] [ 7.911586] pci 0000:06:00.0: BAR 4: assigned [mem 0x800600000-0x800603fff 64bit pref] [ 7.912616] pci 0000:06:00.0: BAR 1: assigned [mem 0x80400000-0x80400fff] [ 7.913472] pcieport 0000:00:02.5: PCI bridge to [bus 06] [ 7.915762] pcieport 0000:00:02.5: bridge window [mem 0x80400000-0x805fffff] [ 7.917525] pcieport 0000:00:02.5: bridge window [mem 0x800600000-0x8007fffff 64bit pref] [ 7.920252] virtio-pci 0000:06:00.0: enabling device (0000 -> 0002) [ 7.924487] virtio_blk virtio4: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.926616] vdb: detected capacity change from 0 to 1073741824 [ .. ] [ 246.751028] INFO: task irq/29-pciehp:173 blocked for more than 120 seconds. [ 246.752801] Not tainted 4.18.0-305.el8.x86_64 #1 [ 246.753902] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 246.755457] irq/29-pciehp D 0 173 2 0x80004000 [ 246.756616] Call Trace: [ 246.757328] __schedule+0x2c4/0x700 [ 246.758185] schedule+0x38/0xa0 [ 246.758966] io_schedule+0x12/0x40 [ 246.759801] do_read_cache_page+0x513/0x770 [ 246.760761] ? blkdev_writepages+0x10/0x10 [ 246.761692] ? file_fdatawait_range+0x20/0x20 [ 246.762659] read_part_sector+0x38/0xda [ 246.763554] read_lba+0x10f/0x220 [ 246.764367] efi_partition+0x1e4/0x6de [ 246.765245] ? snprintf+0x49/0x60 [ 246.766046] ? is_gpt_valid.part.5+0x430/0x430 [ 246.766991] blk_add_partitions+0x164/0x3f0 [ 246.767915] ? blk_drop_partitions+0x91/0xc0 [ 246.768863] bdev_disk_changed+0x65/0xd0 [ 246.769748] __blkdev_get+0x3c4/0x510 [ 246.770595] blkdev_get+0xaf/0x180 [ 246.771394] __device_add_disk+0x3de/0x4b0 [ 246.772302] virtblk_probe+0x4ba/0x8a0 [virtio_blk] [ 246.773313] virtio_dev_probe+0x158/0x1f0 [ 246.774208] really_probe+0x255/0x4a0 [ 246.775046] ? __driver_attach_async_helper+0x90/0x90 [ 246.776091] driver_probe_device+0x49/0xc0 [ 246.776965] bus_for_each_drv+0x79/0xc0 [ 246.777813] __device_attach+0xdc/0x160 [ 246.778669] bus_probe_device+0x9d/0xb0 [ 246.779523] device_add+0x418/0x780 [ 246.780321] register_virtio_device+0x9e/0xe0 [ 246.781254] virtio_pci_probe+0xb3/0x140 [ 246.782124] local_pci_probe+0x41/0x90 [ 246.782937] pci_device_probe+0x105/0x1c0 [ 246.783807] really_probe+0x255/0x4a0 [ 246.784623] ? __driver_attach_async_helper+0x90/0x90 [ 246.785647] driver_probe_device+0x49/0xc0 [ 246.786526] bus_for_each_drv+0x79/0xc0 [ 246.787364] __device_attach+0xdc/0x160 [ 246.788205] pci_bus_add_device+0x4a/0x90 [ 246.789063] pci_bus_add_devices+0x2c/0x70 [ 246.789916] pciehp_configure_device+0x91/0x130 [ 246.790855] pciehp_handle_presence_or_link_change+0x334/0x460 [ 246.791985] pciehp_ist+0x1a2/0x1b0 [ 246.792768] ? irq_finalize_oneshot.part.47+0xf0/0xf0 [ 246.793768] irq_thread_fn+0x1f/0x50 [ 246.794550] irq_thread+0xe7/0x170 [ 246.795299] ? irq_forced_thread_fn+0x70/0x70 [ 246.796190] ? irq_thread_check_affinity+0xe0/0xe0 [ 246.797147] kthread+0x116/0x130 [ 246.797841] ? kthread_flush_work_fn+0x10/0x10 [ 246.798735] ret_from_fork+0x22/0x40 [ 246.799523] INFO: task sfdisk:1129 blocked for more than 120 seconds. [ 246.800717] Not tainted 4.18.0-305.el8.x86_64 #1 [ 246.801733] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 246.803155] sfdisk D 0 1129 1107 0x00004080 [ 246.804225] Call Trace: [ 246.804827] __schedule+0x2c4/0x700 [ 246.805590] ? submit_bio+0x3c/0x160 [ 246.806373] schedule+0x38/0xa0 [ 246.807089] schedule_preempt_disabled+0xa/0x10 [ 246.807990] __mutex_lock.isra.6+0x2d0/0x4a0 [ 246.808876] ? wake_up_q+0x80/0x80 [ 246.809636] ? fdatawait_one_bdev+0x20/0x20 [ 246.810508] iterate_bdevs+0x98/0x142 [ 246.811304] ksys_sync+0x6e/0xb0 [ 246.812041] __ia32_sys_sync+0xa/0x10 [ 246.812820] do_syscall_64+0x5b/0x1a0 [ 246.813613] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 246.814652] RIP: 0033:0x7fa9c04924fb [ 246.815431] Code: Unable to access opcode bytes at RIP 0x7fa9c04924d1. [ 246.816655] RSP: 002b:00007fff47661478 EFLAGS: 00000246 ORIG_RAX: 00000000000000a2 [ 246.818047] RAX: ffffffffffffffda RBX: 000055d79fc512f0 RCX: 00007fa9c04924fb [ 246.824526] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000055d79fc512f0 [ 246.825714] RBP: 0000000000000000 R08: 000055d79fc51012 R09: 0000000000000006 [ 246.826941] R10: 000000000000000a R11: 0000000000000246 R12: 00007fa9c075e6e0 [ 246.828169] R13: 000055d79fc58c80 R14: 0000000000000001 R15: 00007fff47661590 This is caused by the device XML supplied to libvirt missing the driver iommu attribute: <disk type="block" device="disk"> <driver name="qemu" type="raw" cache="none" io="native"/> <source dev="/dev/sdc"/> <target bus="virtio" dev="vdb"/> <serial>b11ce83a-723a-49a2-a5cc-025cb8985b0d</serial> </disk> As called out in the original SEV spec this is required: https://specs.openstack.org/openstack/nova- specs/specs/train/implemented/amd-sev-libvirt-support > The iommu attribute is on for all virtio devices. > Despite the name, this does not require the guest > or host to have an IOMMU device, but merely enables > the virtio flag which indicates that virtualized DMA > should be used. This ties into the SEV code to handle > memory encryption/decryption, and prevents IO buffers > being shared between host and guest. > > The DMA will go through bounce buffers, so some > overhead is expected compared to non-SEV guests. > > (Note: virtio-net device queues are not encrypted.) Steps to reproduce ================== 1. Hot plug a PCIe device into a SEV enabled instance. Expected result =============== Hot plug succeeds and the device is visible within the instance. Actual result ============= Hot plug appears to succeed but the device is never present within the instance and a trace is later logged. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== [OSP 16.2] Volumes and vNICs are being hot plugged into SEV based instances without iommu='on' causing failures to attach and later detach within the guest OS https://bugzilla.redhat.com/show_bug.cgi?id=1967293 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1930734/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
