Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
On Feb 15 12:01, Major Saheb wrote: > > Assuming you are *not* explicitly configuring shadow doorbells, then I > > think you might have a broken driver that does not properly reset the > > controller before using it (are you tripping CC.EN?). That could explain > > the admin queue size of 32 (default admin queue depth for the Linux nvme > > driver) as well as the db/ei_addrs being left over. And behavior wrt. > > how the Linux driver disables the device might have changed between the > > kernel version used in Ubuntu 20.04 and 22.04. > > Thanks Klaus, I didn't had the driver source, so I acquired it and > looked into it, the driver was not toggling the cc.en nor waiting for > csts.ready the right way. So I implemented it and it started working > perfectly. > - R > > On Tue, Feb 14, 2023 at 8:26 PM Klaus Jensen wrote: > > > > On Feb 14 14:05, Klaus Jensen wrote: > > > On Feb 14 17:34, Major Saheb wrote: > > > > Thanks Peter for the reply. I tried to connect gdb to qemu and able to > > > > break 'vtd_iova_to_slpte()', I dumped the following with both Ubuntu > > > > 20.04 base image container which is the success case and Ubuntu 22.04 > > > > base image container which is failure case > > > > One thing I observed is the NvmeSQueue::dma_addr is correctly set to > > > > '0x8', however in failure case this value is 0x1196b1000. A > > > > closer look indicates more fields in NvmeSQueue might be corrupted, > > > > for example we are setting admin queue size as 512 but in failure case > > > > it is showing 32. > > > > > > > > > > Hi Major, > > > > > > It's obviously pretty bad if hw/nvme somehow corrupts the SQ structure, > > > but it's difficult to say from this output. > > > > > > Are you configuring shadow doorbells (the db_addr and ei_addr's are > > > set in both cases)? > > > > > > > > > Following is the partial qemu command line that I am using > > > > > > > > > > > > -device > > > > > > intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 > > > > > > > > > > > > I'm not sure if caching-mode=on and device-iotlb=on leads to any issues > > > here? As far as I understand, this is mostly used with stuff like vhost. > > > I've tested and developed vfio-based drivers against hw/nvme excessively > > > and I'm not using anything besides `-device intel-iommu`. > > > > > > Do I undestand correctly that your setup is "just" a Ubuntu 22.04 guest > > > with a container and a user-space driver to interact with the nvme > > > devices available on the guest? No nested virtualization with vfio > > > passthrough? > > > > Assuming you are *not* explicitly configuring shadow doorbells, then I > > think you might have a broken driver that does not properly reset the > > controller before using it (are you tripping CC.EN?). That could explain > > the admin queue size of 32 (default admin queue depth for the Linux nvme > > driver) as well as the db/ei_addrs being left over. And behavior wrt. > > how the Linux driver disables the device might have changed between the > > kernel version used in Ubuntu 20.04 and 22.04. Awesome. Occam's Razor strikes again ;) signature.asc Description: PGP signature
Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
> Assuming you are *not* explicitly configuring shadow doorbells, then I > think you might have a broken driver that does not properly reset the > controller before using it (are you tripping CC.EN?). That could explain > the admin queue size of 32 (default admin queue depth for the Linux nvme > driver) as well as the db/ei_addrs being left over. And behavior wrt. > how the Linux driver disables the device might have changed between the > kernel version used in Ubuntu 20.04 and 22.04. Thanks Klaus, I didn't had the driver source, so I acquired it and looked into it, the driver was not toggling the cc.en nor waiting for csts.ready the right way. So I implemented it and it started working perfectly. - R On Tue, Feb 14, 2023 at 8:26 PM Klaus Jensen wrote: > > On Feb 14 14:05, Klaus Jensen wrote: > > On Feb 14 17:34, Major Saheb wrote: > > > Thanks Peter for the reply. I tried to connect gdb to qemu and able to > > > break 'vtd_iova_to_slpte()', I dumped the following with both Ubuntu > > > 20.04 base image container which is the success case and Ubuntu 22.04 > > > base image container which is failure case > > > One thing I observed is the NvmeSQueue::dma_addr is correctly set to > > > '0x8', however in failure case this value is 0x1196b1000. A > > > closer look indicates more fields in NvmeSQueue might be corrupted, > > > for example we are setting admin queue size as 512 but in failure case > > > it is showing 32. > > > > > > > Hi Major, > > > > It's obviously pretty bad if hw/nvme somehow corrupts the SQ structure, > > but it's difficult to say from this output. > > > > Are you configuring shadow doorbells (the db_addr and ei_addr's are > > set in both cases)? > > > > > > > Following is the partial qemu command line that I am using > > > > > > > > > > -device > > > > > intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 > > > > > > > > > I'm not sure if caching-mode=on and device-iotlb=on leads to any issues > > here? As far as I understand, this is mostly used with stuff like vhost. > > I've tested and developed vfio-based drivers against hw/nvme excessively > > and I'm not using anything besides `-device intel-iommu`. > > > > Do I undestand correctly that your setup is "just" a Ubuntu 22.04 guest > > with a container and a user-space driver to interact with the nvme > > devices available on the guest? No nested virtualization with vfio > > passthrough? > > Assuming you are *not* explicitly configuring shadow doorbells, then I > think you might have a broken driver that does not properly reset the > controller before using it (are you tripping CC.EN?). That could explain > the admin queue size of 32 (default admin queue depth for the Linux nvme > driver) as well as the db/ei_addrs being left over. And behavior wrt. > how the Linux driver disables the device might have changed between the > kernel version used in Ubuntu 20.04 and 22.04.
Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
On Feb 14 14:05, Klaus Jensen wrote: > On Feb 14 17:34, Major Saheb wrote: > > Thanks Peter for the reply. I tried to connect gdb to qemu and able to > > break 'vtd_iova_to_slpte()', I dumped the following with both Ubuntu > > 20.04 base image container which is the success case and Ubuntu 22.04 > > base image container which is failure case > > One thing I observed is the NvmeSQueue::dma_addr is correctly set to > > '0x8', however in failure case this value is 0x1196b1000. A > > closer look indicates more fields in NvmeSQueue might be corrupted, > > for example we are setting admin queue size as 512 but in failure case > > it is showing 32. > > > > Hi Major, > > It's obviously pretty bad if hw/nvme somehow corrupts the SQ structure, > but it's difficult to say from this output. > > Are you configuring shadow doorbells (the db_addr and ei_addr's are > set in both cases)? > > > > > Following is the partial qemu command line that I am using > > > > > > > > -device > > > > intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 > > > > > > I'm not sure if caching-mode=on and device-iotlb=on leads to any issues > here? As far as I understand, this is mostly used with stuff like vhost. > I've tested and developed vfio-based drivers against hw/nvme excessively > and I'm not using anything besides `-device intel-iommu`. > > Do I undestand correctly that your setup is "just" a Ubuntu 22.04 guest > with a container and a user-space driver to interact with the nvme > devices available on the guest? No nested virtualization with vfio > passthrough? Assuming you are *not* explicitly configuring shadow doorbells, then I think you might have a broken driver that does not properly reset the controller before using it (are you tripping CC.EN?). That could explain the admin queue size of 32 (default admin queue depth for the Linux nvme driver) as well as the db/ei_addrs being left over. And behavior wrt. how the Linux driver disables the device might have changed between the kernel version used in Ubuntu 20.04 and 22.04. signature.asc Description: PGP signature
Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
On Feb 14 17:34, Major Saheb wrote: > Thanks Peter for the reply. I tried to connect gdb to qemu and able to > break 'vtd_iova_to_slpte()', I dumped the following with both Ubuntu > 20.04 base image container which is the success case and Ubuntu 22.04 > base image container which is failure case > One thing I observed is the NvmeSQueue::dma_addr is correctly set to > '0x8', however in failure case this value is 0x1196b1000. A > closer look indicates more fields in NvmeSQueue might be corrupted, > for example we are setting admin queue size as 512 but in failure case > it is showing 32. > Hi Major, It's obviously pretty bad if hw/nvme somehow corrupts the SQ structure, but it's difficult to say from this output. Are you configuring shadow doorbells (the db_addr and ei_addr's are set in both cases)? > > > Following is the partial qemu command line that I am using > > > > > > -device > > > intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 > > > I'm not sure if caching-mode=on and device-iotlb=on leads to any issues here? As far as I understand, this is mostly used with stuff like vhost. I've tested and developed vfio-based drivers against hw/nvme excessively and I'm not using anything besides `-device intel-iommu`. Do I undestand correctly that your setup is "just" a Ubuntu 22.04 guest with a container and a user-space driver to interact with the nvme devices available on the guest? No nested virtualization with vfio passthrough? signature.asc Description: PGP signature
Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
Thanks Peter for the reply. I tried to connect gdb to qemu and able to
break 'vtd_iova_to_slpte()', I dumped the following with both Ubuntu
20.04 base image container which is the success case and Ubuntu 22.04
base image container which is failure case
One thing I observed is the NvmeSQueue::dma_addr is correctly set to
'0x8', however in failure case this value is 0x1196b1000. A
closer look indicates more fields in NvmeSQueue might be corrupted,
for example we are setting admin queue size as 512 but in failure case
it is showing 32.
Ubuntu 22.04 base image (failure case)
===
(gdb) bt
#0 vtd_iova_to_slpte (pasid=4294967295, aw_bits=,
writes=, reads=,
slpte_level=, slptep=,
is_write=, iova=4721414144, ce=0x7ffd791680c0,
s=) at ../hw/i386/intel_iommu.c:1086
#1 vtd_do_iommu_translate (entry=0x7ffd79168090, is_write=, addr=4721414144, devfn=, bus=,
vtd_as=0x55873ec920f0) at ../hw/i386/intel_iommu.c:1942
#2 vtd_iommu_translate (iommu=0x55873ec92160, addr=4721414144,
flag=, iommu_idx=) at
../hw/i386/intel_iommu.c:3149
#3 0x558739bf24d1 in address_space_translate_iommu
(iommu_mr=0x55873ec92160, xlat=xlat@entry=0x7ffd791682c0,
plen_out=plen_out@entry=0x7ffd791682b8,
page_mask_out=page_mask_out@entry=0x0, is_write=is_write@entry=false,
is_mmio=true, target_as=0x7ffd791682c8, attrs=...) at
../softmmu/physmem.c:434
#4 0x558739bf28ba in flatview_do_translate
(addr=addr@entry=4721414144, xlat=xlat@entry=0x7ffd791682c0,
plen_out=plen_out@entry=0x7ffd791682b8,
page_mask_out=page_mask_out@entry=0x0, is_write=false, is_mmio=true,
target_as=0x7ffd791682c8, attrs=..., fv=, fv=) at ../softmmu/physmem.c:507
#5 0x558739bf7bb3 in flatview_translate (attrs=...,
is_write=false, plen=0x7ffd791682b8, xlat=0x7ffd791682c0,
addr=4721414144, fv=0x7f8c38444300) at ../softmmu/physmem.c:567
#6 flatview_read (fv=0x7f8c38444300, addr=addr@entry=4721414144,
attrs=attrs@entry=..., buf=buf@entry=0x7ffd79168410, len=len@entry=64)
at ../softmmu/physmem.c:2930
#7 0x558739bf7f46 in address_space_read_full (len=64,
buf=0x7ffd79168410, attrs=..., addr=4721414144, as=0x55873f323618) at
../softmmu/physmem.c:2947
#8 address_space_rw (as=as@entry=0x55873f323618,
addr=addr@entry=4721414144, attrs=attrs@entry=...,
buf=buf@entry=0x7ffd79168410, len=len@entry=64,
is_write=is_write@entry=false) at ../softmmu/physmem.c:2975
#9 0x5587399af9cc in dma_memory_rw_relaxed (attrs=...,
dir=DMA_DIRECTION_TO_DEVICE, len=64, buf=0x7ffd79168410,
addr=4721414144, as=0x55873f323618) at
/home/arka/qemu/include/sysemu/dma.h:87
#10 0x5587399bb1a5 in nvme_process_sq (opaque=) at
../hw/nvme/ctrl.c:6256
#11 0x558739e03a34 in aio_bh_call (bh=0x7f8c380a8540) at ../util/async.c:150
#12 aio_bh_poll (ctx=ctx@entry=0x55873c3859f0) at ../util/async.c:178
#13 0x558739df0542 in aio_dispatch (ctx=0x55873c3859f0) at
../util/aio-posix.c:421
#14 0x558739e03672 in aio_ctx_dispatch (source=,
callback=, user_data=) at
../util/async.c:320
#15 0x7f9487650d3b in g_main_context_dispatch () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x558739e05cb0 in glib_pollfds_poll () at ../util/main-loop.c:297
#17 os_host_main_loop_wait (timeout=30596200) at ../util/main-loop.c:320
#18 main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:606
#19 0x558739a6c873 in qemu_main_loop () at ../softmmu/runstate.c:739
#20 0x558739c8599b in qemu_default_main () at ../softmmu/main.c:37
#21 0x7f9486826d90 in __libc_start_call_main
(main=main@entry=0x5587398b6ac0 , argc=argc@entry=205,
argv=argv@entry=0x7ffd79168988) at
../sysdeps/nptl/libc_start_call_main.h:58
#22 0x7f9486826e40 in __libc_start_main_impl (main=0x5587398b6ac0
, argc=205, argv=0x7ffd79168988, init=,
fini=, rtld_fini=,
stack_end=0x7ffd79168978) at ../csu/libc-start.c:392
#23 0x5587398b8395 in _start ()
(gdb) f 11
#11 0x558739e03a34 in aio_bh_call (bh=0x7f8c380a8540) at ../util/async.c:150
150 bh->cb(bh->opaque);
(gdb) p bh->opaque
$5 = (void *) 0x55873f327d20
(gdb) f 10
#10 0x5587399bb1a5 in nvme_process_sq (opaque=) at
../hw/nvme/ctrl.c:6256
6256if (nvme_addr_read(n, addr, (void *), sizeof(cmd))) {
(gdb) p *(NvmeSQueue*)0x55873f327d20
$6 = {
ctrl = 0x55873f3233e0,
sqid = 0,
cqid = 0,
head = 0,
tail = 2,
size = 32,
dma_addr = 4721414144,
db_addr = 4735401984,
ei_addr = 4735397888,
bh = 0x7f8c380a8540,
notifier = {
rfd = 0,
wfd = 0,
initialized = false
},
ioeventfd_enabled = false,
io_req = 0x7f8c3848f560,
req_list = {
tqh_first = 0x7f8c3848f560,
tqh_circ = {
tql_next = 0x7f8c3848f560,
tql_prev = 0x7f8c38490f50
}
},
out_req_list = {
tqh_first = 0x0,
tqh_circ = {
tql_next = 0x0,
tql_prev = 0x55873f327d80
}
},
entry = {
tqe_next = 0x0,
tqe_circ = {
tql_next = 0x0,
tql_prev = 0x55873f327df0
}
}
}
(gdb) p/x *(NvmeSQueue*)0x55873f327d20
$7 = {
ctrl = 0x55873f3233e0,
Re: DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
On Mon, Feb 13, 2023 at 10:15:34PM +0530, Major Saheb wrote: > Hi All, Hi, Major, > > I am facing an issue with qemu 7.2 rc2 with nvme. I have a container Is there any known working qemu version? Or should I assume it's failing always? > running Ubuntu 22.04 base image and host is running Ubuntu 22.04 > (Linux node-1 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC > 2023 x86_64 x86_64 x86_64 GNU/Linux), and I am using vfio-pci to > communicate with nvme devices. I am observing DMAR fault as following > > [ 3761.96] DMAR: DRHD: handling fault status reg 2 > [ 3762.001609] DMAR: [DMA Read NO_PASID] Request device [0b:00.0] > fault addr 0x1187a9000 [fault reason 0x06] PTE Read access is not set > > I also enabled vtd_iommu_translate and vtd_dmar_fault traces which is > showing the following > > 2023-02-13T07:02:37.074397Z qemu-system-x86_64: vtd_iova_to_slpte: > detected slpte permission error (iova=0x1187a9000, level=0x3, > slpte=0x0, write=0, pasid=0x) I think slpte=0x0 means the device pgtable entry does not exist at all, rather than an explicit permission issue. Is the guest using generic Intel IOMMU driver? Could it possible that for some reason the pgtable update was not flushed to the guest pages when the driver sent the IOTLB invalidation (probably via QI interface)? I saw that you mentioned your driver is using 0x8 as iova base address, then why here the iova being mapped is 0x1187a9000? Anything special on the device driver being used? > 236498@1676271757.075075:vtd_dmar_fault sid 0xb00 fault 6 addr > 0x1187a9000 write 0 > 2023-02-13T07:02:37.075174Z qemu-system-x86_64: vtd_iommu_translate: > detected translation failure (dev=0b:00:00, iova=0x1187a9000) > > It seems 'vtd_iova_to_slpte()' it is returning 0 slpte resulting this > issue. In our driver code (which is running in a container) we are > using 0x8 as our IOVA base address. We have 10 nvme devices > that we are initializing, and we start initialization by sending > identify controller and get log page commands. Observation is > sometimes the first device is getting DMAR fault, sometimes first is > successfully completed identify and get log page, and second device is > getting DMAR fault. Also if I use Ubuntu 20.04 as base image for the > container, then this issue is not seens as seen in the following trace > output > > 278365@1676297375.587477:vtd_dmar_slpte sid 0xb00 slpte 0x1f2556003 > addr 0x8 write 0 > 278365@1676297375.587513:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff > 278365@1676297375.587527:vtd_dmar_slpte sid 0xb00 slpte 0x1f25fc003 > addr 0x80020a000 write 1 > 278365@1676297375.587532:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x80020a000 -> gpa 0x1f25fc000 mask 0xfff > 278365@1676297375.587566:vtd_dmar_slpte sid 0xb00 slpte 0x1f2560003 > addr 0x88000 write 1 > 278365@1676297375.587572:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff > 278365@1676297375.587814:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff > 278365@1676297375.587850:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff > 278365@1676297375.588455:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff > 278365@1676297375.588473:vtd_dmar_slpte sid 0xb00 slpte 0x1f25fe003 > addr 0x80020b000 write 1 > 278365@1676297375.588479:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x80020b000 -> gpa 0x1f25fe000 mask 0xfff > 278365@1676297375.588507:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff > 278365@1676297375.588737:vtd_dmar_translate pasid 0x: dev > 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff > > Following is the partial qemu command line that I am using > > -device > intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 > > -device pcie-root-port,id=pcie-root-port0,slot=1 -drive > file=/home/raghu/lfd/datadir/rs-bdc0/storage/rs-bdc0-0-0,format=qcow2,cache=unsafe,if=none,id=NVME0 > -device nvme,serial=rs-bdc0-0_0,id=NVME0,bus=pcie-root-port0 -device > nvme-ns,drive=NVME0,eui64=0,uuid=30303030-3030-3030-3030-303030303030 > > cat /proc/cmdline > BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic > root=UUID=102c974c-7a1c-49cf-9bdd-a8e4638cf5c4 ro console=ttyS0,115200 > intel_iommu=on iommu=pt > > I have also tried without 'iommu=pt' producing same result in Ubuntu > 22.04 base image in container, also the host OS version remains same > in both cases. Did you mean using iommu=pt on the host or guest, or both? IIUC iommu=pt on the host at least won't make a difference, because when it's assigned to QEMU it'll ignore the pt setting. Thanks, -- Peter Xu
DMAR fault with qemu 7.2 and Ubuntu 22.04 base image
Hi All, I am facing an issue with qemu 7.2 rc2 with nvme. I have a container running Ubuntu 22.04 base image and host is running Ubuntu 22.04 (Linux node-1 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux), and I am using vfio-pci to communicate with nvme devices. I am observing DMAR fault as following [ 3761.96] DMAR: DRHD: handling fault status reg 2 [ 3762.001609] DMAR: [DMA Read NO_PASID] Request device [0b:00.0] fault addr 0x1187a9000 [fault reason 0x06] PTE Read access is not set I also enabled vtd_iommu_translate and vtd_dmar_fault traces which is showing the following 2023-02-13T07:02:37.074397Z qemu-system-x86_64: vtd_iova_to_slpte: detected slpte permission error (iova=0x1187a9000, level=0x3, slpte=0x0, write=0, pasid=0x) 236498@1676271757.075075:vtd_dmar_fault sid 0xb00 fault 6 addr 0x1187a9000 write 0 2023-02-13T07:02:37.075174Z qemu-system-x86_64: vtd_iommu_translate: detected translation failure (dev=0b:00:00, iova=0x1187a9000) It seems 'vtd_iova_to_slpte()' it is returning 0 slpte resulting this issue. In our driver code (which is running in a container) we are using 0x8 as our IOVA base address. We have 10 nvme devices that we are initializing, and we start initialization by sending identify controller and get log page commands. Observation is sometimes the first device is getting DMAR fault, sometimes first is successfully completed identify and get log page, and second device is getting DMAR fault. Also if I use Ubuntu 20.04 as base image for the container, then this issue is not seens as seen in the following trace output 278365@1676297375.587477:vtd_dmar_slpte sid 0xb00 slpte 0x1f2556003 addr 0x8 write 0 278365@1676297375.587513:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff 278365@1676297375.587527:vtd_dmar_slpte sid 0xb00 slpte 0x1f25fc003 addr 0x80020a000 write 1 278365@1676297375.587532:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x80020a000 -> gpa 0x1f25fc000 mask 0xfff 278365@1676297375.587566:vtd_dmar_slpte sid 0xb00 slpte 0x1f2560003 addr 0x88000 write 1 278365@1676297375.587572:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff 278365@1676297375.587814:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff 278365@1676297375.587850:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff 278365@1676297375.588455:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff 278365@1676297375.588473:vtd_dmar_slpte sid 0xb00 slpte 0x1f25fe003 addr 0x80020b000 write 1 278365@1676297375.588479:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x80020b000 -> gpa 0x1f25fe000 mask 0xfff 278365@1676297375.588507:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x88000 -> gpa 0x1f256 mask 0xfff 278365@1676297375.588737:vtd_dmar_translate pasid 0x: dev 0b:00.00 iova 0x8 -> gpa 0x1f2556000 mask 0xfff Following is the partial qemu command line that I am using -device intel-iommu,intremap=on,caching-mode=on,eim=on,device-iotlb=on,aw-bits=48 -device pcie-root-port,id=pcie-root-port0,slot=1 -drive file=/home/raghu/lfd/datadir/rs-bdc0/storage/rs-bdc0-0-0,format=qcow2,cache=unsafe,if=none,id=NVME0 -device nvme,serial=rs-bdc0-0_0,id=NVME0,bus=pcie-root-port0 -device nvme-ns,drive=NVME0,eui64=0,uuid=30303030-3030-3030-3030-303030303030 cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic root=UUID=102c974c-7a1c-49cf-9bdd-a8e4638cf5c4 ro console=ttyS0,115200 intel_iommu=on iommu=pt I have also tried without 'iommu=pt' producing same result in Ubuntu 22.04 base image in container, also the host OS version remains same in both cases. -R
