Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
On 5/16/25 1:02 AM, Ankit Soni wrote: On Wed, May 14, 2025 at 04:08:09PM -0400, Alejandro Jimenez wrote: On 5/14/25 11:54 AM, Jason Gunthorpe wrote: On Wed, May 14, 2025 at 09:23:49AM +, Ankit Soni wrote: I am experiencing a system hang with a 5-level v2 page table mode, on boot. The NVMe boot drive is not initializing. Below are the relevant dmesg logs with some prints i had added: [6.386439] AMD-Vi v2 domain init [6.390132] AMD-Vi v2 pt init [6.390133] AMD-Vi aperture end last va ff ... [ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr 19351b000 ... [ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable controller [ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable controller [ 72.197176] nvme nvme0: Identify Controller failed (-4) [ 72.203063] nvme nvme1: Identify Controller failed (-4) [ 72.209237] nvme :05:00.0: probe with driver nvme failed with error -5 [ 72.209336] nvme :44:00.0: probe with driver nvme failed with error -5 ... Timed out waiting for the udev queue to be empty. According to the dmesg logs above, the IOVA for the v2 page table appears incorrect and is not aligned with domain->geometry.aperture_end. Which requires domain->geometry.force_aperture = true; to be added at the appropriate location. Proabably here! Thank you for pointing out this issue and its cause. I originally tested on a host with SCSI storage, and after your report I tried but couldn't reproduce the hang on a Zen4 host with an nvme boot drive. I wanted to see if it was a pattern common to NVME, but I suppose it depends on the DMA mask chosen by the specific driver. Alejandro Hi, Can you try with below command line? "amd_iommu=pgtbl_v2 iommu.forcedac=1" Yes, I can reproduce the hang when booting with the combination of: "amd_iommu=pgtbl_v2 iommu.passthrough=0 iommu.forcedac=1" [ 72.763105] nvme nvme0: I/O tag 8 (0008) QID 0 timeout, disable controller [ 72.772093] nvme nvme0: Device not ready; aborting shutdown, CSTS=0x1 [ 72.796372] nvme nvme0: Identify Controller failed (-4) [ 72.802603] nvme :01:00.0: probe with driver nvme failed with error -5 It also triggers failures for the Mellanox driver: [ 134.342120] mlx5_core :61:00.0: wait_func:1185:(pid 3235): ENABLE_HCA(0x104) timeout. Will cause a leak of a command resource [ 134.355465] mlx5_core :61:00.0: mlx5_function_enable:1215:(pid 3235): enable hca failed [ 134.366570] mlx5_core :61:00.0: probe_one:2003:(pid 3235): mlx5_init_one failed with error code -110 [ 134.386593] mlx5_core :61:00.0: probe with driver mlx5_core failed with error -110 Setting force_aperture = true in pt_iommu_init_domain() solves the issue for the AMD v2 format where dynamic top is not available. Thank you, Alejandro Indeed it depends on DMA Mask chose by nvme driver. if force_aperture is not true, iommu driver will use dma_mask in place of end_aperture. -Ankit Yes! It got lost, thanks alot! Jason
Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
On Wed, May 14, 2025 at 04:08:09PM -0400, Alejandro Jimenez wrote: > > > On 5/14/25 11:54 AM, Jason Gunthorpe wrote: > > On Wed, May 14, 2025 at 09:23:49AM +, Ankit Soni wrote: > > > I am experiencing a system hang with a 5-level v2 page table mode, on > > > boot. > > > The NVMe boot drive is not initializing. > > > Below are the relevant dmesg logs with some prints i had added: > > > > > > [6.386439] AMD-Vi v2 domain init > > > [6.390132] AMD-Vi v2 pt init > > > [6.390133] AMD-Vi aperture end last va ff > > > ... > > > [ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr > > > 19351b000 > > > ... > > > [ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable > > > controller > > > [ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable > > > controller > > > [ 72.197176] nvme nvme0: Identify Controller failed (-4) > > > [ 72.203063] nvme nvme1: Identify Controller failed (-4) > > > [ 72.209237] nvme :05:00.0: probe with driver nvme failed with > > > error -5 > > > [ 72.209336] nvme :44:00.0: probe with driver nvme failed with > > > error -5 > > > ... > > > Timed out waiting for the udev queue to be empty. > > > > > > According to the dmesg logs above, the IOVA for the v2 page table appears > > > incorrect and is not aligned with domain->geometry.aperture_end. Which > > > requires domain->geometry.force_aperture = true; to be added at the > > > appropriate location. Proabably here! > > Thank you for pointing out this issue and its cause. I originally tested on > a host with SCSI storage, and after your report I tried but couldn't > reproduce the hang on a Zen4 host with an nvme boot drive. I wanted to see > if it was a pattern common to NVME, but I suppose it depends on the DMA mask > chosen by the specific driver. > > Alejandro > Hi, Can you try with below command line? "amd_iommu=pgtbl_v2 iommu.forcedac=1" Indeed it depends on DMA Mask chose by nvme driver. if force_aperture is not true, iommu driver will use dma_mask in place of end_aperture. -Ankit > > > > Yes! It got lost, thanks alot! > > > > Jason >
Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
On Wed, May 14, 2025 at 04:08:09PM -0400, Alejandro Jimenez wrote: > > > On 5/14/25 11:54 AM, Jason Gunthorpe wrote: > > On Wed, May 14, 2025 at 09:23:49AM +, Ankit Soni wrote: > > > I am experiencing a system hang with a 5-level v2 page table mode, on > > > boot. > > > The NVMe boot drive is not initializing. > > > Below are the relevant dmesg logs with some prints i had added: > > > > > > [6.386439] AMD-Vi v2 domain init > > > [6.390132] AMD-Vi v2 pt init > > > [6.390133] AMD-Vi aperture end last va ff > > > ... > > > [ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr > > > 19351b000 > > > ... > > > [ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable > > > controller > > > [ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable > > > controller > > > [ 72.197176] nvme nvme0: Identify Controller failed (-4) > > > [ 72.203063] nvme nvme1: Identify Controller failed (-4) > > > [ 72.209237] nvme :05:00.0: probe with driver nvme failed with > > > error -5 > > > [ 72.209336] nvme :44:00.0: probe with driver nvme failed with > > > error -5 > > > ... > > > Timed out waiting for the udev queue to be empty. > > > > > > According to the dmesg logs above, the IOVA for the v2 page table appears > > > incorrect and is not aligned with domain->geometry.aperture_end. Which > > > requires domain->geometry.force_aperture = true; to be added at the > > > appropriate location. Proabably here! > > Thank you for pointing out this issue and its cause. I originally tested on > a host with SCSI storage, and after your report I tried but couldn't > reproduce the hang on a Zen4 host with an nvme boot drive. I wanted to see > if it was a pattern common to NVME, but I suppose it depends on the DMA mask > chosen by the specific driver. Yeah, that's a good point I've also been thinking that the dma-iommu.c needs some updating here as allocating top-down like the above shows completely defeats the dynamic top optimization feature AMD has. iova ___e000 will immediately expand to a 6 level table. I think dynamic top can be made to work with vt-d and riscv with some effort. Jason
Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
On 5/14/25 11:54 AM, Jason Gunthorpe wrote: On Wed, May 14, 2025 at 09:23:49AM +, Ankit Soni wrote: I am experiencing a system hang with a 5-level v2 page table mode, on boot. The NVMe boot drive is not initializing. Below are the relevant dmesg logs with some prints i had added: [6.386439] AMD-Vi v2 domain init [6.390132] AMD-Vi v2 pt init [6.390133] AMD-Vi aperture end last va ff ... [ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr 19351b000 ... [ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable controller [ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable controller [ 72.197176] nvme nvme0: Identify Controller failed (-4) [ 72.203063] nvme nvme1: Identify Controller failed (-4) [ 72.209237] nvme :05:00.0: probe with driver nvme failed with error -5 [ 72.209336] nvme :44:00.0: probe with driver nvme failed with error -5 ... Timed out waiting for the udev queue to be empty. According to the dmesg logs above, the IOVA for the v2 page table appears incorrect and is not aligned with domain->geometry.aperture_end. Which requires domain->geometry.force_aperture = true; to be added at the appropriate location. Proabably here! Thank you for pointing out this issue and its cause. I originally tested on a host with SCSI storage, and after your report I tried but couldn't reproduce the hang on a Zen4 host with an nvme boot drive. I wanted to see if it was a pattern common to NVME, but I suppose it depends on the DMA mask chosen by the specific driver. Alejandro Yes! It got lost, thanks alot! Jason
Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
On Wed, May 14, 2025 at 09:23:49AM +, Ankit Soni wrote: > I am experiencing a system hang with a 5-level v2 page table mode, on boot. > The NVMe boot drive is not initializing. > Below are the relevant dmesg logs with some prints i had added: > > [6.386439] AMD-Vi v2 domain init > [6.390132] AMD-Vi v2 pt init > [6.390133] AMD-Vi aperture end last va ff > ... > [ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr 19351b000 > ... > [ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable controller > [ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable controller > [ 72.197176] nvme nvme0: Identify Controller failed (-4) > [ 72.203063] nvme nvme1: Identify Controller failed (-4) > [ 72.209237] nvme :05:00.0: probe with driver nvme failed with error -5 > [ 72.209336] nvme :44:00.0: probe with driver nvme failed with error -5 > ... > Timed out waiting for the udev queue to be empty. > > According to the dmesg logs above, the IOVA for the v2 page table appears > incorrect and is not aligned with domain->geometry.aperture_end. Which > requires domain->geometry.force_aperture = true; to be added at the > appropriate location. Proabably here! Yes! It got lost, thanks alot! Jason
Re: [PATCH v2 03/15] iommupt: Add the basic structure of the iommu implementation
Hi Jason,
On Mon, May 05, 2025 at 11:18:33AM -0300, Jason Gunthorpe wrote:
> +static int pt_iommu_init_domain(struct pt_iommu *iommu_table,
> + struct iommu_domain *domain)
> +{
> + struct pt_common *common = common_from_iommu(iommu_table);
> + struct pt_iommu_info info;
> + struct pt_range range;
> +
> + NS(get_info)(iommu_table, &info);
> +
> + domain->type = __IOMMU_DOMAIN_PAGING;
> + domain->pgsize_bitmap = info.pgsize_bitmap;
> +
> + if (pt_feature(common, PT_FEAT_DYNAMIC_TOP))
> + range = _pt_top_range(common,
> + _pt_top_set(NULL, PT_MAX_TOP_LEVEL));
> + else
> + range = pt_top_range(common);
> +
> + /*
> + * A 64 bit high address space table on a 32 bit system cannot work.
> + */
> + domain->geometry.aperture_start = (unsigned long)range.va;
> + if ((pt_vaddr_t)domain->geometry.aperture_start != range.va ||
> + range.va > ULONG_MAX)
> + return -EOVERFLOW;
> +
> + /*
> + * The aperture is limited to what the API can do after considering all
> + * the different types dma_addr_t/unsigned long/pt_vaddr_t that are used
> + * to store a VA. Set the aperture to something that is valid for all
> + * cases. Saturate instead of truncate the end if the types are smaller
> + * than the top range. aperture_end is a last.
> + */
> + domain->geometry.aperture_end = (unsigned long)range.last_va;
I am experiencing a system hang with a 5-level v2 page table mode, on boot.
The NVMe boot drive is not initializing.
Below are the relevant dmesg logs with some prints i had added:
[6.386439] AMD-Vi v2 domain init
[6.390132] AMD-Vi v2 pt init
[6.390133] AMD-Vi aperture end last va ff
...
[ 10.315372] AMD-Vi gen pt MAP PAGES iova e000 paddr 19351b000
...
[ 72.171930] nvme nvme0: I/O tag 0 () QID 0 timeout, disable controller
[ 72.179618] nvme nvme1: I/O tag 24 (0018) QID 0 timeout, disable controller
[ 72.197176] nvme nvme0: Identify Controller failed (-4)
[ 72.203063] nvme nvme1: Identify Controller failed (-4)
[ 72.209237] nvme :05:00.0: probe with driver nvme failed with error -5
[ 72.209336] nvme :44:00.0: probe with driver nvme failed with error -5
...
Timed out waiting for the udev queue to be empty.
According to the dmesg logs above, the IOVA for the v2 page table appears
incorrect and is not aligned with domain->geometry.aperture_end. Which
requires domain->geometry.force_aperture = true; to be added at the
appropriate location. Proabably here!
- Ankit
> + if ((pt_vaddr_t)domain->geometry.aperture_end != range.last_va) {
> + domain->geometry.aperture_end = ULONG_MAX;
> + domain->pgsize_bitmap &= ULONG_MAX;
> + }
> +
> + return 0;
> +}
> +
