Re: Error when running fio against nvme-of rdma target (mlx5 driver)
Hi, Can you please send the original scenario, setup details and dumps ? I can't find it in my mailbox. you can send it directly to me to avoid spam. -Max. On 5/17/2022 11:26 AM, Mark Ruijter wrote: Hi Robin, I ran into the exact same problem while testing with 4 connect-x6 cards, kernel 5.18-rc6. [ 4878.273016] nvme nvme0: Successfully reconnected (3 attempts) [ 4879.122015] nvme nvme0: starting error recovery [ 4879.122028] infiniband mlx5_4: mlx5_handle_error_cqe:332:(pid 0): WC error: 4, Message: local protection error [ 4879.122035] infiniband mlx5_4: dump_cqe:272:(pid 0): dump error cqe [ 4879.122037] : 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 4879.122039] 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 4879.122040] 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 4879.122040] 0030: 00 00 00 00 a9 00 56 04 00 00 00 ed 0d da ff e2 [ 4881.085547] nvme nvme3: Reconnecting in 10 seconds... I assume this means that the problem has still not been resolved? If so, I'll try to diagnose the problem. Thanks, --Mark On 11/02/2022, 12:35, "Linux-nvme on behalf of Robin Murphy" wrote: On 2022-02-10 23:58, Martin Oliveira wrote: > On 2/9/22 1:41 AM, Chaitanya Kulkarni wrote: >> On 2/8/22 6:50 PM, Martin Oliveira wrote: >>> Hello, >>> >>> We have been hitting an error when running IO over our nvme-of setup, using the mlx5 driver and we are wondering if anyone has seen anything similar/has any suggestions. >>> >>> Both initiator and target are AMD EPYC 7502 machines connected over RDMA using a Mellanox MT28908. Target has 12 NVMe SSDs which are exposed as a single NVMe fabrics device, one physical SSD per namespace. >>> >> >> Thanks for reporting this, if you can bisect the problem on your setup >> it will help others to help you better. >> >> -ck > > Hi Chaitanya, > > I went back to a kernel as old as 4.15 and the problem was still there, so I don't know of a good commit to start from. > > I also learned that I can reproduce this with as little as 3 cards and I updated the firmware on the Mellanox cards to the latest version. > > I'd be happy to try any tests if someone has any suggestions. The IOMMU is probably your friend here - one thing that might be worth trying is capturing the iommu:map and iommu:unmap tracepoints to see if the address reported in subsequent IOMMU faults was previously mapped as a valid DMA address (be warned that there will likely be a *lot* of trace generated). With 5.13 or newer, booting with "iommu.forcedac=1" should also make it easier to tell real DMA IOVAs from rogue physical addresses or other nonsense, as real DMA addresses should then look more like 0x24d08000. That could at least help narrow down whether it's some kind of use-after-free race or a completely bogus address creeping in somehow. Robin. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v3 13/20] PCI/P2PDMA: remove pci_p2pdma_[un]map_sg()
On 9/17/2021 2:40 AM, Logan Gunthorpe wrote: This interface is superseded by support in dma_map_sg() which now supports heterogeneous scatterlists. There are no longer any users, so remove it. Signed-off-by: Logan Gunthorpe --- drivers/pci/p2pdma.c | 65 -- include/linux/pci-p2pdma.h | 27 2 files changed, 92 deletions(-) Looks good, Reviewed-by: Max Gurtovoy ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v3 11/20] RDMA/core: introduce ib_dma_pci_p2p_dma_supported()
Logan, On 9/17/2021 2:40 AM, Logan Gunthorpe wrote: Introduce the helper function ib_dma_pci_p2p_dma_supported() to check if a given ib_device can be used in P2PDMA transfers. This ensures the ib_device is not using virt_dma and also that the underlying dma_device supports P2PDMA. Use the new helper in nvme-rdma to replace the existing check for ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows switching away from pci_p2pdma_[un]map_sg(). Signed-off-by: Logan Gunthorpe --- drivers/nvme/target/rdma.c | 2 +- include/rdma/ib_verbs.h| 11 +++ 2 files changed, 12 insertions(+), 1 deletion(-) Looks good, Reviewed-by: Max Gurtovoy ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v3 12/20] RDMA/rw: use dma_map_sgtable()
On 9/28/2021 10:43 PM, Jason Gunthorpe wrote: On Thu, Sep 16, 2021 at 05:40:52PM -0600, Logan Gunthorpe wrote: dma_map_sg() now supports the use of P2PDMA pages so pci_p2pdma_map_sg() is no longer necessary and may be dropped. Switch to the dma_map_sgtable() interface which will allow for better error reporting if the P2PDMA pages are unsupported. The change to sgtable also appears to fix a couple subtle error path bugs: - In rdma_rw_ctx_init(), dma_unmap would be called with an sg that could have been incremented from the original call, as well as an nents that was not the original number of nents called when mapped. - Similarly in rdma_rw_ctx_signature_init, both sg and prot_sg were unmapped with the incorrect number of nents. Those bugs should definately get fixed.. I might extract the sgtable conversion into a stand alone patch to do it. Yes, we need these fixes before this series will converge. Looks good, Reviewed-by: Max Gurtovoy But as it is, this looks fine Reviewed-by: Jason Gunthorpe Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH v3 10/20] nvme-pci: convert to using dma_map_sgtable()
Logan, On 9/17/2021 2:40 AM, Logan Gunthorpe wrote: The dma_map operations now support P2PDMA pages directly. So remove the calls to pci_p2pdma_[un]map_sg_attrs() and replace them with calls to dma_map_sgtable(). dma_map_sgtable() returns more complete error codes than dma_map_sg() and allows differentiating EREMOTEIO errors in case an unsupported P2PDMA transfer is requested. When this happens, return BLK_STS_TARGET so the request isn't retried. Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/pci.c | 69 + 1 file changed, 29 insertions(+), 40 deletions(-) Looks good, Reviewed-by: Max Gurtovoy ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
