Re: remove dma_virt_ops v2
On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: > Hi Jason, > > this series switches the RDMA core to opencode the special case of > devices bypassing the DMA mapping in the RDMA ULPs. The virt ops > have caused a bit of trouble due to the P2P code node working with > them due to the fact that we'd do two dma mapping iterations for a > single I/O, but also are a bit of layering violation and lead to > more code than necessary. > > Tested with nvme-rdma over rxe. > > Note that the rds changes are untested, as I could not find any > simple rds test setup. > > Changes since v2: > - simplify the INFINIBAND_VIRT_DMA dependencies > - add a ib_uses_virt_dma helper > - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices > - use ib_dma_max_seg_size in umem > - stop using dmapool in rds > > Changes since v1: > - disable software RDMA drivers for highmem configs > - update the PCI commit logs All applied to for-next, thanks everyone Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On 11/17/20 2:50 AM, Ka-Cheong Poon wrote: On 11/13/20 1:36 AM, santosh.shilim...@oracle.com wrote: + Ka-Cheong On 11/12/20 5:23 AM, Jason Gunthorpe wrote: On Thu, Nov 12, 2020 at 10:40:30AM +0100, Christoph Hellwig wrote: ping? On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: Hi Jason, this series switches the RDMA core to opencode the special case of devices bypassing the DMA mapping in the RDMA ULPs. The virt ops have caused a bit of trouble due to the P2P code node working with them due to the fact that we'd do two dma mapping iterations for a single I/O, but also are a bit of layering violation and lead to more code than necessary. Tested with nvme-rdma over rxe. Note that the rds changes are untested, as I could not find any simple rds test setup. Changes since v2: - simplify the INFINIBAND_VIRT_DMA dependencies - add a ib_uses_virt_dma helper - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices - use ib_dma_max_seg_size in umem - stop using dmapool in rds Changes since v1: - disable software RDMA drivers for highmem configs - update the PCI commit logs Santosh can you please check the RDA parts?? Hi Ka-Cheong, Can you please check Christoph change [1] which clean-up dma-pool API to use ib_dma_* and slab allocator ? This was added as part of your "net/rds: Use DMA memory pool allocation for rds_header" commit. I applied the patch and ran some basic testing. And it seems to work fine. Thanks Ka-Cheong. Jason, Feel free to add ack for the RDS part. Regards, Santosh ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On 11/13/20 1:36 AM, santosh.shilim...@oracle.com wrote: + Ka-Cheong On 11/12/20 5:23 AM, Jason Gunthorpe wrote: On Thu, Nov 12, 2020 at 10:40:30AM +0100, Christoph Hellwig wrote: ping? On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: Hi Jason, this series switches the RDMA core to opencode the special case of devices bypassing the DMA mapping in the RDMA ULPs. The virt ops have caused a bit of trouble due to the P2P code node working with them due to the fact that we'd do two dma mapping iterations for a single I/O, but also are a bit of layering violation and lead to more code than necessary. Tested with nvme-rdma over rxe. Note that the rds changes are untested, as I could not find any simple rds test setup. Changes since v2: - simplify the INFINIBAND_VIRT_DMA dependencies - add a ib_uses_virt_dma helper - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices - use ib_dma_max_seg_size in umem - stop using dmapool in rds Changes since v1: - disable software RDMA drivers for highmem configs - update the PCI commit logs Santosh can you please check the RDA parts?? Hi Ka-Cheong, Can you please check Christoph change [1] which clean-up dma-pool API to use ib_dma_* and slab allocator ? This was added as part of your "net/rds: Use DMA memory pool allocation for rds_header" commit. I applied the patch and ran some basic testing. And it seems to work fine. Thanks. -- K. Poon ka-cheong.p...@oracle.com ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
Fixes: 551199aca1c3 ("lib/dma-virt: Add dma_virt_ops")
Note that the drivers had open coded versions of this earlier. I think
this goes back to the addition of the qib driver which is now gone
or the addition of the hfi1 or rxe drivers for something that still
matters
Christoph,Jason
I built a branch using the following recipe:
https://patchwork.kernel.org/project/linux-rdma/patch/:
20201106181941.1878556-11-...@lst.de/ dma-mapping: remove dma_virt_ops
20201106181941.1878556-10-...@lst.de/ PCI/P2PDMA: Cleanup
__pci_p2pdma_map_sg a bit
20201106181941.1878556-9-...@lst.de/ PCI/P2PDMA: Remove the
DMA_VIRT_OPS hacks
20201106181941.1878556-8-...@lst.de/ RDMA/core: remove use of dma_virt_ops
20201106181941.1878556-7-...@lst.de/ RDMA/core: remove
ib_dma_{alloc,free}_coherent
20201106181941.1878556-6-...@lst.de/ rds: stop using dmapool
20201106181941.1878556-5-...@lst.de/ nvme-rdma: use ibdev_to_node
instead of dereferencing ->dma_device
20201106181941.1878556-4-...@lst.de/ RDMA: lift ibdev_to_node from rds
to common code
20201106181941.1878556-3-...@lst.de/ RDMA/umem: use ib_dma_max_seg_size
instead of dma_get_max_seg_size
rdma/for-rc dabbd6abcdbe which has RMDA/sw: don't allow drivers using
dma_virt_ops on highmem configs
All of our rdmavt/hfi1 tests passed.
So I can at least vouch for "RDMA/core: remove use of dma_virt_ops"
Mike
Tested-by: Mike Marciniszyn
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On Thu, Nov 12, 2020 at 01:39:06PM -0400, Jason Gunthorpe wrote:
> On Thu, Nov 12, 2020 at 06:09:56PM +0100, Christoph Hellwig wrote:
> > On Thu, Nov 12, 2020 at 12:59:35PM -0400, Jason Gunthorpe wrote:
> > > RMDA/sw: Don't allow drivers using dma_virt_ops on highmem configs
> >
> > I think this one actually is something needed in 5.10 and -stable.
>
> Done, I added a
>
> Fixes: 551199aca1c3 ("lib/dma-virt: Add dma_virt_ops")
Note that the drivers had open coded versions of this earlier. I think
this goes back to the addition of the qib driver which is now gone
or the addition of the hfi1 or rxe drivers for something that still
matters.
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On Thu, Nov 12, 2020 at 06:09:56PM +0100, Christoph Hellwig wrote:
> On Thu, Nov 12, 2020 at 12:59:35PM -0400, Jason Gunthorpe wrote:
> > RMDA/sw: Don't allow drivers using dma_virt_ops on highmem configs
>
> I think this one actually is something needed in 5.10 and -stable.
Done, I added a
Fixes: 551199aca1c3 ("lib/dma-virt: Add dma_virt_ops")
Jason
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
+ Ka-Cheong On 11/12/20 5:23 AM, Jason Gunthorpe wrote: On Thu, Nov 12, 2020 at 10:40:30AM +0100, Christoph Hellwig wrote: ping? On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: Hi Jason, this series switches the RDMA core to opencode the special case of devices bypassing the DMA mapping in the RDMA ULPs. The virt ops have caused a bit of trouble due to the P2P code node working with them due to the fact that we'd do two dma mapping iterations for a single I/O, but also are a bit of layering violation and lead to more code than necessary. Tested with nvme-rdma over rxe. Note that the rds changes are untested, as I could not find any simple rds test setup. Changes since v2: - simplify the INFINIBAND_VIRT_DMA dependencies - add a ib_uses_virt_dma helper - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices - use ib_dma_max_seg_size in umem - stop using dmapool in rds Changes since v1: - disable software RDMA drivers for highmem configs - update the PCI commit logs Santosh can you please check the RDA parts?? Hi Ka-Cheong, Can you please check Christoph change [1] which clean-up dma-pool API to use ib_dma_* and slab allocator ? This was added as part of your "net/rds: Use DMA memory pool allocation for rds_header" commit. Regards, Santosh [1] https://www.spinics.net/lists/linux-pci/msg101547.html ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On Thu, Nov 12, 2020 at 12:59:35PM -0400, Jason Gunthorpe wrote: > RMDA/sw: Don't allow drivers using dma_virt_ops on highmem configs I think this one actually is something needed in 5.10 and -stable. ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote:
> Hi Jason,
>
> this series switches the RDMA core to opencode the special case of
> devices bypassing the DMA mapping in the RDMA ULPs. The virt ops
> have caused a bit of trouble due to the P2P code node working with
> them due to the fact that we'd do two dma mapping iterations for a
> single I/O, but also are a bit of layering violation and lead to
> more code than necessary.
>
> Tested with nvme-rdma over rxe.
>
> Note that the rds changes are untested, as I could not find any
> simple rds test setup.
>
> Changes since v2:
> - simplify the INFINIBAND_VIRT_DMA dependencies
> - add a ib_uses_virt_dma helper
> - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices
> - use ib_dma_max_seg_size in umem
> - stop using dmapool in rds
>
> Changes since v1:
> - disable software RDMA drivers for highmem configs
> - update the PCI commit logs
Lets give Santosh a little longer for RDS, I've grabbed the precursor
parts to for-next for now:
nvme-rdma: Use ibdev_to_node instead of dereferencing ->dma_device
RDMA: Lift ibdev_to_node from rds to common code
RDMA/core: Remove ib_dma_{alloc,free}_coherent
RDMA/umem: Use ib_dma_max_seg_size instead of dma_get_max_seg_size
RMDA/sw: Don't allow drivers using dma_virt_ops on highmem configs
Will get the rest next week regardless.
Thanks,
Jason
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
On Thu, Nov 12, 2020 at 10:40:30AM +0100, Christoph Hellwig wrote: > ping? > > On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: > > Hi Jason, > > > > this series switches the RDMA core to opencode the special case of > > devices bypassing the DMA mapping in the RDMA ULPs. The virt ops > > have caused a bit of trouble due to the P2P code node working with > > them due to the fact that we'd do two dma mapping iterations for a > > single I/O, but also are a bit of layering violation and lead to > > more code than necessary. > > > > Tested with nvme-rdma over rxe. > > > > Note that the rds changes are untested, as I could not find any > > simple rds test setup. > > > > Changes since v2: > > - simplify the INFINIBAND_VIRT_DMA dependencies > > - add a ib_uses_virt_dma helper > > - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices > > - use ib_dma_max_seg_size in umem > > - stop using dmapool in rds > > > > Changes since v1: > > - disable software RDMA drivers for highmem configs > > - update the PCI commit logs Santosh can you please check the RDA parts?? Jason ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: remove dma_virt_ops v2
ping? On Fri, Nov 06, 2020 at 07:19:31PM +0100, Christoph Hellwig wrote: > Hi Jason, > > this series switches the RDMA core to opencode the special case of > devices bypassing the DMA mapping in the RDMA ULPs. The virt ops > have caused a bit of trouble due to the P2P code node working with > them due to the fact that we'd do two dma mapping iterations for a > single I/O, but also are a bit of layering violation and lead to > more code than necessary. > > Tested with nvme-rdma over rxe. > > Note that the rds changes are untested, as I could not find any > simple rds test setup. > > Changes since v2: > - simplify the INFINIBAND_VIRT_DMA dependencies > - add a ib_uses_virt_dma helper > - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices > - use ib_dma_max_seg_size in umem > - stop using dmapool in rds > > Changes since v1: > - disable software RDMA drivers for highmem configs > - update the PCI commit logs ---end quoted text--- ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
remove dma_virt_ops v2
Hi Jason, this series switches the RDMA core to opencode the special case of devices bypassing the DMA mapping in the RDMA ULPs. The virt ops have caused a bit of trouble due to the P2P code node working with them due to the fact that we'd do two dma mapping iterations for a single I/O, but also are a bit of layering violation and lead to more code than necessary. Tested with nvme-rdma over rxe. Note that the rds changes are untested, as I could not find any simple rds test setup. Changes since v2: - simplify the INFINIBAND_VIRT_DMA dependencies - add a ib_uses_virt_dma helper - use ib_uses_virt_dma in nvmet-rdma to disable p2p for virt_dma devices - use ib_dma_max_seg_size in umem - stop using dmapool in rds Changes since v1: - disable software RDMA drivers for highmem configs - update the PCI commit logs ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
remove dma_virt_ops v2
Hi Jason, this series switches the RDMA core to opencode the special case of devices bypassing the DMA mapping in the RDMA ULPs. The virt ops have caused a bit of trouble due to the P2P code node working with them due to the fact that we'd do two dma mapping iterations for a single I/O, but also are a bit of layering violation and lead to more code than necessary. Tested with nvme-rdma over rxe. Changes since v1: - disable software RDMA drivers for highmem configs - update the PCI commit logs ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
