Re: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
Kevin, Nicolin, On 5/16/2025 8:29 AM, Tian, Kevin wrote: >> From: Nicolin Chen >> Sent: Friday, May 16, 2025 10:30 AM >> >> On Thu, May 15, 2025 at 05:58:41AM +, Tian, Kevin wrote: From: Nicolin Chen Sent: Friday, May 9, 2025 11:03 AM Add IOMMUFD_OBJ_HW_QUEUE with an iommufd_hw_queue structure, representing a HW-accelerated queue type of IOMMU's physical queue that can be >> passed through to a user space VM for direct hardware control, such as: - NVIDIA's Virtual Command Queue - AMD vIOMMU's Command Buffer, Event Log Buffer, and PPR Log Buffer Introduce an allocator iommufd_hw_queue_alloc(). And add a pair of viommu ops for iommufd to forward user space ioctls to IOMMU drivers. Given that the first user of this HW QUEUE (tegra241-cmdqv) will need to ensure the queue memory to be physically contiguous, add a flag >> property in iommufd_viommu_ops and IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA to allow driver to flag it so that the core will validate the physical pages of a given guest queue. >>> >>> 'READS' is confusing here. What about xxx_CONTIG_PAS? >> >> Combining Jason's first comments here: >> https://lore.kernel.org/linux- >> iommu/[email protected]/ >> >> So, pinning should be optional too. And I think there would be >> unlikely a case where HW needs contiguous physical pages while >> not requiring to pin the pages, right? AMD IOMMU needs contiguous GPA space for buffer (like command buffer), not contiguous physical address. >> >> So, we need an flag that could indicate to do both tests. Yet, >> "xxx_CONTIG_PAS" doesn't sound very fitting, compared to this >> "IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA". >> >> Perhaps, we should just add some comments to clarify a bit. Or >> do you have some better naming? >> > > let's wait until that open is closed, i.e. whether we still let the core > manage it and whether AMD requires pinning even when IOVA > is used. I think we may still want to pin those buffer address. -Vasant
RE: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
> From: Nicolin Chen > Sent: Friday, May 16, 2025 10:30 AM > > On Thu, May 15, 2025 at 05:58:41AM +, Tian, Kevin wrote: > > > From: Nicolin Chen > > > Sent: Friday, May 9, 2025 11:03 AM > > > > > > Add IOMMUFD_OBJ_HW_QUEUE with an iommufd_hw_queue structure, > > > representing > > > a HW-accelerated queue type of IOMMU's physical queue that can be > passed > > > through to a user space VM for direct hardware control, such as: > > > - NVIDIA's Virtual Command Queue > > > - AMD vIOMMU's Command Buffer, Event Log Buffer, and PPR Log Buffer > > > > > > Introduce an allocator iommufd_hw_queue_alloc(). And add a pair of > > > viommu > > > ops for iommufd to forward user space ioctls to IOMMU drivers. > > > > > > Given that the first user of this HW QUEUE (tegra241-cmdqv) will need to > > > ensure the queue memory to be physically contiguous, add a flag > property > > > in iommufd_viommu_ops and > > > IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA to allow > > > driver to flag it so that the core will validate the physical pages of a > > > given guest queue. > > > > 'READS' is confusing here. What about xxx_CONTIG_PAS? > > Combining Jason's first comments here: > https://lore.kernel.org/linux- > iommu/[email protected]/ > > So, pinning should be optional too. And I think there would be > unlikely a case where HW needs contiguous physical pages while > not requiring to pin the pages, right? > > So, we need an flag that could indicate to do both tests. Yet, > "xxx_CONTIG_PAS" doesn't sound very fitting, compared to this > "IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA". > > Perhaps, we should just add some comments to clarify a bit. Or > do you have some better naming? > let's wait until that open is closed, i.e. whether we still let the core manage it and whether AMD requires pinning even when IOVA is used.
Re: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
On Thu, May 15, 2025 at 05:58:41AM +, Tian, Kevin wrote: > > From: Nicolin Chen > > Sent: Friday, May 9, 2025 11:03 AM > > > > Add IOMMUFD_OBJ_HW_QUEUE with an iommufd_hw_queue structure, > > representing > > a HW-accelerated queue type of IOMMU's physical queue that can be passed > > through to a user space VM for direct hardware control, such as: > > - NVIDIA's Virtual Command Queue > > - AMD vIOMMU's Command Buffer, Event Log Buffer, and PPR Log Buffer > > > > Introduce an allocator iommufd_hw_queue_alloc(). And add a pair of > > viommu > > ops for iommufd to forward user space ioctls to IOMMU drivers. > > > > Given that the first user of this HW QUEUE (tegra241-cmdqv) will need to > > ensure the queue memory to be physically contiguous, add a flag property > > in iommufd_viommu_ops and > > IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA to allow > > driver to flag it so that the core will validate the physical pages of a > > given guest queue. > > 'READS' is confusing here. What about xxx_CONTIG_PAS? Combining Jason's first comments here: https://lore.kernel.org/linux-iommu/[email protected]/ So, pinning should be optional too. And I think there would be unlikely a case where HW needs contiguous physical pages while not requiring to pin the pages, right? So, we need an flag that could indicate to do both tests. Yet, "xxx_CONTIG_PAS" doesn't sound very fitting, compared to this "IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA". Perhaps, we should just add some comments to clarify a bit. Or do you have some better naming? Thanks Nicolin
Re: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
On Thu, May 15, 2025 at 12:39:03PM -0300, Jason Gunthorpe wrote:
> On Thu, May 08, 2025 at 08:02:31PM -0700, Nicolin Chen wrote:
> > +#define iommufd_hw_queue_alloc(viommu, drv_struct, member)
> > \
> > + ({ \
> > + drv_struct *ret; \
> > + \
> > + static_assert(__same_type(struct iommufd_viommu, *viommu));\
> > + static_assert(__same_type(struct iommufd_hw_queue, \
> > + ((drv_struct *)NULL)->member)); \
> > + static_assert(offsetof(drv_struct, member.obj) == 0); \
> > + ret = (drv_struct *)_iommufd_object_alloc( \
> > + viommu->ictx, sizeof(drv_struct), \
> > + IOMMUFD_OBJ_HW_QUEUE); \
> > + if (!IS_ERR(ret)) {\
> > + ret->member.viommu = viommu; \
> > + ret->member.ictx = viommu->ictx; \
> > + } \
> > + ret; \
> > + })
>
> This should just call
>
> __iommufd_object_alloc(viommu->ictx, ret, IOMMUFD_OBJ_HW_QUEUE member.obj)
>
> And all the casting and asserts should be in that macro, move it to
> this header
Ack.
> > /* Helper for IOMMU driver to destroy structures created by allocators
> > above */
> > #define iommufd_struct_destroy(drv_struct, member)
> > \
> > ({ \
>
> This is abort not destroy, the names should be consistent. But looking
> more at the series I think it will be better to do the little rework I
> suggested and not give this function to the driver in the first place.
Yea, it will be replaced in v5.
Thanks
Nicolin
Re: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
On Thu, May 15, 2025 at 05:58:41AM +, Tian, Kevin wrote: > > From: Nicolin Chen > > Sent: Friday, May 9, 2025 11:03 AM > > > > Add IOMMUFD_OBJ_HW_QUEUE with an iommufd_hw_queue structure, > > representing > > a HW-accelerated queue type of IOMMU's physical queue that can be passed > > through to a user space VM for direct hardware control, such as: > > - NVIDIA's Virtual Command Queue > > - AMD vIOMMU's Command Buffer, Event Log Buffer, and PPR Log Buffer > > > > Introduce an allocator iommufd_hw_queue_alloc(). And add a pair of > > viommu > > ops for iommufd to forward user space ioctls to IOMMU drivers. > > > > Given that the first user of this HW QUEUE (tegra241-cmdqv) will need to > > ensure the queue memory to be physically contiguous, add a flag property > > in iommufd_viommu_ops and > > IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA to allow > > driver to flag it so that the core will validate the physical pages of a > > given guest queue. > > 'READS' is confusing here. What about xxx_CONTIG_PAS? Sure. > > + * @hw_queue_alloc: Allocate a HW QUEUE object for a HW-accelerated > > queue given > > + * the @type (must be defined in > > include/uapi/linux/iommufd.h) > > + * for the @viommu. @index carries the logical HW QUEUE > > ID per > > + * @viommu in a guest VM, for a multi-queue case; @addr > > carries > > + * the guest physical base address of the queue memory; > > s/@addr/@base_addr/ > > Reviewed-by: Kevin Tian Thanks! Nicolin
Re: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
On Thu, May 08, 2025 at 08:02:31PM -0700, Nicolin Chen wrote:
> +#define iommufd_hw_queue_alloc(viommu, drv_struct, member)
> \
> + ({ \
> + drv_struct *ret; \
> +\
> + static_assert(__same_type(struct iommufd_viommu, *viommu));\
> + static_assert(__same_type(struct iommufd_hw_queue, \
> + ((drv_struct *)NULL)->member)); \
> + static_assert(offsetof(drv_struct, member.obj) == 0); \
> + ret = (drv_struct *)_iommufd_object_alloc( \
> + viommu->ictx, sizeof(drv_struct), \
> + IOMMUFD_OBJ_HW_QUEUE); \
> + if (!IS_ERR(ret)) {\
> + ret->member.viommu = viommu; \
> + ret->member.ictx = viommu->ictx; \
> + } \
> + ret; \
> + })
This should just call
__iommufd_object_alloc(viommu->ictx, ret, IOMMUFD_OBJ_HW_QUEUE member.obj)
And all the casting and asserts should be in that macro, move it to
this header
> /* Helper for IOMMU driver to destroy structures created by allocators above
> */
> #define iommufd_struct_destroy(drv_struct, member)
> \
> ({ \
This is abort not destroy, the names should be consistent. But looking
more at the series I think it will be better to do the little rework I
suggested and not give this function to the driver in the first place.
Jason
RE: [PATCH v4 10/23] iommufd/viommu: Introduce IOMMUFD_OBJ_HW_QUEUE and its related struct
> From: Nicolin Chen > Sent: Friday, May 9, 2025 11:03 AM > > Add IOMMUFD_OBJ_HW_QUEUE with an iommufd_hw_queue structure, > representing > a HW-accelerated queue type of IOMMU's physical queue that can be passed > through to a user space VM for direct hardware control, such as: > - NVIDIA's Virtual Command Queue > - AMD vIOMMU's Command Buffer, Event Log Buffer, and PPR Log Buffer > > Introduce an allocator iommufd_hw_queue_alloc(). And add a pair of > viommu > ops for iommufd to forward user space ioctls to IOMMU drivers. > > Given that the first user of this HW QUEUE (tegra241-cmdqv) will need to > ensure the queue memory to be physically contiguous, add a flag property > in iommufd_viommu_ops and > IOMMUFD_VIOMMU_FLAG_HW_QUEUE_READS_PA to allow > driver to flag it so that the core will validate the physical pages of a > given guest queue. 'READS' is confusing here. What about xxx_CONTIG_PAS? > + * @hw_queue_alloc: Allocate a HW QUEUE object for a HW-accelerated > queue given > + * the @type (must be defined in > include/uapi/linux/iommufd.h) > + * for the @viommu. @index carries the logical HW QUEUE ID > per > + * @viommu in a guest VM, for a multi-queue case; @addr > carries > + * the guest physical base address of the queue memory; s/@addr/@base_addr/ Reviewed-by: Kevin Tian
