On Wed, Jul 02, 2025 at 03:05:41PM -0300, Jason Gunthorpe wrote:
> On Wed, Jul 02, 2025 at 01:38:33AM +0000, Pranjal Shrivastava wrote:
> > On Tue, Jul 01, 2025 at 05:46:06PM -0700, Nicolin Chen wrote:
> > > On Wed, Jul 02, 2025 at 12:14:28AM +0000, Pranjal Shrivastava wrote:
> > > > Thus, coming back to the two initial points:
> > > >
> > > > 1) Issuing "non-invalidation" commands through .cache_invalidate could
> > > > be confusing, I'm not asking to change the op name here, but if we
> > > > plan to label it, let's label them as "Trapped commands" OR
> > > > "non-accelerated" commands as you suggested.
> > >
> > > VCMDQ only accelerates limited invalidation commands, not all of
> > > them: STE cache invalidation and CD cache invalidation commands
> > > still go down to that op.
> > >
> >
> > Right, I'm just saying the "other" non-accelerated commands that are
> > NOT invalidations also go down that op. So, if we add a comment, let's
> > not call them "non-invalidation" commands.
>
> There are no non-invalidation commands:
>
> static int arm_vsmmu_convert_user_cmd(struct arm_vsmmu *vsmmu,
> struct arm_vsmmu_invalidation_cmd *cmd)
> {
> switch (cmd->cmd[0] & CMDQ_0_OP) {
> case CMDQ_OP_TLBI_NSNH_ALL:
> case CMDQ_OP_TLBI_NH_VA:
> case CMDQ_OP_TLBI_NH_VAA:
> case CMDQ_OP_TLBI_NH_ALL:
> case CMDQ_OP_TLBI_NH_ASID:
> case CMDQ_OP_ATC_INV:
> case CMDQ_OP_CFGI_CD:
> case CMDQ_OP_CFGI_CD_ALL:
>
> Those are only invalidations.
>
> CD invalidation can't go through the vCMDQ path.
>
Right.. I was however hoping we'd also trap commands like CMD_PRI_RESP
and CMD_RESUME...I'm not sure if they should be accelerated via CMDQV..
I guess I'll need to look and understand a little more if they are..
> > > > 2) The "FIXME" confusion: The comment in arm_vsmmu_cache_invalidate
> > > > mentions we'd like to "fix" the issuing of commands through the main
> > > > cmdq and instead like to group by "type", if that "type" is the queue
> > > > type (which I assume it is because IOMMU_TYPE has to be arm-smmu-v3),
> > >
> > > I recall that FIXME is noted by Jason at that time. And it should
> > > be interpreted as "group by opcode", IIUIC.
> >
> > I see.. I misunderstood that..
>
> Yes, we could use the vCMDQ in the SMMU driver for invalidations which
> would give some minor locking advantage. But it is not really
> important to anyone.
>
Alright, I see. Makes sense. Thanks for the clarification.
> > > The thing is that for a host kernel that enabled in-kernel VCMDQs,
> > > those trapped user commands can be just issued to the smmu->cmdq
> > > or a vcmdq (picked via the get_secondary_cmdq impl_op).
> >
> > Ohh.. so maybe some sort of a load balancing thing?
>
> The goal of the SMMU driver when it detects CMDQV support is to route
> all supported invalidations to CMDQV queues and then balance those
> queues across CPUs to reduce lock contention.
>
I see.. that makes sense.. so it's a relatively small gain (but a nice
one). Thanks for clarifying!
> Jason
Praan
