On 8/24/2018 9:17 PM, Sagi Grimberg wrote:
>
>>> nvme-rdma attempts to map queues based on irq vector affinity.
>>> However, for some devices, completion vector irq affinity is
>>> configurable by the user which can break the existing assumption
>>> that irq vectors are optimally arranged over the host cpu cores.
>>
>> IFF affinity is configurable we should never use this code,
>> as it breaks the model entirely. ib_get_vector_affinity should
>> never return a valid mask if affinity is configurable.
>
> I agree that the model intended initially doesn't fit. But it seems
> that some users like to write into their nic's
> /proc/irq/$IRP/smp_affinity and get mad at us for not letting them with
> using managed affinity.
>
> So instead of falling back to the block mapping function we try
> to do a little better first:
> 1. map according to the device vector affinity
> 2. map vectors that end up without a mapping to cpus that belong
> to the same numa-node
> 3. map all the rest of the unmapped cpus like the block layer
> would do.
>
> We could have device drivers that don't use managed affinity to never
> return a valid mask but that would never allow affinity based mapping
> which is optimal at least for users that do not mangle with device
> irq affinity (which is probably the majority of users).
>
> Thoughts?
Can we please make forward progress on this?
Christoph, Sagi: it seems you think /proc/irq/$IRP/smp_affinity
shouldn't be allowed if drivers support managed affinity. Is that correct?
Perhaps that can be codified and be a way forward? IE: Somehow allow
the admin to choose either "managed by the driver/ulps" or "managed by
the system admin directly"?
Or just use Sagi's patch. Perhaps a WARN_ONCE() if the affinity looks
wonked when set via procfs? Just thinking out loud...
But as it stands, things are just plain borked if an rdma driver
supports ib_get_vector_affinity() yet the admin changes the affinity via
/proc...
Thanks,
Steve.
