> > I would just like to see an approach that is fully thought through and > gives a way for applications/kernel drivers to choose a CQ vector based > on some information about what CPU it will go to. >
Isn't the decision of which CPU an MSI-X is routed to (and hence, to which CPI an EQ is bound to) determined by userspace? (either by the irq balancer process or by manually setting /proc/irq/<vec>/smp_affinity)? I am not sure we aren't better off leaving this to user-space: both application and interrupt affinity are administrative tasks. We can also use installation scripts to set a "default" configuration in which vector 0 is bound to cpu0, vector 1 is bound to cpu1, etc. > If we want to add a way to allow a request for round-robin, that is > fine, but I don't think we want to change the default to round-robin, > unless someone can come up with a workload where it actually helps. Several IPoIB partitions can easily saturate a single core if their Rx interrupts are not handled by several CPUs. This is not any different from multiple Ethernet NICs whose interrupts are balanced today by the irq balancer. We can argue that IPoIB can use a special "round-robin" vector while leaving the default vector fixed to a single EQ. However, there is essentially no difference between IPoIB and other IB ULPs: an IB HCA is actually a platform for other services, each with its own queues that are directly accessed by HW, each with its own CQs and interrupt moderation. Putting all these ULPs on a single EQ will prevent interrupt balancing. What are we risking in making the default action to spread interrupts? --Liran _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
