Michael S. Tsirkin wrote: Michael> Quoting r. Shirley Ma <[EMAIL PROTECTED]>: Michael> > different drivers have different implementations for CQ completion handler.
Michael> Maybe these drivers should be changed then? Its a bit hard for me to imagine a Michael> driver that doesn't get hardware interrupts in IRQ context. So why can't Michael> completion handler be called directly from there as well? The problem we are seeing is not how we handle the hardware interrupt, but what is handled where, after getting the interrupt. All packet processing is handled by a single processor. On a multiprocessor machine with high traffic rate, the bandwidth is limited by a single pro, and to different CPU's than the one handling the interrupt. Running the 10 GigE adapter in classic packet mode ( 1500 byte MTU ) gets better performance with splitting the packet handling across multiple CPU's. IB is suffering from this same problem in the completion handler. If you have a 4 CPU machine and run two sockets over teh same adapter/link using NETPERF, if you use mpstat -P ALL, you will see one CPU pegged and the rest much lower in CPU utilization, The CPU at 100% is the one running the interrupt handler, TCP/IP stack and completion queue. Shirley's patch is to split the processing so that completion queue handling can be done on a different CPU that the interrupt handler. Yes there are locking issues with the completion queue being accessed by different CPU's but there is an overall gain in bandwidth especially in duplex mode. Having all completion queue handling done off the hardware interrupt means you have serialized the completions for a duplex adapter. Michael> -- Michael> MST Bernie King-Smith IBM Corporation Server Group Cluster System Performance [EMAIL PROTECTED] (845)433-8483 Tie. 293-8483 or wombat2 on NOTES "We are not responsible for the world we are born into, only for the world we leave when we die. So we have to accept what has gone before us and work to change the only thing we can, -- The Future." William Shatner _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
