Or, Did you take a look at the cpu utilization while running ? Did u see higher cpu utilization while getting the 850MB/sec ? How many cpu's were actually working ? I will try and reproduce your results. Sagi
-----Original Message----- From: Or Gerlitz [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 25, 2008 6:34 PM To: Sagi Rotem Cc: [email protected] Subject: Re: msi-x seems to cause large performance variation for connectx / datagram mode > digging a little more, I see now that if I disable msi-x, the > performance gets to a fixed value of about 620MB/s with connectx which > is the lower range that msi-x gives me... setting the irq daemon off, I see that if I set the cpu affinity of the netperf process to be as the one of the mlx4 irq I get fixed performance of 650 MB/s If I set it to different CPU, then eventually I get 480MB/s constantly and 620MB/s constantly if is a different core on the same CPU... HCA MB/s msi_x irqdaemon netperf affinity ======================================================================= ConnectX-ConnectX 620 0 on none ConnectX-ConnectX 620-850 1 on none ConnectX-ConnectX 650 1 off same as irq ConnectX-ConnectX 620 1 off different core ConnectX-ConnectX 480 1 off different cpu Or. _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
