On Tue, Oct 12, 2010 at 2:01 PM, Tzachi Dar <[email protected]> wrote: > Do you believe that interupt moderation will be able to help?
Yes. The CPU's pegged seem to be the current bottleneck. If that load could be distributed to other CPU's or more work done per interrupt, then that should help. Of course, I don't understand what's going on; I assume that the HCA can buffer the work... if coalescing just backs-up the pipeline, then it will not help. > > There is currently no way for the user can control that but I hope that this > is a small change in the code. > > By the way, are you running connect2 or connect? I believe these are ConnectX QDR HCA's, as neither are brand new... but I think I have access to newer HCA's if that would help. Thanks, Chris > > Thanks > Tzachi > >> -----Original Message----- >> From: Chris Worley [mailto:[email protected]] >> Sent: Tuesday, October 12, 2010 7:10 PM >> To: Tzachi Dar >> Cc: [email protected] >> Subject: Re: [ofw] Any interrupt coalescing in WinOF? >> >> On Tue, Oct 12, 2010 at 10:54 AM, Tzachi Dar <[email protected]> >> wrote: >> > There is no interrupt coalescing on winof although the hw does >> support that (I assume that you are running connectx). >> > >> > You can verify if the number of interrupts is big by running perfmon. >> Select the cpu, and choose number of interrupts per second. >> > >> > Look at this when your test is running and when it is not. >> >> Although there are some Heisenberg issues w/ this test (perfmon locks >> up trying to measure cores that are pegged), I was able to confirm >> that all the interrupts are hitting these two cores, and assume each >> core is getting the interrupts for one HCA. >> >> Thaks, >> >> Chris >> > >> > Thanks >> > Tzachi >> > >> >> -----Original Message----- >> >> From: Chris Worley [mailto:[email protected]] >> >> Sent: Tuesday, October 12, 2010 6:49 PM >> >> To: Tzachi Dar >> >> Cc: [email protected] >> >> Subject: Re: [ofw] Any interrupt coalescing in WinOF? >> >> >> >> On Tue, Oct 12, 2010 at 10:34 AM, Tzachi Dar >> <[email protected]> >> >> wrote: >> >> > What test/ulp are you using? >> >> >> >> IOMeter with one worker atop each of 4 SRP-imported/multipathed >> >> (round-robin) drives and a queue depth of 64. I'm seeing ~450K >> random >> >> 4KB write IOPS. >> >> >> >> With two HCA's, both running the latest firmware (as of Friday), I >> see >> >> two cores on one NUMA node totally pegged/consumed, and affinity is >> >> set on the Dynamo's to avoid these cores... so something is either >> >> polling or interrupt handling on these. >> >> >> >> I'm running the latest 2.3RC4 WinOF. >> >> >> >> Thanks, >> >> >> >> Chris >> >> > >> >> > Thanks >> >> > Tzachi >> >> > >> >> >> -----Original Message----- >> >> >> From: [email protected] [mailto:ofw- >> >> >> [email protected]] On Behalf Of Chris Worley >> >> >> Sent: Tuesday, October 12, 2010 6:33 PM >> >> >> To: [email protected] >> >> >> Subject: [ofw] Any interrupt coalescing in WinOF? >> >> >> >> >> >> I'm seeing the interrupt handlers completely consuming a CPU and >> >> >> becoming a bottleneck, and no interrupt load balancing across >> CPU's >> >> >> (except w/ multiple HCA's, each HCA consumes one CPU w/ >> interrupts). >> >> >> >> >> >> Is interrupt coalescing manually tunable in WinOF? >> >> >> >> >> >> Or, am I totally off-base, and seeing polling and not interrupt >> >> >> behavior? >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Chris >> >> >> _______________________________________________ >> >> >> ofw mailing list >> >> >> [email protected] >> >> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw >> >> > >> > > _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
