Do you believe that interupt moderation will be able to help? There is currently no way for the user can control that but I hope that this is a small change in the code.
By the way, are you running connect2 or connect? Thanks Tzachi > -----Original Message----- > From: Chris Worley [mailto:[email protected]] > Sent: Tuesday, October 12, 2010 7:10 PM > To: Tzachi Dar > Cc: [email protected] > Subject: Re: [ofw] Any interrupt coalescing in WinOF? > > On Tue, Oct 12, 2010 at 10:54 AM, Tzachi Dar <[email protected]> > wrote: > > There is no interrupt coalescing on winof although the hw does > support that (I assume that you are running connectx). > > > > You can verify if the number of interrupts is big by running perfmon. > Select the cpu, and choose number of interrupts per second. > > > > Look at this when your test is running and when it is not. > > Although there are some Heisenberg issues w/ this test (perfmon locks > up trying to measure cores that are pegged), I was able to confirm > that all the interrupts are hitting these two cores, and assume each > core is getting the interrupts for one HCA. > > Thaks, > > Chris > > > > Thanks > > Tzachi > > > >> -----Original Message----- > >> From: Chris Worley [mailto:[email protected]] > >> Sent: Tuesday, October 12, 2010 6:49 PM > >> To: Tzachi Dar > >> Cc: [email protected] > >> Subject: Re: [ofw] Any interrupt coalescing in WinOF? > >> > >> On Tue, Oct 12, 2010 at 10:34 AM, Tzachi Dar > <[email protected]> > >> wrote: > >> > What test/ulp are you using? > >> > >> IOMeter with one worker atop each of 4 SRP-imported/multipathed > >> (round-robin) drives and a queue depth of 64. I'm seeing ~450K > random > >> 4KB write IOPS. > >> > >> With two HCA's, both running the latest firmware (as of Friday), I > see > >> two cores on one NUMA node totally pegged/consumed, and affinity is > >> set on the Dynamo's to avoid these cores... so something is either > >> polling or interrupt handling on these. > >> > >> I'm running the latest 2.3RC4 WinOF. > >> > >> Thanks, > >> > >> Chris > >> > > >> > Thanks > >> > Tzachi > >> > > >> >> -----Original Message----- > >> >> From: [email protected] [mailto:ofw- > >> >> [email protected]] On Behalf Of Chris Worley > >> >> Sent: Tuesday, October 12, 2010 6:33 PM > >> >> To: [email protected] > >> >> Subject: [ofw] Any interrupt coalescing in WinOF? > >> >> > >> >> I'm seeing the interrupt handlers completely consuming a CPU and > >> >> becoming a bottleneck, and no interrupt load balancing across > CPU's > >> >> (except w/ multiple HCA's, each HCA consumes one CPU w/ > interrupts). > >> >> > >> >> Is interrupt coalescing manually tunable in WinOF? > >> >> > >> >> Or, am I totally off-base, and seeing polling and not interrupt > >> >> behavior? > >> >> > >> >> Thanks, > >> >> > >> >> Chris > >> >> _______________________________________________ > >> >> ofw mailing list > >> >> [email protected] > >> >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw > >> > > > _______________________________________________ ofw mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
