Roland Dreier <[EMAIL PROTECTED]> wrote on 04/18/2006 01:45:17 PM:
> > > Actually, do you have some explanation for why this helps performance?
> > > My intuition would be that it just generates more interrupts for the
> > > same workload.
>
> > The only lock contension in IPoIB I saw is tx_lock. When seperating
> > the completion queue to have seperate completion handler. It could improve
> > the performance. I didn't look at driver code, it might have some impact
> > there?
>
> A clever way to avoid taking the TX lock on send completions would be
> very nice, but I never saw a way to do it. Does splitting the CQ
> reduce contention? I don't see why that would be, since the
> contention is between sending and getting send completions. The
> receive path of course never touches the tx_lock.
tx_lock contension will block the CQ handler to process next wiki in CQ,
which could be recv wiki or send wiki.
> By the way, are your numbers with mthca or ehca? I don't know ehca
> very well, but at least with current mthca, all CQ events will be
> delivered on the same interrupt and hence all CQ handling will run on
> the same CPU. So I'm puzzled why changing things from:
>
> -> interrupt
> -> CQ event handler
> -> handle all IPoIB completions
>
> to:
>
> -> interrupt
> -> TX CQ event handler
> -> handle TX completions
> [possibly another interrupt]
> -> RX CQ event handler
> -> handle RX completions
>
> helps throughput. It just seems like it's more CQ locking/unlocking
> and in general more work.
>
> - R.
If recv CQ and send CQ are at different rate, splitting CQ would reduce CQ locking
unlocking.
Thanks
Shirley Ma
IBM Linux Technology Center
15300 SW Koll Parkway
Beaverton, OR 97006-6063
Phone(Fax): (503) 578-7638
_______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
