You know, I have the following in a few of my MTT configurations:
-----
# See if this makes the CQ overrun errors go away
cq_depth = " --mca btl_openib_cq_size 65536 "
-----
And then I use that variable as an mpirun CLI option in a few places. It looks
like something left over from a long time ago that never got followed up on...
So yes, I'm guessing there's some kind of incorrect CQ sizing issue going on.
Can someone point Steve to the right place to look in the openib BTL?
On Jul 2, 2012, at 11:24 AM, Steve Wise wrote:
> Hello,
>
> I'm debugging an issue with openmpi-1.4.5 and the openib btl over chelsio
> iwarp devices. I am the iwarp driver developer for this device. I have
> debug code that detects cq overflows, and I'm seeing rcq overflows during
> finalize for certain IMB runs with ompi. So as the recv wrs are flushed, I
> am seeing an overflow in the rcq for that qp. Note chelsio iwarp uses
> non-shared rqs and its default .ini is: receive_queues = P,65536,256,192,128
>
> Here's the job details:
>
> NP=16; mpirun -np ${NP} --host core96b1,core96b2,core96b3,core96b4 --mca btl
> openib,sm,self /opt/openmpi-1.4.5/tests/IMB-3.2/IMB-MPI1 -npmin ${NP} alltoall
>
> The nodes have 4 port iwarp adapters in them so there are rdma connections
> setup over each port. As the alltoall IO size hits 256, we end up with 192
> qps per node. And that seems to be the stable qp count until the test
> finishes and we see the overflow.
>
> I added further debug code in my rdma provider library to track the total
> depth of all the qps bound to each cq to see if the application is
> oversubscribing the cqs. I see that for these jobs, OMPI is in fact
> oversubscribing the cqs. Here's a snipit of my debug output:
>
> warning, potential SCQ overflow: total_qp_depth 3120 SCQ depth 1088
> warning, potential RCQ overflow: total_qp_depth 3312 RCQ depth 1088
> warning, potential SCQ overflow: total_qp_depth 3120 SCQ depth 1088
> warning, potential RCQ overflow: total_qp_depth 3312 RCQ depth 1088
>
> I realize that OMPI can in fact be flow controlling such that the cq won't
> overflow even if the total qp depths exceeds the cq depth. But I do see
> overflows. And it seems that a cq depth of 1088 is quite small given the
> size of the sq or rq in the above debug output. So it seems that ompi isn't
> scaling the CQ depth according to the job.
>
> As an experiment, I overrode the cq depth by adding '--mca
> btl_openib_cq_size 16000' to the mpirun line and I don't see the overflow
> anymore.
>
> Can all you openib btl experts out there describe the CQ sizing logic and
> point me to the code that I can dig into to see why we're overflowing the RCQ
> on finalize operations? Also, does the cq depth of 1088 seem reasonable for
> this type of work load?
>
> Thanks in advance!
>
> Steve.
>
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
