One of our users is having trouble scaling his code up to 3584 cores (i.e. 128 
28-core nodes). It runs fine on 1792 cores (64 nodes), but fails with this at 

A process failed to create a queue pair. This usually means either
the device has run out of queue pairs (too many connections) or
there are insufficient resources available to allocate a queue pair
(out of memory). The latter can happen if either 1) insufficient
memory is available, or 2) no more physical memory can be registered
with the device.
For more information on memory registration see the Open MPI FAQs at:
Local host: r3735
Local device: mlx5_0
Queue pair type: Reliable connected (RC)

Looking on the node in question, sure enough there’s a message in dmesg:

[347071.005636] mlx5_core 0000:06:00.0: mlx5_cmd_check:727:(pid 31507): 
CREATE_QP(0x500) op_mod(0x0) failed, status bad resource(0x5), syndrome 

I’m pretty sure 0x65b500 means "out of queue pairs”.

Our HCAs support 262144 QPs, and while some of these will be used for e.g. 
IPoIB and Lustre, I wouldn’t expect to be running out at such a low number of 
cores — and indeed, I’ve run much larger jobs without seeing this issue.

This is using the 1.10 series, with the ob1 PML with the openib BTL. If they 
use Yalla, it works fine, but it would still be good to get it working using 
the “standard” communication path, without needing the accelerators.

I was wondering if anyone seen this before, and if anyone had any suggestions 
for how to proceed?


users mailing list

Reply via email to