Re: [OMPI users] btl_openib_receive_queues & mca-btl-openib-device-params.ini

[Sasso, John (GE Power & Water, Non-GE)]

I did some code-digging and I found the answer.

If the MCA parameter btl_openib_receive_queues is not spec'd on the mpirun 
command line and not specified in 
MPI_HOME/share/openmpi/mca-btl-openib-device-params.ini  (via receive_queues 
parameter), then OpenMPI derives the default setting from the code in 
ompi/mca/btl/openib/btl_openib_mca.c.  If you look in that code you will see:

snprintf(default_qps, 100,

"P,128,256,192,128:S,%u,1024,1008,64:S,%u,1024,1008,64:S,%u,1024,1008,64",
mid_qp_size,   // For S,%u,1024,1008,64 where %u == 2048
(uint32_t)mca_btl_openib_module.super.btl_eager_limit,
(uint32_t)mca_btl_openib_module.super.btl_max_send_size);

So for OpenMPI 1.6.5 we have:


P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64

So you can examine the code and see how the %u's are filled in to get the 
resultant default btl_openib_receive_queues.

Nice if this was documented!

--john


From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Shamis, Pavel
Sent: Thursday, November 05, 2015 2:41 PM
To: Open MPI Users
Subject: Re: [OMPI users] btl_openib_receive_queues & 
mca-btl-openib-device-params.ini

Below is from ompi_info:

$ ompi_info --all | grep btl_openib_receive
 MCA btl: parameter "btl_openib_receive_queues" (current value: 
<P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64>,
 data source: default value)

This tunning does make sense for me.



#receive_queues = P,128,256,192,128:S,65536,256,192,128

Most likely it was not updated for a long time.
I'm pretty sure the ompi_info version is the one that actually used.



So comparing the two above, they don't coincide, in particular for the SRQ 
settings.  Thus, I gather the settings reported by ompi_info above are those 
that are actually used, and if so, I am wondering where/how they are derived?

The idea behind this parameters is coming from here:
http://www.open-mpi.org/papers/euro-pvmmpi-2007-ib/euro-pvmmpi-2007-ib.pdf<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.open-2Dmpi.org_papers_euro-2Dpvmmpi-2D2007-2Dib_euro-2Dpvmmpi-2D2007-2Dib.pdf=CwMF-g=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI=tqKZ2vRCLufSSXPvzNxBrKr01YPimBPnb-JT-Js0Fmk=ful34HOXqlPDIxbufExLGHQRNlgb5Oy60k_TKVG3VmI=YOBOBrSC7_mvy-6Z2gqNRLZGDJTQ9UoTa1pZpzfxIcY=>
And later on XRC support was introduced:
http://www.open-mpi.org/papers/euro-pvmmpi-2008-xrc/euro-pvmmpi-2008-xrc.pdf<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.open-2Dmpi.org_papers_euro-2Dpvmmpi-2D2008-2Dxrc_euro-2Dpvmmpi-2D2008-2Dxrc.pdf=CwMF-g=IV_clAzoPDE253xZdHuilRgztyh_RiV3wUrLrDQYWSI=tqKZ2vRCLufSSXPvzNxBrKr01YPimBPnb-JT-Js0Fmk=ful34HOXqlPDIxbufExLGHQRNlgb5Oy60k_TKVG3VmI=wKpbgirfQeTb_SW5gKhPep2khno1MHxnc-LdWVkk7Og=>

Best,
Pasha

Re: [OMPI users] btl_openib_receive_queues & mca-btl-openib-device-params.ini

[Shamis, Pavel]

Below is from ompi_info:

$ ompi_info --all | grep btl_openib_receive
 MCA btl: parameter "btl_openib_receive_queues" (current value: 
,
 data source: default value)

This tunning does make sense for me.


#receive_queues = P,128,256,192,128:S,65536,256,192,128

Most likely it was not updated for a long time.
I'm pretty sure the ompi_info version is the one that actually used.


So comparing the two above, they don’t coincide, in particular for the SRQ 
settings.  Thus, I gather the settings reported by ompi_info above are those 
that are actually used, and if so, I am wondering where/how they are derived?

The idea behind this parameters is coming from here:
http://www.open-mpi.org/papers/euro-pvmmpi-2007-ib/euro-pvmmpi-2007-ib.pdf
And later on XRC support was introduced:
http://www.open-mpi.org/papers/euro-pvmmpi-2008-xrc/euro-pvmmpi-2008-xrc.pdf

Best,
Pasha

[OMPI users] btl_openib_receive_queues & mca-btl-openib-device-params.ini

[Sasso, John (GE Power & Water, Non-GE)]

Inquiring about how btl_openib_receive_queues actually gets its default 
setting, since what I am seeing is not joving with documentation.  We are using 
OpenMPI 1.6.5, but I gather the version is moot.

Below is from ompi_info:

$ ompi_info --all | grep btl_openib_receive
 MCA btl: parameter "btl_openib_receive_queues" (current value: 
,
 data source: default value)


And based on the IB HCA part ID, this is the entry in 
mca-btl-openib-device-params.ini which is applicable:

# A.k.a. ConnectX
[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708,0x03ba,0x15b3,0x119f
vendor_part_id = 
25408,25418,25428,25448,26418,26428,26438,26448,26468,26478,26488,4099,4103,4100
use_eager_rdma = 1
mtu = 2048
max_inline_data = 128


Yet according to the doc in mca-btl-openib-device-params.ini, the default 
setting if receive_queues is not specified should be:

#receive_queues = P,128,256,192,128:S,65536,256,192,128

So comparing the two above, they don't coincide, in particular for the SRQ 
settings.  Thus, I gather the settings reported by ompi_info above are those 
that are actually used, and if so, I am wondering where/how they are derived?

--john