Hi Michael,
When Open MPI run on Omni-Path it will choose the PSM2 MTL by default, to use
the libpsm2.so. Strictly speaking, it has compatibility to run using the openib
BTL. However, the performance so significantly impacted that it is, not only
discouraged, but no tuning would make sense. Regarding the PSM2 MTL, currently
it only supports two mca parameters ("mtl_psm2_connect_timeout" and
"mtl_psm2_priority") which are not for what you are looking for. Instead, you
can set values directly in the PSM2 library with environment variables.
Further info in the Programmers Guide:
http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v3_0.pdf
More docs:
https://www-ssl.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/000016242.html?wapkw=psm2
Now, for your parameters:
btl = openib,vader,self -> Ignore this one
btl_openib_eager_limit = 160000 -> I don’t clearly see the diff with the
below parameter. However, they are set to the same value. Just in case
PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode.
btl_openib_rndv_eager_limit = 160000 -> PSM2_MQ_RNDV_HFI_THRESH
btl_openib_max_send_size = 160000 -> does not apply to PSM2
btl_openib_receive_queues =
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,160000,1024,512,512
-> does not apply for PSM2.
Thanks,
Regards,
_MAC
BTW, should change the subject OMA -> OPA
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui
Sent: Tuesday, August 09, 2016 2:22 PM
To: users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
Subject: [OMPI users] runtime performance tuning for Intel OMA interconnect
I used to tune the performance of OpenMPI on InfiniBand by changing the OpenMPI
MCA parameters for openib component (see
https://www.open-mpi.org/faq/?category=openfabrics). Now I migrate to a new
cluster that deploys Intel's omni-path interconnect, and my previous approach
does not work any more. Does anyone know how to tune the performance for
omni-path interconnect (what OpenMPI component to change) ?
The version of OpenMPI is openmpi-1.10.2-hfi. I have included the output from
opmi_info and openib parameters that I used to change. Thanks!
Sincerely,
Michael
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
