Hi Dave, Sorry for the delayed response.
Anyway, you have to use rdmacm for connection management when using ROCE. However, with 2.0.1 and later, you have to specify per peer QP info manually on the mpirun command line. Could you try rerunning with mpirun --mca btl_openib_receive_queues P,128,64,32,32,32:S,2048,1024,128,32:S, 12288,1024,128,32:S,65536,1024,128,32 (all the reset of the command line args) and see if it then works? Howard 2017-01-04 16:37 GMT-07:00 Dave Turner <drdavetur...@gmail.com>: > -------------------------------------------------------------------------- > No OpenFabrics connection schemes reported that they were able to be > used on a specific port. As such, the openib BTL (OpenFabrics > support) will be disabled for this port. > > Local host: elf22 > Local device: mlx4_2 > Local port: 1 > CPCs attempted: rdmacm, udcm > -------------------------------------------------------------------------- > > I posted this to the user list but got no answer so I'm reposting to > the devel list. > > We recently upgraded to OpenMPI 2.0.1. Everything works fine > on our QDR connections but we get the error above for our > 40 GbE connections running RoCE. I traced through the code and > it looks like udcm cannot be used with RoCE. I've also read that > there are currently some problems with rdmacm under 2.0.1, which > would mean 2.0.1 does not currently work on RoCE. We've tested > 10.4 using rdmacm and that works fine so I don't think we have anything > configured wrong on the RoCE side. > Could someone please verify whether this information is correct that > RoCE requires rdmacm only and not udcm, and that rdmacm is currently > not working. If so, is it being worked on? > > Dave > > > -- > Work: davetur...@ksu.edu (785) 532-7791 > 2219 Engineering Hall, Manhattan KS 66506 > Home: drdavetur...@gmail.com > cell: (785) 770-5929 > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/devel >
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
