Howard,
The btl_openib_receive_queue parameters you provided work for me on
a version of 2.0.1 I compiled in my home directory, but segfault on our
globally installed version for some reason. Anyway, thanks for the info.
I'll get together with my sys admin and we should be able to figure out the
differences in the configuration that may have caused this problem.
Dave
On Thu, Jan 5, 2017 at 10:53 AM, Howard Pritchard <hpprit...@gmail.com>
wrote:
> Hi Dave,
>
> Sorry for the delayed response.
>
> Anyway, you have to use rdmacm for connection management when using ROCE.
> However, with 2.0.1 and later, you have to specify per peer QP info
> manually
> on the mpirun command line.
>
> Could you try rerunning with
>
> mpirun --mca btl_openib_receive_queues P,128,64,32,32,32:S,2048,1024,128,32:S,
> 12288,1024,128,32:S,65536,1024,128,32 (all the reset of the command line
> args)
>
> and see if it then works?
>
> Howard
>
>
> 2017-01-04 16:37 GMT-07:00 Dave Turner <drdavetur...@gmail.com>:
>
>> ------------------------------------------------------------
>> --------------
>> No OpenFabrics connection schemes reported that they were able to be
>> used on a specific port. As such, the openib BTL (OpenFabrics
>> support) will be disabled for this port.
>>
>> Local host: elf22
>> Local device: mlx4_2
>> Local port: 1
>> CPCs attempted: rdmacm, udcm
>> ------------------------------------------------------------
>> --------------
>>
>> I posted this to the user list but got no answer so I'm reposting to
>> the devel list.
>>
>> We recently upgraded to OpenMPI 2.0.1. Everything works fine
>> on our QDR connections but we get the error above for our
>> 40 GbE connections running RoCE. I traced through the code and
>> it looks like udcm cannot be used with RoCE. I've also read that
>> there are currently some problems with rdmacm under 2.0.1, which
>> would mean 2.0.1 does not currently work on RoCE. We've tested
>> 10.4 using rdmacm and that works fine so I don't think we have anything
>> configured wrong on the RoCE side.
>> Could someone please verify whether this information is correct that
>> RoCE requires rdmacm only and not udcm, and that rdmacm is currently
>> not working. If so, is it being worked on?
>>
>> Dave
>>
>>
>> --
>> Work: davetur...@ksu.edu (785) 532-7791
>> 2219 Engineering Hall, Manhattan KS 66506
>> Home: drdavetur...@gmail.com
>> cell: (785) 770-5929
>>
>> _______________________________________________
>> devel mailing list
>> devel@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
>>
>
>
--
Work: davetur...@ksu.edu (785) 532-7791
2219 Engineering Hall, Manhattan KS 66506
Home: drdavetur...@gmail.com
cell: (785) 770-5929
_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
