Re: [OMPI users] Open MPI over RoCE using breakout cable and switch

2017-02-01 Thread Brendan Myers
Hello Howard, I was wondering if you have been able to look at this issue at all, or if anyone has any ideas on what to try next. Thank you, Brendan From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Brendan Myers Sent: Tuesday, January 24, 2017 11:11 AM To: 'Open MPI

Re: [OMPI users] Open MPI over RoCE using breakout cable and switch

2017-02-07 Thread Brendan Myers
setup problem that needs to be resolved first before trying custom Open MPI tarballs. Thanks, Howard 2017-02-01 15:08 GMT-07:00 Brendan Myers <brendan.my...@soft-forge.com <mailto:brendan.my...@soft-forge.com> >: Hello Howard, I was wondering if you have

Re: [OMPI users] Open MPI over RoCE using breakout cable and switch

2017-01-23 Thread Brendan Myers
e list? Howard Brendan Myers <brendan.my...@soft-forge.com <mailto:brendan.my...@soft-forge.com> > schrieb am Fr. 20. Jan. 2017 um 15:04: Hello, I am attempting to get Open MPI to run over 2 nodes using a switch and a single breakout cable with this design: (100GbE)QSFP <

Re: [OMPI users] Open MPI over RoCE using breakout cable and switch

2017-01-24 Thread Brendan Myers
ersion of Open MPI by adding --enable-debug to the config options and rerun the test with the breakout cable setup and keeping the --mca btl_base_verbose 100 command line option? Thanks Howard 2017-01-23 8:23 GMT-07:00 Brendan Myers <brendan.my...@soft-forge.com <mailto:brenda

[OMPI users] Open MPI over RoCE using breakout cable and switch

2017-01-20 Thread Brendan Myers
Hello, I am attempting to get Open MPI to run over 2 nodes using a switch and a single breakout cable with this design: (100GbE)QSFP <> 2x (50GbE)QSFP Hardware Layout: Breakout cable module A connects to switch (100GbE) Breakout cable module B1 connects to node 1 RoCE NIC

[OMPI users] How to verify RDMA traffic (RoCE) is being sent over a fabric when running OpenMPI

2016-11-08 Thread Brendan Myers
Hello, I am trying to figure out how I can verify that the OpenMPI traffic is actually being transmitted over my RoCE fabric connecting my cluster. My MPI job runs quickly and error free but I cannot seem to verify that significant amounts of data is being transferred to the other endpoint in my

Re: [OMPI users] rdmacm and udcm failure in 2.0.1 on RoCE

2016-12-16 Thread Brendan Myers
Hello, I can confirm that using these flags: --mca btl_openib_receive_queues P,65536,120,64,32 --mca btl_openib_cpc_include rdmacm I am able to run Open MPI version 2.0.1 over a RoCE fabric. Hope this helps Thank you, Brendan Myers Software Forge From: users [mailto:users-boun

[OMPI users] RoCE device performance with large message size

2017-10-10 Thread Brendan Myers
Hello All, I have a RoCE interoperability event starting next week and I was wondering if anyone had any ideas to help me with a new vendor I am trying to help get ready. I am using: * Open MPI 2.1 * Intel MPI Benchmarks 2018 * OFED 3.18 (requirement from vendor) *