Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-27 Thread Barrett, Brian via users
On Sep 11, 2018, at 10:46 AM, Benjamin Brock mailto:br...@cs.berkeley.edu>> wrote: Thanks for your response. One question: why would RoCE still requiring host processing of every packet? I thought the point was that some nice server Ethernet NICs can handle RDMA requests directly? Or am I

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-26 Thread Benjamin Brock
In case anyone comes across this thread in an attempt to get RDMA over Ethernet working on AWS, here's the conclusion I came to: There are two kinds of NICs exposed to VMs on AWS: - Intel 82599 VF - This NIC is old and does not support RoCE or iWARP. - It's a virtualized view of an

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-11 Thread Jeff Hammond
Are you trying to run UPC++ over MPI in the cloud? Jeff On Tue, Sep 11, 2018 at 10:46 AM, Benjamin Brock wrote: > Thanks for your response. > > One question: why would RoCE still requiring host processing of every > packet? I thought the point was that some nice server Ethernet NICs can >

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-11 Thread Benjamin Brock
Thanks for your response. One question: why would RoCE still requiring host processing of every packet? I thought the point was that some nice server Ethernet NICs can handle RDMA requests directly? Or am I misunderstanding RoCE/how Open MPI's RoCE transport? Ben

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-10 Thread Barrett, Brian via users
It sounds like what you’re asking is “how do I get the best performance from Open MPI in AWS?”. The TCP BTL is your best option for performance in AWS. RoCE is going to be a bunch of work to get setup, and you’ll still end up with host processing of every packet. There are a couple simple

Re: [OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-07 Thread John Hearns via users
Ben, ping me off list. I know the guy who heads the HPC Solutions Architect team for AWS and an AWS Solutions Architect here in the UK. On Fri, 7 Sep 2018 at 03:11, Benjamin Brock wrote: > > I'm setting up a cluster on AWS, which will have a 10Gb/s or 25Gb/s Ethernet > network. Should I expect

[OMPI users] RDMA over Ethernet in Open MPI - RoCE on AWS?

2018-09-06 Thread Benjamin Brock
I'm setting up a cluster on AWS, which will have a 10Gb/s or 25Gb/s Ethernet network. Should I expect to be able to get RoCE to work in Open MPI on AWS? More generally, what optimizations and performance tuning can I do to an Open MPI installation to get good performance on an Ethernet network?