Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-13 Thread Jackson, Gary L.
e: Friday, March 11, 2016 at 11:19 AM To: Open MPI Users <us...@open-mpi.org> Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP Gary, The current fine tuning of our TCP layer was done on a 1Gb network, and might result in the performance degradation you see. There is

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-11 Thread George Bosilca
lt;mailto:us...@open-mpi.org>> > Date: Tuesday, March 8, 2016 at 11:07 PM > To: Open MPI Users <us...@open-mpi.org <mailto:us...@open-mpi.org>> > Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP > > Jackson, > > one more thing, how did you build

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-11 Thread Gilles Gouaillardet
ardet <gil...@rist.or.jp > <javascript:_e(%7B%7D,'cvml','gil...@rist.or.jp');>> > Reply-To: Open MPI Users <us...@open-mpi.org > <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>> > Date: Tuesday, March 8, 2016 at 11:07 PM > To: Open MPI Users <us...@open-mp

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-10 Thread Jackson, Gary L.
: Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP Jackson, one more thing, how did you build openmpi ? if you built from git (and without VPATH), then --enable-debug is automatically set, and this is hurting p

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
Jackson, one more thing, how did you build openmpi ? if you built from git (and without VPATH), then --enable-debug is automatically set, and this is hurting performance. if not already done, i recommend you download the latest openmpi tarball (1.10.2) and ./configure

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Rayson Ho
If you are using instance types that support SR-IOV (aka. "enhanced networking" in AWS), then turn it on. We saw huge differences when SR-IOV is enabled http://blogs.scalablelogic.com/2013/12/enhanced-networking-in-aws-cloud.html

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
PI Users <us...@open-mpi.org> > Date: Tuesday, March 8, 2016 at 9:39 AM > To: Open MPI Users <us...@open-mpi.org> > Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP > > > Jason, > > how many Ethernet interfaces are there ? > if several, can

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Jackson, Gary L.
8, 2016 at 9:39 AM To: Open MPI Users <us...@open-mpi.org> Subject: Re: [OMPI users] Poor performance on Amazon EC2 with TCP Jason, how many Ethernet interfaces are there ? if several, can you try again with one only mpirun --mca btl_tcp_if_include eth0 ... Cheers, Gilles On Tuesday, March

Re: [OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Gilles Gouaillardet
Jason, how many Ethernet interfaces are there ? if several, can you try again with one only mpirun --mca btl_tcp_if_include eth0 ... Cheers, Gilles On Tuesday, March 8, 2016, Jackson, Gary L. wrote: > > I've built OpenMPI 1.10.1 on Amazon EC2. Using NetPIPE, I'm

[OMPI users] Poor performance on Amazon EC2 with TCP

2016-03-08 Thread Jackson, Gary L.
I've built OpenMPI 1.10.1 on Amazon EC2. Using NetPIPE, I'm seeing about half the performance for MPI over TCP as I do with raw TCP. Before I start digging in to this more deeply, does anyone know what might cause that? For what it's worth, I see the same issues with MPICH, but I do not see it