Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-14 Thread Ralph Castain
I'm still poking around, but would appreciate a little more info to ensure I'm looking in the right places. How many nodes are you running your application across for your verification suite? I suspect it isn't just one :-) On Apr 10, 2014, at 9:19 PM, Ralph Castain wrote: > I shaved about 30

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-11 Thread Ralph Castain
I shaved about 30% off the time - the patch is waiting for 1.8.1, but you can try it now (see the ticket for the changeset): https://svn.open-mpi.org/trac/ompi/ticket/4510#comment:1 I've added you to the ticket so you can follow what I'm doing. Getting any further improvement will take a little

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Victor Vysotskiy
Hi again, > Okay, I'll try to do a little poking around. Meantime, please send along the > output from >"ompi_info" so we can see how this was configured and what built. enclosed please find the requested information. It would be great to have an workaround for 1.8 because with 1.8 our verific

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Ralph Castain
On Apr 10, 2014, at 7:58 AM, Victor Vysotskiy wrote: > Dear Ralph, > >> it appears that 1.8 is much faster than 1.6.5 with the default settings, but >> slower when you set btl=tcp,self? > > Precisely. However, with the default settings both versions are much slower > compared to other MPI

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Victor Vysotskiy
Dear Ralph, > it appears that 1.8 is much faster than 1.6.5 with the default settings, but > slower when you set btl=tcp,self? Precisely. However, with the default settings both versions are much slower compared to other MPI distributions such as MPICH, MVAPICH, and proprietary ones. The 'b

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Ralph Castain
Going on the assumption that it was a copy/paste error, the next question is: how many nodes were in your allocation? I ask because there is a change in the way we launch between 1.8 and 1.6.5. Starting in the 1.7 series, mpirun launches daemons across your allocation at startup so it can colle

Re: [OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Ralph Castain
Just to ensure I understand what you are saying: it appears that 1.8 is much faster than 1.6.5 with the default settings, but slower when you set btl=tcp,self? This seems rather strange. I note that the 1.8 value is identical in the two cases, but somehow 1.6.5 went much faster in the latter ca

[OMPI users] Performance issue of mpirun/mpi_init

2014-04-10 Thread Victor Vysotskiy
Dear Developers, I have faced a performance degradation on multi-core single processor machine. Specifically, in the most recent Open MPI v1.8 the initialization and process startup stage became ~10x slower compared to v1.6.5. In order to measure timings I have used the following code snippet