Re: [OMPI users] alltoallv

[Peter Kjellström]

On Tue, 10 Oct 2017 11:57:51 -0400
Michael Di Domenico  wrote:

> i'm getting stuck trying to run some fairly large IMB-MPI alltoall
> tests under openmpi 2.0.2 on rhel 7.4

What is the IB stack used, just RHEL inbox?

Do you run openmpi on the psm mtl for qlogic and openib btl for
mellanox or something different?

> i have two different clusters, one running mellanox fdr10 and one
> running qlogic qdr
> 
> if i issue
> 
> mpirun -n 1024 ./IMB-MPI1 -npmin 1024 -iter 1 -mem 2.001 alltoallv

Does it work if you run with something that more obviously fits in RAM?
Like "-mem 0.2"

/Peter K
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] alltoallv

[Michael Di Domenico]

i'm getting stuck trying to run some fairly large IMB-MPI alltoall
tests under openmpi 2.0.2 on rhel 7.4

i have two different clusters, one running mellanox fdr10 and one
running qlogic qdr

if i issue

mpirun -n 1024 ./IMB-MPI1 -npmin 1024 -iter 1 -mem 2.001 alltoallv

the job just stalls after the "List of Benchmarks to run: Alltoallv"
line outputs from IMB-MPI

if i switch it to alltoall the test does progress

often when running various size alltoall's i'll get

"too many retries sending message to <>:<>, giving up

i'm able to use infiniband just fine (our lustre filesystem mounts
over it) and i have other mpi programs running

it only seems to stem when i run alltoall type primitives

any thoughts on debugging where the failures are, i might just need to
turn up the debugging, but i'm not sure where
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] AlltoallV (with some zero send count values)

[Shamis, Pavel]

> 
> Can anyone tell me whether it is legal to pass zero values for some of the 
> send count elements in an MPI_AlltoallV() call? I want to perform an 
> all-to-all operation but for performance reasons do not want to send data to 
> various ranks who don't need to receive any useful values. If it is legal, 
> can I assume the implementation is smart enough to not send messages when the 
> send count is 0?

Well, if the message size is small I would suggest you to use alltoall instead 
of alltoallv. If the message size is large, alltoallV probably will be little 
bit faster.

FYI the implementation sends a message for count 0.

Regards,
Pasha

Re: [OMPI users] AlltoallV (with some zero send count values)

[Jeffrey Squyres]

On Mar 6, 2012, at 4:43 PM, Timothy Stitt wrote:

> Can anyone tell me whether it is legal to pass zero values for some of the 
> send count elements in an MPI_AlltoallV() call? I want to perform an 
> all-to-all operation but for performance reasons do not want to send data to 
> various ranks who don't need to receive any useful values. If it is legal, 
> can I assume the implementation is smart enough to not send messages when the 
> send count is 0?
> 
> FYI: my tests show that AlltoallV operations with various send count values 
> set to 0...hangs.

Ouch.  Could you send a small reproducer?

Per your other question:

> How well-implemented are the one-sided communication routines? Are they 
> something that could be trusted in a production code? 

Meh.  They work, but I don't know if they've been strenuously tested.  At least 
one or two users have reported problems with them in some scenarios (for 
not-well-defined "some scenarios").  Personally, I'd wait for the re-write that 
is coming for the new MPI-3 collectives.  The definitions for RMA in MPI-3 are 
*much* better than they were in MPI-2, and I know Brian is working hard on the 
new OMPI MPI-3 RMA implementation.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] AlltoallV (with some zero send count values)

[Timothy Stitt]

Thanks Jed for the advice. How well-implemented are the one-sided communication 
routines? Are they something that could be trusted in a production code?

Sent from my iPad

On Mar 6, 2012, at 6:06 PM, "Jed Brown" mailto:j...@59a2.org>> 
wrote:

On Tue, Mar 6, 2012 at 15:43, Timothy Stitt 
mailto:timothy.stit...@nd.edu>> wrote:
Can anyone tell me whether it is legal to pass zero values for some of the send 
count elements in an MPI_AlltoallV() call? I want to perform an all-to-all 
operation but for performance reasons do not want to send data to various ranks 
who don't need to receive any useful values. If it is legal, can I assume the 
implementation is smart enough to not send messages when the send count is 0?

FYI: my tests show that AlltoallV operations with various send count values set 
to 0...hangs.

This is allowed by the standard, but be warned that it is likely to perform 
poorly compared to what could be done with point-to-point or one-sided 
operations if most links are empty.
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] AlltoallV (with some zero send count values)

[Jed Brown]

On Tue, Mar 6, 2012 at 15:43, Timothy Stitt  wrote:

> Can anyone tell me whether it is legal to pass zero values for some of the
> send count elements in an MPI_AlltoallV() call? I want to perform an
> all-to-all operation but for performance reasons do not want to send data
> to various ranks who don't need to receive any useful values. If it is
> legal, can I assume the implementation is smart enough to not send messages
> when the send count is 0?
>
> FYI: my tests show that AlltoallV operations with various send count
> values set to 0...hangs.
>

This is allowed by the standard, but be warned that it is likely to perform
poorly compared to what could be done with point-to-point or one-sided
operations if most links are empty.

[OMPI users] AlltoallV (with some zero send count values)

[Timothy Stitt]

Hi all,

Can anyone tell me whether it is legal to pass zero values for some of the send 
count elements in an MPI_AlltoallV() call? I want to perform an all-to-all 
operation but for performance reasons do not want to send data to various ranks 
who don't need to receive any useful values. If it is legal, can I assume the 
implementation is smart enough to not send messages when the send count is 0?

FYI: my tests show that AlltoallV operations with various send count values set 
to 0...hangs.

Thanks,

Tim.