Hi Craig, Terry, Neeraj, list
Craig: A fellow here runs WRF.
I grep the code and there is plenty of collectives there:
MPI_[All]Gather[v], MPI_[All]Reduce, etc.
Domain decomposition code like WRF, MITgcm, and other atmosphere
and ocean codes has point-to-point communication to exchange
subdomain boundaries, but also collective operations to calculate
sums, etc, in various types of PDE (matrix) solvers that require
global information.
Terry: On the MITgcm, the apparent culprit is MPI_Allreduce,
which seems to be bad on **small** messages (rather than big ones).
This is the same behavior pattern that was reported here on May,
regarding MPI_Alltoall, by Roman Martonak, a list subscriber using a
computational chemistry package on an IB cluster:
http://www.open-mpi.org/community/lists/users/2009/07/10045.php
http://www.open-mpi.org/community/lists/users/2009/05/9419.php
At that point Pavel Shamis, Peter Kjellstrom, and others gave
very good suggestions, but they were only focused on MPI_Alltoall.
No other collectives were considered.
All: Any insights on how to tune MPI_Allreduce?
Maybe a hint on the other collectives also?
Any benchmark tool available that one can use to find the
sweet spot of each collective?
Many thanks,
Gus Correa
---------------------------------------------------------------------
Gustavo Correa
Lamont-Doherty Earth Observatory - Columbia University
Palisades, NY, 10964-8000 - USA
---------------------------------------------------------------------
nee...@crlindia.com wrote:
Hi Terry,
I had tested mostly MPI_Bcast, MPI_Reduce, MPI_Gather kind of
MPI with openmpi-1.3 and hierarchical option enabled.In all these, i
found results slower than regular tuned collectives.
We have HP Blade with intel clovertown processor(two quad core)
connected with DDR infiniband clos network.
Results were tested on 12-16 nodes with 8 mpi process each node.
Regards
Neeraj Chourasia (MTS)
Computational Research Laboratories Ltd.
(A wholly Owned Subsidiary of TATA SONS Ltd)
B-101, ICC Trade Towers, Senapati Bapat Road
Pune 411016 (Mah) INDIA
(O) +91-20-6620 9863 (Fax) +91-20-6620 9862
M: +91.9225520634
*Terry Dontje <terry.don...@sun.com>*
Sent by: users-boun...@open-mpi.org
08/07/2009 05:15 PM
Please respond to
Open MPI Users <us...@open-mpi.org>
To
us...@open-mpi.org
cc
Subject
Re: [OMPI users] Performance question about OpenMPI and MVAPICH2
on IB
Hi Neeraj,
Were there specific collectives that were slower? Also what kind of
cluster were you running on? How many nodes and cores per node?
thanks,
--td
> Message: 3
> Date: Fri, 7 Aug 2009 16:51:05 +0530
> From: nee...@crlindia.com
> Subject: Re: [OMPI users] Performance question about OpenMPI and
> MVAPICH2 on IB
> To: Open MPI Users <us...@open-mpi.org>
> Cc: us...@open-mpi.org, users-boun...@open-mpi.org
> Message-ID:
>
<of62a95e62.d6758124-on6525760b.003e2874-6525760b.003e1...@crlindia.com>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Terry,
>
> I feel hierarchical collectives are slower compare to tuned
one. I
> had done some benchmark in the past specific to collectives, and this is
> what i feel based on my observation.
>
> Regards
>
> Neeraj Chourasia (MTS)
> Computational Research Laboratories Ltd.
> (A wholly Owned Subsidiary of TATA SONS Ltd)
> B-101, ICC Trade Towers, Senapati Bapat Road
> Pune 411016 (Mah) INDIA
> (O) +91-20-6620 9863 (Fax) +91-20-6620 9862
> M: +91.9225520634
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
=====-----=====-----===== Notice: The information contained in this
e-mail message and/or attachments to it may contain confidential or
privileged information. If you are not the intended recipient, any
dissemination, use, review, distribution, printing or copying of the
information contained in this e-mail message and/or attachments to it
are strictly prohibited. If you have received this communication in
error, please notify us by reply e-mail or telephone and immediately and
permanently delete the message and any attachments. Internet
communications cannot be guaranteed to be timely, secure, error or
virus-free. The sender does not accept liability for any errors or
omissions.Thank you =====-----=====-----=====
------------------------------------------------------------------------
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
