Jon
You said you had twice as many MPI processes as CPUs running on each
node.
The number of MPI processes per node is very dependent on the
application.
If the app is mostly CPU bound, running 2x processes per node probably
doesnt help your performance. If it's doing lots of IO, then
multiple processes per node can be ok. Remove every MPI_Barrier
that you can. If your app is doing lots of communicating between
MPI processes, you can get a nice speed-up by running less
processes per node and let the free processor handle the MPI
communication.
If you have just 2 CPUs per node, these hints are less useful.
--Don
On Oct 5, 2006, at 11:06 AM, [EMAIL PROTECTED] wrote:
----------------------------------------------------------------------
Message: 1
Date: Wed, 4 Oct 2006 16:08:45 -0500 (CDT)
From: Ray Plante <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Using both cores of our nodes
To: LSST Data Management <[email protected]>
Message-ID:
<[EMAIL PROTECTED]>
Content-Type: TEXT/PLAIN; charset=US-ASCII
Hey Jon,
On Tue, 3 Oct 2006, Jonathan Myers wrote:
I've committed r.557 into subversion, which is a simple
modification to
SetupHarnessPipeline.py. It lists every element of PBS_NODEFILE
twice
in our nodelist, which actually doubles our MPI universe size and
doubles the number of stageslices on each node.
One more question so that I understand the results...Did you test
just one
image or several? If it was the latter, did you have multiple
images in
the pipeline at the same time?
Also, when you get a chance, could you explain a little more about
how you
calculated flops for the different stages. If it's straight-
forward, it
would be could to capture this as part of our regular statistics.
thanks!
Ray
------------------------------
Message: 2
Date: Wed, 04 Oct 2006 14:55:02 -0700
From: Jonathan Myers <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Using both cores of our nodes
To: LSST Data Management <[email protected]>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Ray (and others),
I tested on several images. I've tried it with the r. 556 stageleader
(only processes one image at a time) and the older version (several
images in the pipeline at once).
For future reference, the best latency time I've gotten is about 93
seconds per image, which is using the new stageleader (one image in
the
pipeline at a time) and the new SetupHarnessPipeline.py (twice as many
MPI nodes as physical nodes, one stageslice per core). I tested got
about 12 images through before walltime shut me down.
Hope this clarifies things,
-Jon
Ray Plante wrote:
Hey Jon,
On Tue, 3 Oct 2006, Jonathan Myers wrote:
I've committed r.557 into subversion, which is a simple
modification to
SetupHarnessPipeline.py. It lists every element of PBS_NODEFILE
twice
in our nodelist, which actually doubles our MPI universe size and
doubles the number of stageslices on each node.
One more question so that I understand the results...Did you test
just one
image or several? If it was the latter, did you have multiple
images in
the pipeline at the same time?
Also, when you get a chance, could you explain a little more about
how you
calculated flops for the different stages. If it's straight-
forward, it
would be could to capture this as part of our regular statistics.
thanks!
Ray
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
