Don,
As I understand it, you're saying that we'll get higher performance if
we have each 2-CPU node run a single process which does our 6 units of
work (the 6 stageslices). I've tested this in the debug queue, and
having each node run two processes, each of which does 3 units of work,
is demonstrably faster, at least in my experience.
Are you hinting that we'd do better if we simply increased the number of
nodes and only had a single process on each, with one core handling
computation and one handling MPI? This seems to be the case - we run
faster than either of the above cases if we only have 3 stageslices per
node and only one process, but it deviates from the design we have
specified.
How did you discover this interesting behavior you describe in MPI?
-Jon
Don Dossa wrote:
Jon
You said you had twice as many MPI processes as CPUs running on each
node.
The number of MPI processes per node is very dependent on the
application.
If the app is mostly CPU bound, running 2x processes per node probably
doesnt help your performance. If it's doing lots of IO, then
multiple processes per node can be ok. Remove every MPI_Barrier
that you can. If your app is doing lots of communicating between
MPI processes, you can get a nice speed-up by running less
processes per node and let the free processor handle the MPI
communication.
If you have just 2 CPUs per node, these hints are less useful.
--Don
On Oct 5, 2006, at 11:06 AM, [EMAIL PROTECTED] wrote:
----------------------------------------------------------------------
Message: 1
Date: Wed, 4 Oct 2006 16:08:45 -0500 (CDT)
From: Ray Plante <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Using both cores of our nodes
To: LSST Data Management <[email protected]>
Message-ID:
<[EMAIL PROTECTED]>
Content-Type: TEXT/PLAIN; charset=US-ASCII
Hey Jon,
On Tue, 3 Oct 2006, Jonathan Myers wrote:
I've committed r.557 into subversion, which is a simple modification to
SetupHarnessPipeline.py. It lists every element of PBS_NODEFILE twice
in our nodelist, which actually doubles our MPI universe size and
doubles the number of stageslices on each node.
One more question so that I understand the results...Did you test
just one
image or several? If it was the latter, did you have multiple images in
the pipeline at the same time?
Also, when you get a chance, could you explain a little more about
how you
calculated flops for the different stages. If it's straight-forward, it
would be could to capture this as part of our regular statistics.
thanks!
Ray
------------------------------
Message: 2
Date: Wed, 04 Oct 2006 14:55:02 -0700
From: Jonathan Myers <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] Using both cores of our nodes
To: LSST Data Management <[email protected]>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Ray (and others),
I tested on several images. I've tried it with the r. 556 stageleader
(only processes one image at a time) and the older version (several
images in the pipeline at once).
For future reference, the best latency time I've gotten is about 93
seconds per image, which is using the new stageleader (one image in the
pipeline at a time) and the new SetupHarnessPipeline.py (twice as many
MPI nodes as physical nodes, one stageslice per core). I tested got
about 12 images through before walltime shut me down.
Hope this clarifies things,
-Jon
Ray Plante wrote:
Hey Jon,
On Tue, 3 Oct 2006, Jonathan Myers wrote:
I've committed r.557 into subversion, which is a simple
modification to
SetupHarnessPipeline.py. It lists every element of PBS_NODEFILE twice
in our nodelist, which actually doubles our MPI universe size and
doubles the number of stageslices on each node.
One more question so that I understand the results...Did you test
just one
image or several? If it was the latter, did you have multiple
images in
the pipeline at the same time?
Also, when you get a chance, could you explain a little more about
how you
calculated flops for the different stages. If it's
straight-forward, it
would be could to capture this as part of our regular statistics.
thanks!
Ray
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
