On Jan 18, 2006, at 9:02 PM, Paul Greidanus wrote:
I've got a new cluster I'm deploying (with trunk ~4105), and I had
a question that came up:
My nodes are 4 and 8 core (8 and 32Gb RAM), and we were wondering
if LAM was compiled to take advantage of the interconnect
(hypertransport) between CPUs, or does it use the IP stack/network
to connect for cpus on the same machine?
LAM has less-than-optimal IB support, but it is functional. We have
not expressly tested on machines with hypertransport -- I actually
don't know if they export normal HCA interfaces (which LAM can
support) or whether a hypertransport-enabled MPI is required.
The LAM that ships with OSCAR does not include Infiniband support
because of the variety of IB libraries and versions available. If
hypertransport exports a VAPI-like interface, LAM can be compiled
from source to take advantage of it. However, if you're going to
compile from source, you might want to try Open MPI (Open MPI
supports VAPI and OpenIB) -- the IB support is much better than LAM's.
(an Open MPI OSCAR package is coming soon -- I swear it :-) -- but it
won't include IB support for the same reason that LAM's OSCAR package
does not include IB support: massive logistics issues)
Also, is it usually preferable to fill up a node, rather then
spread it out across the nodes? i.e. 16 cpu job over 16 nodes, or
2 8 cpu nodes?
It is usually preferable to fill up a node before spreading to a
second node because latencies between processors can be smaller
(e.g., via using shared memory, or even using the HCA's DMA engine).
--
{+} Jeff Squyres
{+} The Open MPI Project
{+} http://www.open-mpi.org/
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Oscar-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/oscar-devel
