Ralph Castain wrote:
On Apr 19, 2010, at 9:12 AM, Oliver Geisler wrote:
Ah, that could do it. Open MPI's shared memory files are under /tmp. So if /tmp is NFS, you could get extremely high latencies because of dirty page writes out through NFS.
You don't necessarily have to make /tmp disk-full -- if you just make OMPI's session directories go into a ramdisk instead of to NFS, that should also be sufficient.
I just browsed FAQ and "ompi_info --param all all", but didn't find the
answer:
How do I set the OMPI session directory to point it to a ramdisk?
Set the MCA param orte_tmpdir_base to point at the ramdisk using any of the MCA parameter methods (cmd line, envar, default mca param file).
I'll add that to http://www.open-mpi.org/faq/?category=sm#where-sm-file
And another question:
What would be a good size for the ram disk? One general value was
supposed by the FAQ with 128MB, but what is your experience?
(maybe a large topic by itself, so I have to try out, I guess)
I don't recall the default shared memory size per process, but you can get that value from ompi_info --param btl sm. Take that value, multiply by your expected ppn, and then give yourself a fudge factor.
Sizing proportionately to the number of processes was a poor heuristic
and starting in 1.3.2 we don't employ it any more. In all likelihood,
the default size of the shared-memory backing file will be set by
mpool_sm_min_size... 64 Mbytes. Try "ompi_info --param mpool sm".
There's some other stuff in addition to this backing file... so, need
a little fudge factor. Probably 128 MB will be enough for the
shared-memory stuff.
|
