On 8/11/10 3:15 PM, "Samuel K. Gutierrez" <sam...@lanl.gov> wrote:
Hi Terry,
On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:
I've done some minor testing on Linux looking at resident and
shared memory sizes for np=4, 8 and 16 jobs. I could not see any
appreciable differences in sizes in the process between sysv,
posix or mmap usage in the SM btl.
So I am still somewhat non-plussed about making this the
default. It seems like the biggest gain out of using posix might
be one doesn't need to worry about the location of the backing
file. This seems kind of frivolous to me since there is a
warning that happens if the backing file is not memory based.
If I'm not mistaken, the warning is only issued if the backing
files is stored on the following file systems: Lustre, NFS,
Panasas, and GPFS (see: opal_path_nfs in opal/util/path.c).
Based on the performance numbers that Sylvain provided on June
9th of this year (see attached), there was a performance
difference between mmap on /tmp and mmap on a tmpfs-like file
system (/dev/shm in that particular case). Using the new POSIX
component provides us with a portable way to provide similar
shared memory performance gains without having to worry about
where the OMPI session directory is rooted.
--
Samuel K. Gutierrez
Los Alamos National Laboratory
[cid:3364459484_11867134]
I still working on testing the code on Solaris but I don't
imagine I will see anything that will change my mind.
--td
Samuel K. Gutierrez wrote:
Hi Rich,
It's a modification to the existing common sm component. The
modifications do include the addition of a new POSIX shared
memory facility, however.
Sam
On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:
Is this a modification of the existing component, or a new
component ?
Rich
On 8/10/10 10:52 AM, "Samuel K. Gutierrez" <sam...@lanl.gov> <mailto:sam...@lanl.gov
> wrote:
Hi,
I wanted to give everyone a heads-up about a new POSIX shared
memory
component
that has been in the works for a while now and is ready to be
pushed
into the
trunk.
http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new
Some highlights:
o New posix component now the new default.
o May address some of the shared memory performance
issues users
encounter
when OMPI's session directories are inadvertently
placed on a non-
local
filesystem.
o Silent component failover.
o In the default case, if the posix component fails
initialization,
mmap will be selected.
o The sysv component will only be queried for selection if it is
placed before
the mmap component (for example, -mca mpi_common_sm
sysv,posix,mmap). In the
default case, sysv will never be queried/selected.
o Per some on-list discussion, now unlinking mmaped file in both
mmap
and posix
components (see: "System V Shared Memory for Open MPI:
Request for
Community
Input and Testing" thread).
o Assuming local process homogeneity with respect to all utilized
shared
memory facilities. That is, if one local process deems a
particular shared
memory facility acceptable, then ALL local processes should be
able to
utilize that facility. As it stands, this is an important
point
because one
process dictates to all other local processes which common sm
component will
be selected based on its own, local run-time test.
o Addressed some of George's code reuse concerns.
If there are no major objections by August 17th, I'll commit the
code
after the
Tuesday morning conference call.
Thanks!
--
Samuel K. Gutierrez
Los Alamos National Laboratory
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
