Hi Chris,
Try setting OMPI_MCA_orte_tmpdir_base.
Going back to a related earlier OMPI users thread ("How to set up state-less node /tmp for OpenMPI usage"), here are sm pingpong latencies (using 1.4.3) for
session dirs on Lustre, an SSD and tmpfs:
[dbs900@v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/short/z00/dbs900/tmp
[dbs900@v1490 ~/MPI]$ mpirun -n 2 ./pp143
| Best | Average | Worst
| Streaming
n (test/loop) | usec MB/s | usec MB/s | usec
MB/s | ( average, worst)
1 ( 64/1100) | 0.3 30.45 | 0.3 29.63 | 0.3
26.00 | ( 0.3, 0.3)
2 ( 64/1100) | 0.3 60.29 | 0.3 59.11 | 0.3
50.86 | ( 0.3, 0.3)
4 ( 64/1100) | 0.3 117.92 | 0.3 115.01 | 0.3
96.18 | ( 0.3, 0.3)
...
[dbs900@v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/jobfs/flash/tmp
[dbs900@v1490 ~/MPI]$ mpirun -n 2 ./pp143
| Best | Average | Worst
| Streaming
n (test/loop) | usec MB/s | usec MB/s | usec
MB/s | ( average, worst)
1 ( 64/1100) | 0.3 30.24 | 0.3 29.54 | 0.3
24.86 | ( 0.3, 0.3)
2 ( 64/1100) | 0.3 60.09 | 0.3 58.91 | 0.3
50.51 | ( 0.3, 0.3)
4 ( 64/1100) | 0.3 117.92 | 0.3 115.39 | 0.3
99.55 | ( 0.3, 0.3)
...
[dbs900@v1490 ~/MPI]$ export OMPI_MCA_orte_tmpdir_base=/tmp
[dbs900@v1490 ~/MPI]$ mpirun -n 2 ./pp143
| Best | Average | Worst
| Streaming
n (test/loop) | usec MB/s | usec MB/s | usec
MB/s | ( average, worst)
1 ( 64/1100) | 0.3 30.34 | 0.3 29.62 | 0.3
24.96 | ( 0.3, 0.3)
2 ( 64/1100) | 0.3 60.58 | 0.3 59.30 | 0.3
51.39 | ( 0.3, 0.3)
4 ( 64/1100) | 0.3 116.94 | 0.3 114.69 | 0.3
99.32 | ( 0.3, 0.3)
...
This is why I was hoping sites could be given the opportunity to control that
warning message. Some Lustre sites may not want their users getting that
message.
David
On 12/13/2011 05:10 PM, Christopher Samuel wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/12/11 09:13, Jeff Squyres wrote:
Gearing up for a pre-Christmas release -- please test!
Had a quick test here (we're still on 1.4.2 as our default).
It now complains that $TMPDIR points to our Panasas filesystem,
and there's a minor typo in its message, saying:
# Note, too, that system administrators can set a list of filesystems
# where Open MPI is disallowed from creating temporary files by settings
# the MCA parameter "orte_no_session_dir".
I think you want s/settings/setting/ there.
Also I can not seem to make it accept the orte_no_session_dir settings,
neither:
mpirun -mca orte_no_session_dir /vlsci ./hello_c
nor:
mpirun -mca orte_no_session_dir /vlsci/tmp ./hello_c
nor:
$ export OMPI_MCA_orte_no_session_dir=/vlsci
nor:
export OMPI_MCA_orte_no_session_dir=/vlsci/tmp
prevent it whinging about (for example):
Local host: bruce001
File Name:
/vlsci/tmp/979325.bruce-m.vlsci.unimelb.edu.au/openmpi-sessions-samuel@bruce001_0/14488/1/shared_mem_pool.bruce001
Is there a way to tell it to use /tmp without changing what
$TMPDIR is set to ?
cheers,
Chris
