Hi,
Is the behaviour of --export correct (seen both with 15.08.0 and 14.11.6)?
If you don't use --export, the complete environment is exported:
$ srun -n 1 env | grep "\(HOME\|DISPLAY\)"
...
HOME=/home/users/...
DISPLAY=localhost:20.0
...
However, when you use --export only the environment variables listed are
exported:
$ srun -n 1 --export MY_HOME=IS_MY_CASTLE env | grep "\(HOME\|DISPLAY\)"
slurmstepd: execve(): env: No such file or directory
Not even PATH is exported, so the above command will fail. You have to do:
$ srun -n 1 --export MY_HOME=IS_MY_CASTLE,PATH env | grep -v SLURM
MY_HOME=IS_MY_CASTLE
PATH=/opt/slurm/15.08.0/bin:...
TMPDIR=/tmp
...
This works, but beside the explicitly --exported MY_HOME,PATH (and lots
of SLURM_INTERNAL_VARS) nothing from the original environment is exported.
Now, this behaviour is described in the man page:
--export=<environment variables | NONE>
Identify which environment variables are propagated to the launched
application.
But it also states:
... or specific values for the variables may be exported (e.g..
"--export=EDITOR=/bin/vi") *in addition* to the environment variables
that would otherwise be set.
So what is the right behaviour?
I don't consider the current behaviour good behaviour from the user's
point, as the above example (--export=EDITOR=/bin/vi) will "unset" PATH
for example and the user will ask why does my srun command fail now?
Another example where "--export=EDITOR=/bin/vi" will break things, is
when using Intel MPI with SLURM/srun: e.g. the system administrator has
setup a module file for Intel MPI with "export
I_MPI_PMI_LIBRARY=/path/to/slurm/pmi/library/libpmi.so".
When you do "srun -n 2 --export=EDITOR=/bin/vi ./my/bin" you will get this:
srun: job 12345 has been allocated resources
Unknown option: pmi_args
Unknown option: pmi_args
...
The user will ask what is this strange "Unknown option: pmi_args" error
and where is this coming from. You explicitly need to export I_MPI_*
variables to make it work again.
*A bit background why I am asking / found this*
I work on Allinea Forge (parallel debugger & profiler for MPI with
support for SLURM) and we need to preload certain libraries with the
user's binary (e.g. our dmalloc to find memory errors or a profiler
sampler). To do this we use --export=LD_PRELOAD=... which then failed
sometimes with above errors.
(We can not set LD_PRELOAD in the environment before starting srun, as
LD_PRELOAD would then also apply to srun example which we don't want).
Our current workaround is to explicitly build a complete env var list,
e.g --export=...,HOME,PATH,LD_PRELOAD=libdmalloc.so,... to make it work.
However, with the risk of producing a potentially huge command line
which might not always be handled gracefully.
It would really be helpful to modify the behaviour of --export to export
in addition to the current environment or add another option to srun
with that behaviour.
Kind regards,
Dirk
--
Dirk Schubert - Lead Software Developer || Allinea Software
