On Mar 21, 2011, at 11:59 AM, Jeff Squyres wrote:
> I no longer run Torque on my cluster, so my Torqueology is pretty rusty --
> but I think there's a Torque command to launch on remote nodes. tmrsh or
> pbsrsh or something like that...?
pbsrsh, IIRC
So run pbsrsh <hostname> printenv to see the environment on a remote node. Etc.
>
> Try that and make sure it works. Open MPI should be using the same API as
> that command under the covers.
>
> I also have a dim recollection that the TM API support library(ies?) may not
> be installed by default. You may have to ensure that they're available on
> all nodes...?
This is true - usually not installed by default, and need to be available on
all nodes since Torque starts mpiexec on a backend node.
>
>
> On Mar 21, 2011, at 1:53 PM, Randall Svancara wrote:
>
>> I am not sure if there is any extra configuration necessary for torque
>> to forward the environment. I have included the output of printenv
>> for an interactive qsub session. I am really at a loss here because I
>> never had this much difficulty making torque run with openmpi. It has
>> been mostly a good experience.
>>
>> Permissions of /tmp
>>
>> drwxrwxrwt 4 root root 140 Mar 20 08:57 tmp
>>
>> mpiexec hostname single node:
>>
>> [rsvancara@login1 ~]$ qsub -I -lnodes=1:ppn=12
>> qsub: waiting for job 1667.mgt1.wsuhpc.edu to start
>> qsub: job 1667.mgt1.wsuhpc.edu ready
>>
>> [rsvancara@node100 ~]$ mpiexec hostname
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>> node100
>>
>> mpiexec hostname two nodes:
>>
>> [rsvancara@node100 ~]$ mpiexec hostname
>> [node100:09342] plm:tm: failed to poll for a spawned daemon, return
>> status = 17002
>> --------------------------------------------------------------------------
>> A daemon (pid unknown) died unexpectedly on signal 1 while attempting to
>> launch so we are aborting.
>>
>> There may be more information reported by the environment (see above).
>>
>> This may be because the daemon was unable to find all the needed shared
>> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
>> location of the shared libraries on the remote nodes and this will
>> automatically be forwarded to the remote nodes.
>> --------------------------------------------------------------------------
>> --------------------------------------------------------------------------
>> mpiexec noticed that the job aborted, but has no info as to the process
>> that caused that situation.
>> --------------------------------------------------------------------------
>> --------------------------------------------------------------------------
>> mpiexec was unable to cleanly terminate the daemons on the nodes shown
>> below. Additional manual cleanup may be required - please refer to
>> the "orte-clean" tool for assistance.
>> --------------------------------------------------------------------------
>> node99 - daemon did not report back when launched
>>
>>
>> MPIexec on one node with one cpu:
>>
>> [rsvancara@node164 ~]$ mpiexec printenv
>> OMPI_MCA_orte_precondition_transports=5fbd0d3c8e4195f1-80f964226d1575ea
>> MODULE_VERSION_STACK=3.2.8
>> MANPATH=/home/software/mpi/intel/openmpi-1.4.3/share/man:/home/software/intel/Compiler/11.1/075/man/en_US:/home/software/intel/Compiler/11.1/075/mkl/man/en_US:/home/software/intel/Compiler/11.1/075/mkl/../man/en_US:/home/software/Modules/3.2.8/share/man:/usr/share/man
>> HOSTNAME=node164
>> PBS_VERSION=TORQUE-2.4.7
>> TERM=xterm
>> SHELL=/bin/bash
>> HISTSIZE=1000
>> PBS_JOBNAME=STDIN
>> PBS_ENVIRONMENT=PBS_INTERACTIVE
>> PBS_O_WORKDIR=/home/admins/rsvancara
>> PBS_TASKNUM=1
>> USER=rsvancara
>> LD_LIBRARY_PATH=/home/software/mpi/intel/openmpi-1.4.3/lib:/home/software/intel/Compiler/11.1/075/lib/intel64:/home/software/intel/Compiler/11.1/075/ipp/em64t/sharedlib:/home/software/intel/Compiler/11.1/075/mkl/lib/em64t:/home/software/intel/Compiler/11.1/075/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/home/software/intel/Compiler/11.1/075/lib
>> LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:
>> PBS_O_HOME=/home/admins/rsvancara
>> CPATH=/home/software/intel/Compiler/11.1/075/ipp/em64t/include:/home/software/intel/Compiler/11.1/075/mkl/include:/home/software/intel/Compiler/11.1/075/tbb/include
>> PBS_MOMPORT=15003
>> PBS_O_QUEUE=batch
>> NLSPATH=/home/software/intel/Compiler/11.1/075/lib/intel64/locale/%l_%t/%N:/home/software/intel/Compiler/11.1/075/ipp/em64t/lib/locale/%l_%t/%N:/home/software/intel/Compiler/11.1/075/mkl/lib/em64t/locale/%l_%t/%N:/home/software/intel/Compiler/11.1/075/idb/intel64/locale/%l_%t/%N
>> MODULE_VERSION=3.2.8
>> MAIL=/var/spool/mail/rsvancara
>> PBS_O_LOGNAME=rsvancara
>> PATH=/home/software/mpi/intel/openmpi-1.4.3/bin:/home/software/intel/Compiler/11.1/075/bin/intel64:/home/software/Modules/3.2.8/bin:/bin:/usr/bin:/usr/lpp/mmfs/bin
>> PBS_O_LANG=en_US.UTF-8
>> PBS_JOBCOOKIE=D52DE562B685A462849C1136D6B581F9
>> INPUTRC=/etc/inputrc
>> PWD=/home/admins/rsvancara
>> _LMFILES_=/home/software/Modules/3.2.8/modulefiles/modules:/home/software/Modules/3.2.8/modulefiles/null:/home/software/modulefiles/intel/11.1.075:/home/software/modulefiles/openmpi/1.4.3_intel
>> PBS_NODENUM=0
>> LANG=C
>> MODULEPATH=/home/software/Modules/versions:/home/software/Modules/$MODULE_VERSION/modulefiles:/home/software/modulefiles
>> LOADEDMODULES=modules:null:intel/11.1.075:openmpi/1.4.3_intel
>> PBS_O_SHELL=/bin/bash
>> PBS_SERVER=mgt1.wsuhpc.edu
>> PBS_JOBID=1670.mgt1.wsuhpc.edu
>> SHLVL=1
>> HOME=/home/admins/rsvancara
>> INTEL_LICENSES=/home/software/intel/Compiler/11.1/075/licenses:/opt/intel/licenses
>> PBS_O_HOST=login1
>> DYLD_LIBRARY_PATH=/home/software/intel/Compiler/11.1/075/tbb/intel64/cc4.1.0_libc2.4_kernel2.6.16.21/lib
>> PBS_VNODENUM=0
>> LOGNAME=rsvancara
>> PBS_QUEUE=batch
>> MODULESHOME=/home/software/mpi/intel/openmpi-1.4.3
>> LESSOPEN=|/usr/bin/lesspipe.sh %s
>> PBS_O_MAIL=/var/spool/mail/rsvancara
>> G_BROKEN_FILENAMES=1
>> PBS_NODEFILE=/var/spool/torque/aux//1670.mgt1.wsuhpc.edu
>> PBS_O_PATH=/home/software/mpi/intel/openmpi-1.4.3/bin:/home/software/intel/Compiler/11.1/075/bin/intel64:/home/software/Modules/3.2.8/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/lpp/mmfs/bin
>> module=() { eval `/home/software/Modules/$MODULE_VERSION/bin/modulecmd bash
>> $*`
>> }
>> _=/home/software/mpi/intel/openmpi-1.4.3/bin/mpiexec
>> OMPI_MCA_orte_local_daemon_uri=3236233216.0;tcp://172.20.102.82:33559;tcp://172.40.102.82:33559
>> OMPI_MCA_orte_hnp_uri=3236233216.0;tcp://172.20.102.82:33559;tcp://172.40.102.82:33559
>> OMPI_MCA_mpi_yield_when_idle=0
>> OMPI_MCA_orte_app_num=0
>> OMPI_UNIVERSE_SIZE=1
>> OMPI_MCA_ess=env
>> OMPI_MCA_orte_ess_num_procs=1
>> OMPI_COMM_WORLD_SIZE=1
>> OMPI_COMM_WORLD_LOCAL_SIZE=1
>> OMPI_MCA_orte_ess_jobid=3236233217
>> OMPI_MCA_orte_ess_vpid=0
>> OMPI_COMM_WORLD_RANK=0
>> OMPI_COMM_WORLD_LOCAL_RANK=0
>> OPAL_OUTPUT_STDERR_FD=19
>>
>> MPIExec with -mca plm rsh:
>>
>> [rsvancara@node164 ~]$ mpiexec -mca plm rsh -mca orte_tmpdir_base
>> /fastscratch/admins/tmp hostname
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node164
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>> node163
>>
>>
>> On Mon, Mar 21, 2011 at 9:22 AM, Ralph Castain <r...@open-mpi.org> wrote:
>>> Can you run anything under TM? Try running "hostname" directly from Torque
>>> to see if anything works at all.
>>>
>>> The error message is telling you that the Torque daemon on the remote node
>>> reported a failure when trying to launch the OMPI daemon. Could be that
>>> Torque isn't setup to forward environments so the OMPI daemon isn't finding
>>> required libs. You could directly run "printenv" to see how your remote
>>> environ is being setup.
>>>
>>> Could be that the tmp dir lacks correct permissions for a user to create
>>> the required directories. The OMPI daemon tries to create a session
>>> directory in the tmp dir, so failure to do so would indeed cause the launch
>>> to fail. You can specify the tmp dir with a cmd line option to mpirun. See
>>> "mpirun -h" for info.
>>>
>>>
>>> On Mar 21, 2011, at 12:21 AM, Randall Svancara wrote:
>>>
>>>> I have a question about using OpenMPI and Torque on stateless nodes.
>>>> I have compiled openmpi 1.4.3 with --with-tm=/usr/local
>>>> --without-slurm using intel compiler version 11.1.075.
>>>>
>>>> When I run a simple "hello world" mpi program, I am receiving the
>>>> following error.
>>>>
>>>> [node164:11193] plm:tm: failed to poll for a spawned daemon, return
>>>> status = 17002
>>>> --------------------------------------------------------------------------
>>>> A daemon (pid unknown) died unexpectedly on signal 1 while attempting to
>>>> launch so we are aborting.
>>>>
>>>> There may be more information reported by the environment (see above).
>>>>
>>>> This may be because the daemon was unable to find all the needed shared
>>>> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
>>>> location of the shared libraries on the remote nodes and this will
>>>> automatically be forwarded to the remote nodes.
>>>> --------------------------------------------------------------------------
>>>> --------------------------------------------------------------------------
>>>> mpiexec noticed that the job aborted, but has no info as to the process
>>>> that caused that situation.
>>>> --------------------------------------------------------------------------
>>>> --------------------------------------------------------------------------
>>>> mpiexec was unable to cleanly terminate the daemons on the nodes shown
>>>> below. Additional manual cleanup may be required - please refer to
>>>> the "orte-clean" tool for assistance.
>>>> --------------------------------------------------------------------------
>>>> node163 - daemon did not report back when launched
>>>> node159 - daemon did not report back when launched
>>>> node158 - daemon did not report back when launched
>>>> node157 - daemon did not report back when launched
>>>> node156 - daemon did not report back when launched
>>>> node155 - daemon did not report back when launched
>>>> node154 - daemon did not report back when launched
>>>> node152 - daemon did not report back when launched
>>>> node151 - daemon did not report back when launched
>>>> node150 - daemon did not report back when launched
>>>> node149 - daemon did not report back when launched
>>>>
>>>>
>>>> But if I include:
>>>>
>>>> -mca plm rsh
>>>>
>>>> The job runs just fine.
>>>>
>>>> I am not sure what the problem is with torque or openmpi that prevents
>>>> the process from launching on remote nodes. I have posted to the
>>>> torque list and someone suggested that it may be temporary directory
>>>> space that can be causing issues. I have 100MB allocated to /tmp
>>>>
>>>> Any ideas as to why I am having this problem would be appreciated.
>>>>
>>>>
>>>> --
>>>> Randall Svancara
>>>> http://knowyourlinux.com/
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>>
>>
>> --
>> Randall Svancara
>> http://knowyourlinux.com/
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
