Hi Jacob --
I was hoping someone else would beat me to this because I don't have much
experience with slurm or GASNet over ibv, but just to keep this thread
from stalling out:
I'm fairly certain that the message you're getting here is from GASNet's
gasnetrun_ibv utility which is documented in
$CHPL_HOME/third-party/gasnet/gasnet-*/ibv-conduit/README:
https://github.com/chapel-lang/chapel/blob/master/third-party/gasnet/gasnet-src/ibv-conduit/README#L34
I'm hoping that knowing this may help you get unstuck, but perhaps someone
with more srun+gasnet/ibv experience will be able to chime in as well.
Here at Cray, we're working on getting some nightly testing over this
combination up and running, so I'll check whether anyone involved has any
further tips.
Best wishes,
-Brad
On Mon, 1 Aug 2016, Caswell, Jacob wrote:
Hello all,
Thank you very much for all of your help! I'm no longer getting the mpi-linking
issues. Also thank you for letting me know that Chapel passes through those
environment variables.
Also, Michael (in case it's helpful), it looks like openMPI version 1.8 is the
latest version installed on the machine I was using.
However, now I am finding that if I try to run my compiled program, it is
requiring that SSH_SERVERS is missing. Is there any way to use the chapel
launcher without specifying the SSH_SERVERS? I've tried using both
CHPL_COMM="gasnet"
CHPL_LAUNCHER="slurm-srun"
And
CHPL_COMM="gasnet"
CHPL_LAUNCHER="slurm-gasnetrun-ibv"
But both times that I've tried to run this using sbatch or salloc / srun, I get:
*** FATAL ERROR: Error spawning SPMD worker threads. Exiting...
Environment variable SSH_SERVERS is missing.
/var/spool/slurmd/job1035561/slurm_script: line 9: 99690 Aborted
Again, thank you all for your help,
Jake
-----Original Message-----
From: Michael Ferguson [mailto:[email protected]]
Sent: Thursday, July 28, 2016 7:16 AM
To: Caswell, Jacob <[email protected]>
Cc: [email protected]
Subject: [EXTERNAL] Re: [Chapel-developers] Issues setting up Chapel for
multilocale use
Hi -
On 7/27/16, 10:14 AM, "Caswell, Jacob" <[email protected]> wrote:
Hello all,
I’m quite new to Chapel, so I apologize if I’m sending this email to
the wrong mailing lists.
Welcome. Anyway, I'd expect questions like this to go to chapel-users or to
chapel-bugs.
This question being on chapel-users will allow others who might have experience
with Chapel over IBV to share.
I’ve been having some issues setting up Chapel for multilocale use on
some of our machines here. I’ve been trying to follow the Multilocale
and Launcher pages in the documentation, however, I still haven’t been
successful. I think one of the issues might be that, on some of our
machines, slurm requires an account variable, which may be keeping me
from using native slurm. Is there an environment variable that Chapel
can accept that would specify an account to be used by srun, similar to
SLURM_PARTITION that’s not listed in the documentation?
I believe the Chapel launcher passes through most/all environment variables to
the srun/salloc command, but I might be wrong. Is there already a slurm
environment variable that does what you want? (I'd be surprised if the account
to be used is only available as a command line
option...)
Looking at the man page for salloc... does SALLOC_ACCOUNT do the trick?
I've had some success using using SALLOC_PARTITION and the default
gasnetrun_ibv launcher.
Additionally, when I try to setup chapel to use slurm over ibv, I think
I’m getting linking issues for mpi calls.
I'm trying to reproduce your set-up, but Chapel with GASNet over IBV doesn't
necessarily need to use MPI at all. It looks like you're trying to build on a
system with OpenMPI, so I tried with OpenMPI version 2.0. Is that the version
you are using?
Did GASNet report that the MPI conduit was enabled during configuration?
Did it report that
it found mpicc?
You could try adjusting the GASNet configure command-line options in
third-party/gasnet/Makefile or by trying e.g.
cd third-party/gasnet/
make clobber
cd ../..
make CHPL_GASNET_CFG_OPTIONS="--without-mpi-root --without-mpi-cc
--disable-aligned-segments"
to request that GASNet not try to use MPI.
I tried reproducing on an IBV system with OpenMPI 2.0 but I didn't see the link
error. I was using master rather than 1.13 though.
I'm also a bit surprised that GASNet's mpi-spawner is linked with a program
that's trying to use slurm-gasnetrun_ibv, but of course it might be doing that.
Generally,
try to set all of your environment variables, then make in the Chapel
directory, and then build the test Chapel program.
Cheers,
-michael
The setup I used to get these results is as follows:
% $CHPL_HOME/util/printchplenv
machine info: Linux
CHPL_HOME: /ascldap/users/jcaswel/chapel-1.13.0 * script location:
/home/jcaswel/chapel-1.13.0/util
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet *
CHPL_COMM_SUBSTRATE: ibv *
CHPL_GASNET_SEGMENT: large
CHPL_TASKS: qthreads
CHPL_LAUNCHER: slurm-gasnetrun_ibv *
CHPL_TIMERS: generic
CHPL_MEM: jemalloc
CHPL_MAKE: gmake
CHPL_ATOMICS: intrinsics
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_WIDE_POINTERS: struct
CHPL_AUX_FILESYS: none
Additionally, I have set:
CHPL_COMM_SUBSTRATE: ibv
CHPL_LAUNCHER_WALLTIME=00:15:00
As well as the slurm partition. When I do this, I remake the runtime
which completes. However, when I try to compile a program, for
instance, the hello6-taskpar-dist.chpl found in
$CHPL_HOME/examples/hello6-taskpar-dist.chpl, I get the following
output:
% chpl -o hello
/ascldap/users/jcaswel/chapel-1.13.0/examples/hello6-taskpar-dist.chpl
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapSNodeBroadcast_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x19b): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x1ba): undefined reference to `MPI_Isend'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x1dd): undefined reference to `MPI_Waitall'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x1fb): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x212): undefined reference to `MPI_Recv'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapBroadcast_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x25b): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x26a): undefined reference to `MPI_Bcast'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapAlltoall_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x291): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x2ab): undefined reference to `MPI_Alltoall'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapExchange_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x2e0): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x2fa): undefined reference to `MPI_Allgather'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapBarrier_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x319): undefined reference to `MPI_Barrier'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapAbort_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x32b): undefined reference to `MPI_Abort'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapFini_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x357): undefined reference to `MPI_Comm_free'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x365): undefined reference to `MPI_Finalize'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapInit_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x389): undefined reference to `MPI_Initialized'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3a3): undefined reference to `ompi_mpi_comm_world'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3ac): undefined reference to `MPI_Comm_group'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3b1): undefined reference to `ompi_mpi_comm_world'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3bf): undefined reference to `MPI_Comm_create'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3c8): undefined reference to `MPI_Group_free'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x3d9): undefined reference to `MPI_Comm_size'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x407): undefined reference to `MPI_Comm_rank'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo
ots
trap_mpi.c:(.text+0x43a): undefined reference to `MPI_Init'
collect2: ld returned 1 exit status
gmake: *** [/tmp/chpl-jcaswel-185203.deleteme/hello.tmp] Error 1
error: compiling generated source
I’m planning on updating to 1.13.1 today as well to see if that
resolves the issue.
Thank you very much in advance,
Jake
------------------------------------------------------------------------------
_______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
------------------------------------------------------------------------------
_______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
