Hello all,
Thank you very much for all of your help! I'm no longer getting the mpi-linking
issues. Also thank you for letting me know that Chapel passes through those
environment variables.
Also, Michael (in case it's helpful), it looks like openMPI version 1.8 is the
latest version installed on the machine I was using.
However, now I am finding that if I try to run my compiled program, it
is requiring that SSH_SERVERS is missing. Is there any way to use the
chapel launcher without specifying the SSH_SERVERS? I've tried using
both
CHPL_COMM="gasnet"
CHPL_LAUNCHER="slurm-srun"
And
CHPL_COMM="gasnet"
CHPL_LAUNCHER="slurm-gasnetrun-ibv"
But both times that I've tried to run this using sbatch or salloc / srun, I get:
*** FATAL ERROR: Error spawning SPMD worker threads. Exiting...
Environment variable SSH_SERVERS is missing.
/var/spool/slurmd/job1035561/slurm_script: line 9: 99690 Aborted
Again, thank you all for your help,
Jake
-----Original Message-----
From: Michael Ferguson [mailto:[email protected]]
Sent: Thursday, July 28, 2016 7:16 AM
To: Caswell, Jacob <[email protected]>
Cc: [email protected]
Subject: [EXTERNAL] Re: [Chapel-developers] Issues setting up Chapel
for multilocale use
Hi -
On 7/27/16, 10:14 AM, "Caswell, Jacob" <[email protected]> wrote:
Hello all,
I’m quite new to Chapel, so I apologize if I’m sending this email to
the wrong mailing lists.
Welcome. Anyway, I'd expect questions like this to go to chapel-users or to
chapel-bugs.
This question being on chapel-users will allow others who might have experience
with Chapel over IBV to share.
I’ve been having some issues setting up Chapel for multilocale use on
some of our machines here. I’ve been trying to follow the Multilocale
and Launcher pages in the documentation, however, I still haven’t
been successful. I think one of the issues might be that, on some of
our machines, slurm requires an account variable, which may be
keeping me from using native slurm. Is there an environment variable
that Chapel can accept that would specify an account to be used by
srun, similar to SLURM_PARTITION that’s not listed in the documentation?
I believe the Chapel launcher passes through most/all environment
variables to the srun/salloc command, but I might be wrong. Is there
already a slurm environment variable that does what you want? (I'd be
surprised if the account to be used is only available as a command
line
option...)
Looking at the man page for salloc... does SALLOC_ACCOUNT do the trick?
I've had some success using using SALLOC_PARTITION and the default
gasnetrun_ibv launcher.
Additionally, when I try to setup chapel to use slurm over ibv, I
think I’m getting linking issues for mpi calls.
I'm trying to reproduce your set-up, but Chapel with GASNet over IBV doesn't
necessarily need to use MPI at all. It looks like you're trying to build on a
system with OpenMPI, so I tried with OpenMPI version 2.0. Is that the version
you are using?
Did GASNet report that the MPI conduit was enabled during configuration?
Did it report that
it found mpicc?
You could try adjusting the GASNet configure command-line options in
third-party/gasnet/Makefile or by trying e.g.
cd third-party/gasnet/
make clobber
cd ../..
make CHPL_GASNET_CFG_OPTIONS="--without-mpi-root --without-mpi-cc
--disable-aligned-segments"
to request that GASNet not try to use MPI.
I tried reproducing on an IBV system with OpenMPI 2.0 but I didn't see the link
error. I was using master rather than 1.13 though.
I'm also a bit surprised that GASNet's mpi-spawner is linked with a program
that's trying to use slurm-gasnetrun_ibv, but of course it might be doing that.
Generally,
try to set all of your environment variables, then make in the Chapel
directory, and then build the test Chapel program.
Cheers,
-michael
The setup I used to get these results is as follows:
% $CHPL_HOME/util/printchplenv
machine info: Linux
CHPL_HOME: /ascldap/users/jcaswel/chapel-1.13.0 * script location:
/home/jcaswel/chapel-1.13.0/util
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet *
CHPL_COMM_SUBSTRATE: ibv *
CHPL_GASNET_SEGMENT: large
CHPL_TASKS: qthreads
CHPL_LAUNCHER: slurm-gasnetrun_ibv *
CHPL_TIMERS: generic
CHPL_MEM: jemalloc
CHPL_MAKE: gmake
CHPL_ATOMICS: intrinsics
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_WIDE_POINTERS: struct
CHPL_AUX_FILESYS: none
Additionally, I have set:
CHPL_COMM_SUBSTRATE: ibv
CHPL_LAUNCHER_WALLTIME=00:15:00
As well as the slurm partition. When I do this, I remake the runtime
which completes. However, when I try to compile a program, for
instance, the hello6-taskpar-dist.chpl found in
$CHPL_HOME/examples/hello6-taskpar-dist.chpl, I get the following
output:
% chpl -o hello
/ascldap/users/jcaswel/chapel-1.13.0/examples/hello6-taskpar-dist.chp
l
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapSNodeBroadcast_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x19b): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x1ba): undefined reference to `MPI_Isend'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x1dd): undefined reference to `MPI_Waitall'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x1fb): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x212): undefined reference to `MPI_Recv'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapBroadcast_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x25b): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x26a): undefined reference to `MPI_Bcast'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapAlltoall_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x291): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x2ab): undefined reference to `MPI_Alltoall'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapExchange_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x2e0): undefined reference to `ompi_mpi_byte'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x2fa): undefined reference to `MPI_Allgather'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapBarrier_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x319): undefined reference to `MPI_Barrier'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapAbort_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x32b): undefined reference to `MPI_Abort'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapFini_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x357): undefined reference to `MPI_Comm_free'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x365): undefined reference to `MPI_Finalize'
$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n
od
bg/
lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function
`gasneti_bootstrapInit_mpi':
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x389): undefined reference to `MPI_Initialized'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3a3): undefined reference to `ompi_mpi_comm_world'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3ac): undefined reference to `MPI_Comm_group'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3b1): undefined reference to `ompi_mpi_comm_world'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3bf): undefined reference to `MPI_Comm_create'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3c8): undefined reference to `MPI_Group_free'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x3d9): undefined reference to `MPI_Comm_size'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x407): undefined reference to `MPI_Comm_rank'
$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_
bo
ots
trap_mpi.c:(.text+0x43a): undefined reference to `MPI_Init'
collect2: ld returned 1 exit status
gmake: *** [/tmp/chpl-jcaswel-185203.deleteme/hello.tmp] Error 1
error: compiling generated source
I’m planning on updating to 1.13.1 today as well to see if that
resolves the issue.
Thank you very much in advance,
Jake
----------------------------------------------------------------------
-------- _______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
