Hello all, Thank you very much for all of your help! I'm no longer getting the mpi-linking issues. Also thank you for letting me know that Chapel passes through those environment variables.
Also, Michael (in case it's helpful), it looks like openMPI version 1.8 is the latest version installed on the machine I was using. However, now I am finding that if I try to run my compiled program, it is requiring that SSH_SERVERS is missing. Is there any way to use the chapel launcher without specifying the SSH_SERVERS? I've tried using both CHPL_COMM="gasnet" CHPL_LAUNCHER="slurm-srun" And CHPL_COMM="gasnet" CHPL_LAUNCHER="slurm-gasnetrun-ibv" But both times that I've tried to run this using sbatch or salloc / srun, I get: *** FATAL ERROR: Error spawning SPMD worker threads. Exiting... Environment variable SSH_SERVERS is missing. /var/spool/slurmd/job1035561/slurm_script: line 9: 99690 Aborted Again, thank you all for your help, Jake -----Original Message----- From: Michael Ferguson [mailto:[email protected]] Sent: Thursday, July 28, 2016 7:16 AM To: Caswell, Jacob <[email protected]> Cc: [email protected] Subject: [EXTERNAL] Re: [Chapel-developers] Issues setting up Chapel for multilocale use Hi - On 7/27/16, 10:14 AM, "Caswell, Jacob" <[email protected]> wrote: >Hello all, > >I’m quite new to Chapel, so I apologize if I’m sending this email to >the wrong mailing lists. Welcome. Anyway, I'd expect questions like this to go to chapel-users or to chapel-bugs. This question being on chapel-users will allow others who might have experience with Chapel over IBV to share. >I’ve been having some issues setting up Chapel for multilocale use on >some of our machines here. I’ve been trying to follow the Multilocale >and Launcher pages in the documentation, however, I still haven’t been >successful. I think one of the issues might be that, on some of our >machines, slurm requires an account variable, which may be keeping me >from using native slurm. Is there an environment variable that Chapel >can accept that would specify an account to be used by srun, similar to >SLURM_PARTITION that’s not listed in the documentation? I believe the Chapel launcher passes through most/all environment variables to the srun/salloc command, but I might be wrong. Is there already a slurm environment variable that does what you want? (I'd be surprised if the account to be used is only available as a command line option...) Looking at the man page for salloc... does SALLOC_ACCOUNT do the trick? I've had some success using using SALLOC_PARTITION and the default gasnetrun_ibv launcher. > >Additionally, when I try to setup chapel to use slurm over ibv, I think >I’m getting linking issues for mpi calls. I'm trying to reproduce your set-up, but Chapel with GASNet over IBV doesn't necessarily need to use MPI at all. It looks like you're trying to build on a system with OpenMPI, so I tried with OpenMPI version 2.0. Is that the version you are using? Did GASNet report that the MPI conduit was enabled during configuration? Did it report that it found mpicc? You could try adjusting the GASNet configure command-line options in third-party/gasnet/Makefile or by trying e.g. cd third-party/gasnet/ make clobber cd ../.. make CHPL_GASNET_CFG_OPTIONS="--without-mpi-root --without-mpi-cc --disable-aligned-segments" to request that GASNet not try to use MPI. I tried reproducing on an IBV system with OpenMPI 2.0 but I didn't see the link error. I was using master rather than 1.13 though. I'm also a bit surprised that GASNet's mpi-spawner is linked with a program that's trying to use slurm-gasnetrun_ibv, but of course it might be doing that. Generally, try to set all of your environment variables, then make in the Chapel directory, and then build the test Chapel program. Cheers, -michael >The setup I used to get these results is as follows: > >% $CHPL_HOME/util/printchplenv >machine info: Linux >CHPL_HOME: /ascldap/users/jcaswel/chapel-1.13.0 * script location: >/home/jcaswel/chapel-1.13.0/util >CHPL_TARGET_PLATFORM: linux64 >CHPL_TARGET_COMPILER: gnu >CHPL_TARGET_ARCH: unknown >CHPL_LOCALE_MODEL: flat >CHPL_COMM: gasnet * > CHPL_COMM_SUBSTRATE: ibv * > CHPL_GASNET_SEGMENT: large >CHPL_TASKS: qthreads >CHPL_LAUNCHER: slurm-gasnetrun_ibv * >CHPL_TIMERS: generic >CHPL_MEM: jemalloc >CHPL_MAKE: gmake >CHPL_ATOMICS: intrinsics > CHPL_NETWORK_ATOMICS: none >CHPL_GMP: gmp >CHPL_HWLOC: hwloc >CHPL_REGEXP: re2 >CHPL_WIDE_POINTERS: struct >CHPL_AUX_FILESYS: none > >Additionally, I have set: >CHPL_COMM_SUBSTRATE: ibv >CHPL_LAUNCHER_WALLTIME=00:15:00 > >As well as the slurm partition. When I do this, I remake the runtime >which completes. However, when I try to compile a program, for >instance, the hello6-taskpar-dist.chpl found in >$CHPL_HOME/examples/hello6-taskpar-dist.chpl, I get the following >output: >% chpl -o hello >/ascldap/users/jcaswel/chapel-1.13.0/examples/hello6-taskpar-dist.chpl >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapSNodeBroadcast_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x19b): undefined reference to `ompi_mpi_byte' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x1ba): undefined reference to `MPI_Isend' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x1dd): undefined reference to `MPI_Waitall' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x1fb): undefined reference to `ompi_mpi_byte' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x212): undefined reference to `MPI_Recv' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapBroadcast_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x25b): undefined reference to `ompi_mpi_byte' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x26a): undefined reference to `MPI_Bcast' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapAlltoall_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x291): undefined reference to `ompi_mpi_byte' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x2ab): undefined reference to `MPI_Alltoall' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapExchange_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x2e0): undefined reference to `ompi_mpi_byte' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x2fa): undefined reference to `MPI_Allgather' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapBarrier_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x319): undefined reference to `MPI_Barrier' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapAbort_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x32b): undefined reference to `MPI_Abort' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapFini_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x357): undefined reference to `MPI_Comm_free' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x365): undefined reference to `MPI_Finalize' >$CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nod >bg/ >lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >`gasneti_bootstrapInit_mpi': >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x389): undefined reference to `MPI_Initialized' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3a3): undefined reference to `ompi_mpi_comm_world' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3ac): undefined reference to `MPI_Comm_group' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3b1): undefined reference to `ompi_mpi_comm_world' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3bf): undefined reference to `MPI_Comm_create' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3c8): undefined reference to `MPI_Group_free' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x3d9): undefined reference to `MPI_Comm_size' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x407): undefined reference to `MPI_Comm_rank' >$CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_bo >ots >trap_mpi.c:(.text+0x43a): undefined reference to `MPI_Init' >collect2: ld returned 1 exit status >gmake: *** [/tmp/chpl-jcaswel-185203.deleteme/hello.tmp] Error 1 >error: compiling generated source > >I’m planning on updating to 1.13.1 today as well to see if that >resolves the issue. > > >Thank you very much in advance, >Jake > ------------------------------------------------------------------------------ _______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
