Hi Brad, Thank you so much for your response. I'm not seeing the gasnetrun_ibv utility. The README mentions that, for MPI support, we should use --with-ibv-spawner=mpi to avoid using ssh. I set that using CHPL_GASNET_CFG_OPTIONS, and then with the environment variable GASNET_IBV_SPAWNER="mpi", however I still can't find the utility. Is it automatically added to the path?
Thank you, Jake -----Original Message----- From: Brad Chamberlain [mailto:[email protected]] Sent: Tuesday, August 2, 2016 10:27 AM To: Caswell, Jacob <[email protected]> Cc: Michael Ferguson <[email protected]>; [email protected] Subject: RE: [EXTERNAL] Re: [Chapel-developers] Issues setting up Chapel for multilocale use Hi Jacob -- I was hoping someone else would beat me to this because I don't have much experience with slurm or GASNet over ibv, but just to keep this thread from stalling out: I'm fairly certain that the message you're getting here is from GASNet's gasnetrun_ibv utility which is documented in $CHPL_HOME/third-party/gasnet/gasnet-*/ibv-conduit/README: https://github.com/chapel-lang/chapel/blob/master/third-party/gasnet/gasnet-src/ibv-conduit/README#L34 I'm hoping that knowing this may help you get unstuck, but perhaps someone with more srun+gasnet/ibv experience will be able to chime in as well. Here at Cray, we're working on getting some nightly testing over this combination up and running, so I'll check whether anyone involved has any further tips. Best wishes, -Brad On Mon, 1 Aug 2016, Caswell, Jacob wrote: > Hello all, > > Thank you very much for all of your help! I'm no longer getting the > mpi-linking issues. Also thank you for letting me know that Chapel passes > through those environment variables. > > Also, Michael (in case it's helpful), it looks like openMPI version 1.8 is > the latest version installed on the machine I was using. > > However, now I am finding that if I try to run my compiled program, it > is requiring that SSH_SERVERS is missing. Is there any way to use the > chapel launcher without specifying the SSH_SERVERS? I've tried using > both > > CHPL_COMM="gasnet" > CHPL_LAUNCHER="slurm-srun" > > And > > CHPL_COMM="gasnet" > CHPL_LAUNCHER="slurm-gasnetrun-ibv" > > But both times that I've tried to run this using sbatch or salloc / srun, I > get: > > *** FATAL ERROR: Error spawning SPMD worker threads. Exiting... > Environment variable SSH_SERVERS is missing. > /var/spool/slurmd/job1035561/slurm_script: line 9: 99690 Aborted > > Again, thank you all for your help, > Jake > -----Original Message----- > From: Michael Ferguson [mailto:[email protected]] > Sent: Thursday, July 28, 2016 7:16 AM > To: Caswell, Jacob <[email protected]> > Cc: [email protected] > Subject: [EXTERNAL] Re: [Chapel-developers] Issues setting up Chapel > for multilocale use > > Hi - > > On 7/27/16, 10:14 AM, "Caswell, Jacob" <[email protected]> wrote: > >> Hello all, >> >> I’m quite new to Chapel, so I apologize if I’m sending this email to >> the wrong mailing lists. > > Welcome. Anyway, I'd expect questions like this to go to chapel-users or to > chapel-bugs. > This question being on chapel-users will allow others who might have > experience with Chapel over IBV to share. > >> I’ve been having some issues setting up Chapel for multilocale use on >> some of our machines here. I’ve been trying to follow the Multilocale >> and Launcher pages in the documentation, however, I still haven’t >> been successful. I think one of the issues might be that, on some of >> our machines, slurm requires an account variable, which may be >> keeping me from using native slurm. Is there an environment variable >> that Chapel can accept that would specify an account to be used by >> srun, similar to SLURM_PARTITION that’s not listed in the documentation? > > I believe the Chapel launcher passes through most/all environment > variables to the srun/salloc command, but I might be wrong. Is there > already a slurm environment variable that does what you want? (I'd be > surprised if the account to be used is only available as a command > line > option...) > > Looking at the man page for salloc... does SALLOC_ACCOUNT do the trick? > I've had some success using using SALLOC_PARTITION and the default > gasnetrun_ibv launcher. > >> >> Additionally, when I try to setup chapel to use slurm over ibv, I >> think I’m getting linking issues for mpi calls. > > I'm trying to reproduce your set-up, but Chapel with GASNet over IBV doesn't > necessarily need to use MPI at all. It looks like you're trying to build on a > system with OpenMPI, so I tried with OpenMPI version 2.0. Is that the version > you are using? > > Did GASNet report that the MPI conduit was enabled during configuration? > Did it report that > it found mpicc? > > You could try adjusting the GASNet configure command-line options in > third-party/gasnet/Makefile or by trying e.g. > > cd third-party/gasnet/ > make clobber > cd ../.. > make CHPL_GASNET_CFG_OPTIONS="--without-mpi-root --without-mpi-cc > --disable-aligned-segments" > > > to request that GASNet not try to use MPI. > > I tried reproducing on an IBV system with OpenMPI 2.0 but I didn't see the > link error. I was using master rather than 1.13 though. > > I'm also a bit surprised that GASNet's mpi-spawner is linked with a program > that's trying to use slurm-gasnetrun_ibv, but of course it might be doing > that. > Generally, > try to set all of your environment variables, then make in the Chapel > directory, and then build the test Chapel program. > > Cheers, > > -michael > >> The setup I used to get these results is as follows: >> >> % $CHPL_HOME/util/printchplenv >> machine info: Linux >> CHPL_HOME: /ascldap/users/jcaswel/chapel-1.13.0 * script location: >> /home/jcaswel/chapel-1.13.0/util >> CHPL_TARGET_PLATFORM: linux64 >> CHPL_TARGET_COMPILER: gnu >> CHPL_TARGET_ARCH: unknown >> CHPL_LOCALE_MODEL: flat >> CHPL_COMM: gasnet * >> CHPL_COMM_SUBSTRATE: ibv * >> CHPL_GASNET_SEGMENT: large >> CHPL_TASKS: qthreads >> CHPL_LAUNCHER: slurm-gasnetrun_ibv * >> CHPL_TIMERS: generic >> CHPL_MEM: jemalloc >> CHPL_MAKE: gmake >> CHPL_ATOMICS: intrinsics >> CHPL_NETWORK_ATOMICS: none >> CHPL_GMP: gmp >> CHPL_HWLOC: hwloc >> CHPL_REGEXP: re2 >> CHPL_WIDE_POINTERS: struct >> CHPL_AUX_FILESYS: none >> >> Additionally, I have set: >> CHPL_COMM_SUBSTRATE: ibv >> CHPL_LAUNCHER_WALLTIME=00:15:00 >> >> As well as the slurm partition. When I do this, I remake the runtime >> which completes. However, when I try to compile a program, for >> instance, the hello6-taskpar-dist.chpl found in >> $CHPL_HOME/examples/hello6-taskpar-dist.chpl, I get the following >> output: >> % chpl -o hello >> /ascldap/users/jcaswel/chapel-1.13.0/examples/hello6-taskpar-dist.chp >> l >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapSNodeBroadcast_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x19b): undefined reference to `ompi_mpi_byte' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x1ba): undefined reference to `MPI_Isend' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x1dd): undefined reference to `MPI_Waitall' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x1fb): undefined reference to `ompi_mpi_byte' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x212): undefined reference to `MPI_Recv' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapBroadcast_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x25b): undefined reference to `ompi_mpi_byte' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x26a): undefined reference to `MPI_Bcast' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapAlltoall_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x291): undefined reference to `ompi_mpi_byte' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x2ab): undefined reference to `MPI_Alltoall' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapExchange_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x2e0): undefined reference to `ompi_mpi_byte' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x2fa): undefined reference to `MPI_Allgather' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapBarrier_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x319): undefined reference to `MPI_Barrier' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapAbort_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x32b): undefined reference to `MPI_Abort' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapFini_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x357): undefined reference to `MPI_Comm_free' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x365): undefined reference to `MPI_Finalize' >> $CHPL_HOME/third-party/gasnet/install/linux64-gnu-unknown/seg-large/n >> od >> bg/ >> lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function >> `gasneti_bootstrapInit_mpi': >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x389): undefined reference to `MPI_Initialized' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3a3): undefined reference to `ompi_mpi_comm_world' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3ac): undefined reference to `MPI_Comm_group' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3b1): undefined reference to `ompi_mpi_comm_world' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3bf): undefined reference to `MPI_Comm_create' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3c8): undefined reference to `MPI_Group_free' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x3d9): undefined reference to `MPI_Comm_size' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x407): undefined reference to `MPI_Comm_rank' >> $CHPL_HOME/third-party/gasnet/GASNet-1.26.0/other/mpi-spawner/gasnet_ >> bo >> ots >> trap_mpi.c:(.text+0x43a): undefined reference to `MPI_Init' >> collect2: ld returned 1 exit status >> gmake: *** [/tmp/chpl-jcaswel-185203.deleteme/hello.tmp] Error 1 >> error: compiling generated source >> >> I’m planning on updating to 1.13.1 today as well to see if that >> resolves the issue. >> >> >> Thank you very much in advance, >> Jake >> > > ---------------------------------------------------------------------- > -------- _______________________________________________ > Chapel-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/chapel-users ------------------------------------------------------------------------------ _______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
