Hi again,
it took me some time to get everything working now ... let me explain what problems I've faced:
At the beginning everything worked fine although I had to use CHPL_GASNET_SEGMENT=everything instead of the default "large" with the IBV conduit. Trying to use the large segmentation algorithm ended up with segfaults when running with 1 or 2 locales ... more than 2 locales worked but it had used much more memory than usual. After that I've debugged my program but couldn't find anything wrong so I've built Chapel and GASNet with lots of different settings (ibv spawner, segmentation, tasking layer, lots of different gasnet options, etc ...) but unfortunately without success. Then I noticed that the program just runs fine with MPI (and also UDP) regardless of how many locales are used.
Just to sum up what happened when using CHPL_GASNET_SEGMENT=everything:
The problems started after I've replaced a LinkedList with a Queue that was based on an array ... elements from this queue have to be send to other locales at runtime (on every locale exactly one such queue is instantiated). If there were a lot of data to sent around between the locales (-> workers stealing/retrieving tasks) then an error were thrown:
"ibv_reg_mr failed in firehose_move_callback errno=14" followed by a sigsegv 6
Just to sum up what happened when using CHPL_GASNET_SEGMENT=everything:
The problems started after I've replaced a LinkedList with a Queue that was based on an array ... elements from this queue have to be send to other locales at runtime (on every locale exactly one such queue is instantiated). If there were a lot of data to sent around between the locales (-> workers stealing/retrieving tasks) then an error were thrown:
"ibv_reg_mr failed in firehose_move_callback errno=14" followed by a sigsegv 6
So I started to try out the large segmentation algorithm again with different options (as described in your post and some stuff I found on the GASNet mailing list). But then again, my program didn't worked for nl <= 2...
I was able to reproduce the error when it crashs with 1 locale with a slightly modified example from the Chapel spec... the funny thing is that it won't fail always. Most times I get a SIGSEGV 11 and sometimes there seems to be a memory leak sometimes causing that my job is canceled by the batch system after a few runs.
I was able to reproduce the error when it crashs with 1 locale with a slightly modified example from the Chapel spec... the funny thing is that it won't fail always. Most times I get a SIGSEGV 11 and sometimes there seems to be a memory leak sometimes causing that my job is canceled by the batch system after a few runs.
Sample program:
use CyclicDist;
const Space = {0..numLocales-1};
const D : domain(1) dmapped Cyclic(startIdx=Space.low) = Space;
var A : [D] int;
forall a in A do
a = a.locale.id;
writeln("A: ", A);
Output (2 runs - 1 fail, 1 success):
------------------------------------------------- OUTPUT success -------------------------------------------------
/home/mp27lupo/opt/chapel-1.10.0/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/bin/gasnetrun_ibv -n 1 -E LESSKEY,MAN
PATH,INFODIR,EGO_CONFDIR,NNTPSERVER,GASNET_ROUTE_OUTPUT,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_PASS,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCREQ,XKE
YSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,CHPL_TASKS,LSB_BATCH_JID,SHELL,HOST,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LSB_TRAP
SIGS,CHPL_GASNET_SEGMENT,LS_JOBPID,LSB_ERRORFILE,LSB_JOBRES_CALLBACK,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_SERVER
DIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,WORK_LOCAL,CHPL_COMM,LSF_LIBDIR,SSH_OPTIONS,USER,LD_LIBRARY_P
ATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,ENV,HOSTTYPE,CPATH,CHPL_MAKE_COMM_SEGMENT,LSF_INVOKE_CMD,CHPL_
COMM_SUBSTRATE,CHPL_HOST_PLATFORM,LS_EXEC_T,FROM_HEADER,LSF_EAUTH_SERVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVAL,LSB_
HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENAME,MINICOM,GASNET_BACKTRACE,LSB_JOBINDEX,PATH,MAIL,CPU,SSH_S
ERVERS,CHPL_MAKE_GASNET_SEGMENT,LSB_EXIT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_COMM_DEBUG,LSB_JOBEXIT_STAT,PWD,INPUTRC,JAVA_HOME,_LMFILES_,L
ANG,WORK_SCRATCH,LSB_CHKFILENAME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,SSH_CMD,LSB_JOB_STARTER,X10
_NTHREADS,LSB_DJOB_NUMPROC,LSB_EXEC_HOSTTYPE,LSF_BINDIR,CHPL_HOME,HOME,SHLVL,QT_SYSTEM_DIR,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB_ACCT_
FILE,XCURSOR_THEME,BINARY_TYPE_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,LSB_MCP
U_HOSTS,LSB_OUTPUTFILE,LOGNAME,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,XDG_DAT
A_DIRS,LSB_ECHKPNT_RSH_CMD,MODULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_HOME,GASNET_IBV_SPAWNER,INFOPATH,LSB_EEXEC_REAL_GID,GASNET_S
PAWNFN,LSF_ENVDIR,LESSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_ /home/mp27lupo/prog/bin/example_real -nl 1 -v
QTHREADS: Using 16 Shepherds
QTHREADS: Using 1 Workers per Shepherd
QTHREADS: Guard Pages Enabled
QTHREADS: Using 8388608 byte stack size.
executing on node 0 of 1 node(s): hpa0021
hello 1
------------------------------------------------- OUTPUT fail -------------------------------------------------
/home/mp27lupo/opt/chapel-1.10.0-debug/third-party/gasnet/install/linux64-gnu-unknown/seg-large/debug/bin/gasnetrun_ibv -n 1 -E LESSK
EY,MANPATH,INFODIR,EGO_CONFDIR,NNTPSERVER,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_PASS,GASNET_ROUTE_OUTPUT,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCR
EQ,XKEYSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,LSB_BATCH_JID,SHELL,HOST,CHPL_TASKS,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LS
B_TRAPSIGS,LS_JOBPID,LSB_ERRORFILE,CHPL_GASNET_SEGMENT,LSB_JOBRES_CALLBACK,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_
SERVERDIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,WORK_LOCAL,LSF_LIBDIR,CHPL_COMM,USER,SSH_OPTIONS,LD_LIB
RARY_PATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,ENV,HOSTTYPE,CPATH,LSF_INVOKE_CMD,CHPL_MAKE_COMM_SEGMENT
,LS_EXEC_T,FROM_HEADER,CHPL_HOST_PLATFORM,CHPL_COMM_SUBSTRATE,LSF_EAUTH_SERVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVA
L,LSB_HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENAME,MINICOM,GASNET_BACKTRACE,LSB_JOBINDEX,PATH,MAIL,CPU
,SSH_SERVERS,LSB_EXIT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_MAKE_GASNET_SEGMENT,LSB_JOBEXIT_STAT,CHPL_COMM_DEBUG,PWD,INPUTRC,JAVA_HOME,_LMFI
LES_,LANG,WORK_SCRATCH,LSB_CHKFILENAME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,LSB_JOB_STARTER,X10_N
THREADS,SSH_CMD,LSB_DJOB_NUMPROC,LSB_EXEC_HOSTTYPE,LSF_BINDIR,HOME,SHLVL,QT_SYSTEM_DIR,CHPL_HOME,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB
_ACCT_FILE,XCURSOR_THEME,BINARY_TYPE_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,L
SB_MCPU_HOSTS,LSB_OUTPUTFILE,LOGNAME,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,X
DG_DATA_DIRS,LSB_ECHKPNT_RSH_CMD,MODULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_HOME,INFOPATH,GASNET_IBV_SPAWNER,LSB_EEXEC_REAL_GID,LS
F_ENVDIR,GASNET_SPAWNFN,LESSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_ /home/mp2
7lupo/prog/bin/example_real -nl 1 -v
QTHREADS: Using 16 Shepherds
QTHREADS: Using 1 Workers per Shepherd
QTHREADS: Guard Pages Enabled
QTHREADS: Using 8388608 byte stack size.
executing on node 0 of 1 node(s): hpa0010
0: enter barrier for 'barrier before main'
0: enter barrier for 'barrier before broadcasting globals'
0: enter barrier for 'barrier after broadcasting globals'
0: enter barrier for 'pre-user-code hook begin'
0: enter barrier for 'pre-user-code hook end'
*** Caught a fatal signal: SIGSEGV(11) on node 0/1
*** Caught a fatal signal: SIGSEGV(11) on node 0/1
Cleaning up orphaned processes...
--------------------------------------------------------------------------------------------------------
The memory usage is also very interesting:
------------------------------------------------------------
Exited with exit code 11.
Resource usage summary:
CPU time : 8.12 sec.
Max Memory : 1284 MB
Average Memory : 1284.00 MB
Total Requested Memory : -
Delta Memory : -
(Delta: the difference between total requested memory and actual max usage.)
Max Swap : 1388 MB
Max Processes : 6
Max Threads : 7
The output (if any) follows:
------------------------------------------------------------
Exited with exit code 11.
Resource usage summary:
CPU time : 8.12 sec.
Max Memory : 1284 MB
Average Memory : 1284.00 MB
Total Requested Memory : -
Delta Memory : -
(Delta: the difference between total requested memory and actual max usage.)
Max Swap : 1388 MB
Max Processes : 6
Max Threads : 7
The output (if any) follows:
------------------------------------------------------------
For comparison running the same program with the same configuration except that CHPL_COMM_SEGMENT=everything:
------------------------------------------------------------
Successfully completed.
Resource usage summary:
CPU time : 0.97 sec.
Max Memory : 5 MB
Average Memory : 5.00 MB
Total Requested Memory : -
Delta Memory : -
(Delta: the difference between total requested memory and actual max usage.)
Max Swap : 166 MB
Max Processes : 6
Max Threads : 7
The output (if any) follows:
------------------------------------------------------------
Successfully completed.
Resource usage summary:
CPU time : 0.97 sec.
Max Memory : 5 MB
Average Memory : 5.00 MB
Total Requested Memory : -
Delta Memory : -
(Delta: the difference between total requested memory and actual max usage.)
Max Swap : 166 MB
Max Processes : 6
Max Threads : 7
The output (if any) follows:
------------------------------------------------------------
Chapel environment variables that were set:
export CHPL_HOME=~/opt/chapel
export CHPL_HOST_PLATFORM=`$CHPL_HOME/util/chplenv/chpl_platform.py`
export GASNET_ROUTE_OUTPUT=0
export GASNET_SPAWNFN=S
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ibv
export CHPL_GASNET_SEGMENT=large
export CHPL_HOST_PLATFORM=`$CHPL_HOME/util/chplenv/chpl_platform.py`
export GASNET_ROUTE_OUTPUT=0
export GASNET_SPAWNFN=S
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ibv
export CHPL_GASNET_SEGMENT=large
export CHPL_MAKE_GASNET_SEGMENT=large
#export CHPL_MAKE_COMM_SEGMENT=large
export MANPATH=$MANPATH:$CHPL_HOME/man
export PATH=$PATH:$CHPL_HOME/bin/$CHPL_HOST_PLATFORM/
export SSH_CMD=ssh
export SSH_OPTIONS=
export CHPL_TASKS=qthreads
#export CHPL_MAKE_COMM_SEGMENT=large
export MANPATH=$MANPATH:$CHPL_HOME/man
export PATH=$PATH:$CHPL_HOME/bin/$CHPL_HOST_PLATFORM/
export SSH_CMD=ssh
export SSH_OPTIONS=
export CHPL_TASKS=qthreads
--------- default variables that were printed by $CHPL_HOME/util/printchplenv ------
CHPL_MEM=dlmalloc
CHPL_GMP=gmp
CHPL_HWLOC=hwloc
CHPL_REGEXP=re2
CHPL_GMP=gmp
CHPL_HWLOC=hwloc
CHPL_REGEXP=re2
Unfortunately I don't have the exact logs any more otherwise I would have copied the whole output of the printchplenv script... if you need them just let me know and I will run with IBV again. I had also GASNet tracing enabled so I could send this logs as well if you're interested.
Furthermore I've never found out why my program is crashing for 2 locales ... I wasn't able to create a simple program that reproduces this error.
So finally I've ended up with replacing the GASNet 1.22.0 version with the newest 1.24.0 release... sadly the ibv error still occured but luckily now I was able to build the MXM conduit. Now everything seems to work fine but I'm getting some warnings where I don't know how to fix them:
[1425347486.475355] sys.c:719 MXM WARN Conflicting CPU frequencies detected, using: 2601.00
...
[1425347486.909148] [hpa0273:3056 :0] mxm.c:185 MXM WARN The 'ulimit -s' on the system is set to 'unlimited'. This may have
negative performance implications. Please set the heap size to the default value (10240)
negative performance implications. Please set the heap size to the default value (10240)
Both warnings are printed for each locale that I'm starting. I couldn't find anything yet how to get rid of these warnings / set at least the heap size myself so I would appreciate any help anyone could offer :) The output of printchplenv with mxm conduit:
machine info: Linux hpa0273 3.0.101-0.40-default #1 SMP Thu Sep 18 13:09:38 UTC 2014 (44b8c95) x86_64
CHPL_HOME: /home/mp27lupo/opt/chapel
script location: /home/mp27lupo/opt/chapel-1.10.0/util
CHPL_HOST_PLATFORM: linux64
CHPL_HOST_COMPILER: gnu
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet
CHPL_COMM_SUBSTRATE: mxm
CHPL_GASNET_SEGMENT: everything
CHPL_TASKS: qthreads
CHPL_LAUNCHER: gasnetrun_ibv
CHPL_TIMERS: generic
CHPL_MEM: cstdlib
CHPL_MAKE: gmake
CHPL_ATOMICS: intrinsics
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_WIDE_POINTERS: struct
CHPL_LLVM: none
CHPL_AUX_FILESYS: none
CHPL_HOME: /home/mp27lupo/opt/chapel
script location: /home/mp27lupo/opt/chapel-1.10.0/util
CHPL_HOST_PLATFORM: linux64
CHPL_HOST_COMPILER: gnu
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet
CHPL_COMM_SUBSTRATE: mxm
CHPL_GASNET_SEGMENT: everything
CHPL_TASKS: qthreads
CHPL_LAUNCHER: gasnetrun_ibv
CHPL_TIMERS: generic
CHPL_MEM: cstdlib
CHPL_MAKE: gmake
CHPL_ATOMICS: intrinsics
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: gmp
CHPL_HWLOC: hwloc
CHPL_REGEXP: re2
CHPL_WIDE_POINTERS: struct
CHPL_LLVM: none
CHPL_AUX_FILESYS: none
System info:
Operating System: SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11, PATCHLEVEL = 3
Other (maybe) useful information:
gcc (GCC) 4.8.3
openmpi/gcc/1.8.2
python/3.4.2
chapel/1.10.0
gcc (GCC) 4.8.3
openmpi/gcc/1.8.2
python/3.4.2
chapel/1.10.0
Used batch system: LSF
Best regards,
Marco
Gesendet: Mittwoch, 18. Februar 2015 um 17:57 Uhr
Von: "Michael Ferguson" <[email protected]>
An: "Marco Postigo" <[email protected]>
Cc: "[email protected]" <[email protected]>
Betreff: Re: Problem with compiling chapel programs if ibv conduit is used
Von: "Michael Ferguson" <[email protected]>
An: "Marco Postigo" <[email protected]>
Cc: "[email protected]" <[email protected]>
Betreff: Re: Problem with compiling chapel programs if ibv conduit is used
Hi Marco -
I understand that you got the IBV conduit working, but I wanted to leave a pointer to my earlier experience with Chapel on IBV and MXM conduits:
I was able to get both IBV and MXM conduits to work with CHPL_GASNET_SEGMENT=fast ... but had to --disable-pshm when configuring GASNet - although in a current Chapel, GASNet should build with --disable-pshm for these conduits by default.
Cheers,
-michael
On Tue, Feb 10, 2015 at 7:19 PM, Marco Postigo <[email protected]> wrote:
Hello again :)
I've just installed Chapel on a computing cluster of a partner university to make some benchmarks of my GLB implementation. Unfortunately I wasn't able to use ibv or mxm conduit (only udp works).
When I've tried to simply build Chapel (with make / gmake in Chapels home directory) it already failed while trying to building Gasnet because of the mxm conduit... so I've checked the Gasnet doc and edited the Makefile provided from Chapel in third-party/gasnet to disable mxm support (and prevent it from being built). I don't have the error log any more but if you're interested I will send it as well.
The main problem is when I'm trying to compile my program having ibv activated... the error log will follow at the end of this mail. I have the following environment variables set:
export CHPL_HOME=~/opt/chapel
export CHPL_HOST_PLATFORM=`$CHPL_HOME/util/chplenv/chpl_platform.py`
export GASNET_ROUTE_OUTPUT=0
export GASNET_SPAWNFN=S
export CHPL_COMM=gasnet
export CHPL_COMM_SUBSTRATE=ibv
export CHPL_GASNET_SEGMENT=everything
export MANPATH=$MANPATH:$CHPL_HOME/man
export PATH=$PATH:$CHPL_HOME/bin/$CHPL_HOST_PLATFORM/
export SSH_CMD=ssh
export SSH_OPTIONS=-x
export CHPL_TASKS=fifo
Operating System: SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11, PATCHLEVEL = 3
Other (maybe) useful information:
gcc (GCC) 4.8.3
openmpi/gcc/1.8.2
python/3.4.2
chapel/1.10.0
Compilation error occuring when trying to compile any program (example is here hello2-module.chpl)... I've given the --print-commands parameter as well because it may help.
~/opt/chapel> chpl --print-commands -o hello examples/hello2-module.chpl
/home/mp27lupo/opt/chapel/util/chplenv/check_huge_pages.py
/shared/apps/gcc/4.8.3/bin/gcc -DGASNET_PAR -D_REENTRANT -D_GNU_SOURCE -I/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/include -I/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/include/ibv-conduit -I/opt/mellanox/fca/include -std=c99 -Wno-strict-overflow -I/home/mp27lupo/opt/chapel/third-party/dlmalloc/include -I/home/mp27lupo/opt/chapel/runtime//include/hwloc/none -std=c99 -DCHPL_TASKS_MODEL_H=\"tasks-fifo.h\" -DCHPL_THREADS_MODEL_H=\"threads-pthreads.h\" -DCHPL_WIDE_POINTER_STRUCT -DUSE_LOCKS -DONLY_MSPACES -DMSPACES -DCHPL_HAS_GMP -c -o /tmp/chpl-mp27lupo-27468.deleteme/hello.tmp.o -I. -I/home/mp27lupo/opt/chapel/runtime//include/localeModels/flat -I/home/mp27lupo/opt/chapel/runtime//include/localeModels -I/home/mp27lupo/opt/chapel/runtime//include/comm/gasnet -I/home/mp27lupo/opt/chapel/runtime//include/comm -I/home/mp27lupo/opt/chapel/runtime//include/tasks/fifo -I/home/mp27lupo/opt/chapel/runtime//include/threads/pthreads -I/home/mp27lupo/opt/chapel/runtime//include/comp-gnu -I/home/mp27lupo/opt/chapel/runtime//include/linux64 -I/home/mp27lupo/opt/chapel/runtime//include -I/home/mp27lupo/opt/chapel/runtime//include/qio -I/home/mp27lupo/opt/chapel/runtime//include/atomics/intrinsics -I/home/mp27lupo/opt/chapel/runtime//include/mem/dlmalloc -I/home/mp27lupo/opt/chapel/third-party/utf8-decoder -I/home/mp27lupo/opt/chapel/third-party/dlmalloc/include -I/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/include -I/home/mp27lupo/opt/chapel/runtime//include/hwloc/none /tmp/chpl-mp27lupo-27468.deleteme/_main.c
g++ -L/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/lib -Wl,-rpath -Wl,/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/lib -L/home/mp27lupo/opt/chapel/third-party/re2/install/linux64-gnu-unknown/lib -Wl,-rpath -Wl,/home/mp27lupo/opt/chapel/third-party/re2/install/linux64-gnu-unknown/lib -o /tmp/chpl-mp27lupo-27468.deleteme/hello.tmp -L/home/mp27lupo/opt/chapel/lib/linux64.gnu.arch-unknown.loc-flat.comm-gasnet.ibv.large.tasks-fifo.tmr-generic.mem-dlmalloc.atomics-intrinsics.none.gmp.hwloc-none.re2.wide-struct.fs-none /tmp/chpl-mp27lupo-27468.deleteme/hello.tmp.o /home/mp27lupo/opt/chapel/lib/linux64.gnu.arch-unknown.loc-flat.comm-gasnet.ibv.large.tasks-fifo.tmr-generic.mem-dlmalloc.atomics-intrinsics.none.gmp.hwloc-none.re2.wide-struct.fs-none/main.o -lchpl -lm -lpthread -L/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib -L/usr/lib64 -L/opt/mellanox/fca/lib -lgasnet-ibv-par -libverbs -lfca -lpthread -L/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -lgcc -lm -L/home/mp27lupo/opt/chapel/third-party/dlmalloc/install/linux64-gnu-unknown -ldlmalloc -lgmp -lre2 -lpthread
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapExchange_mpi':
gasnet_bootstrap_mpi.c:(.text+0xcd): undefined reference to `ompi_mpi_byte'
gasnet_bootstrap_mpi.c:(.text+0xf9): undefined reference to `MPI_Allgather'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapBroadcast_mpi':
gasnet_bootstrap_mpi.c:(.text+0x132): undefined reference to `ompi_mpi_byte'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapInit_mpi':
gasnet_bootstrap_mpi.c:(.text+0x17c): undefined reference to `MPI_Initialized'
gasnet_bootstrap_mpi.c:(.text+0x1a9): undefined reference to `MPI_Init'
gasnet_bootstrap_mpi.c:(.text+0x1b5): undefined reference to `ompi_mpi_comm_world'
gasnet_bootstrap_mpi.c:(.text+0x1ba): undefined reference to `MPI_Comm_group'
gasnet_bootstrap_mpi.c:(.text+0x1c8): undefined reference to `ompi_mpi_comm_world'
gasnet_bootstrap_mpi.c:(.text+0x1cd): undefined reference to `MPI_Comm_create'
gasnet_bootstrap_mpi.c:(.text+0x1d5): undefined reference to `MPI_Group_free'
gasnet_bootstrap_mpi.c:(.text+0x1e6): undefined reference to `MPI_Comm_size'
gasnet_bootstrap_mpi.c:(.text+0x20a): undefined reference to `MPI_Comm_rank'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapFini_mpi':
gasnet_bootstrap_mpi.c:(.text+0x24a): undefined reference to `MPI_Comm_free'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapAbort_mpi':
gasnet_bootstrap_mpi.c:(.text+0x27e): undefined reference to `MPI_Abort'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapAlltoall_mpi':
gasnet_bootstrap_mpi.c:(.text+0x2c9): undefined reference to `ompi_mpi_byte'
gasnet_bootstrap_mpi.c:(.text+0x2db): undefined reference to `MPI_Alltoall'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapBroadcast_mpi':
gasnet_bootstrap_mpi.c:(.text+0x13a): undefined reference to `MPI_Bcast'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapFini_mpi':
gasnet_bootstrap_mpi.c:(.text+0x265): undefined reference to `MPI_Finalize'
/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In function `gasneti_bootstrapBarrier_mpi':
gasnet_bootstrap_mpi.c:(.text+0x2a8): undefined reference to `MPI_Barrier'
collect2: Fehler: ld gab 1 als Ende-Status zurück
gmake: *** [/tmp/chpl-mp27lupo-27468.deleteme/hello.tmp] Fehler 1
error: compiling generated source
Am I doing anything wrong? I've tried to make a hello world mpi program which worked fine... I have no clue why it can't find the MPI references in gasnet.
Thanks in advance and best regards,
Marco
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
------------------------------------------------------------------------------ Dive into the World of Parallel Programming The Go Parallel Website, sponsored by Intel and developed in partnership with Slashdot Media, is your hub for all things parallel software development, from weekly thought leadership blogs to news, videos, case studies, tutorials and more. Take a look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________ Chapel-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-users
