Hi Michael,
first of all thanks for your quick reply.
> Unfortunately most of the Chapel developers are working hard to
> prepare the next release of Chapel, so we might be slow in
> getting back to you for now.
Oh thats no problem. I wish you all the best for that. I'm lucky that I can
already do some benchmarks (definitely better than with MPI and UDP) and can
compare it with an implementation in X10. Although I would like to see if using
IBV helps my programs performance a little because I would need only small
enhancements to catch up the X10 program ;)
So, the Cyclic example works always with everything. It only fails
intermittently with large but only if numLocales == 1 -> if it is greater than
1 then the most times it runs as expected but the memory usage is very high...
sometimes the program will be killed by LSF because the memory usage exceeds
the maximum.
> IBV CHPL_GASNET_SEGMENT=? Cyclic test intermittently fails with SEGV
IBV CHPL_GASNET_SEGMENT=large Cyclic test intermittently fails with SEGV for nl
== 1, else it can happen that the memory limit is exceeded which leads the
batch system to cancel the job
> IBV CHPL_GASNET_SEGMENT=everything Queue fails with firehose error
right.
--> I'm not sure any more but with fast segmenting it also hadn't work... but
don't know exactly what went wrong. I could do a test with fast as well in the
next few days if you want to.
> IBV CHPL_TASKS=fifo CHPL_GASNET_SEGMENT=large Cyclic test ?
Because of fifo I have bad news ... I don't know why but I had always problems
with fifo if I'm not running the program local on my machine. I've tested it on
two clusters (already while I've written my bachelor thesis where I used chapel
as well -> version 0.7.x) and it fails on both. The first few runs always work
until I get a warning:
warning: max threads per locale is 255, but unable to create more than 1 threads
-> then the program stucks with this one thread.... if this warning isn't
coming then it is like using qthreads: exceeding the memory limit leads to LSF
killing my job. I've set my limit to 16 GB because I really don't need more and
just ran the same Cyclic example and got the following output (but it also
happens if I use 32GB):
--------------------------- STUCK BECAUSE CAN'T CREATE THREADS
----------------------------------
/home/mp27lupo/opt/chapel-1.10.0-ibv/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/bin/gasnetrun_ibv
-n 1 -E LESSKEY
,MANPATH,INFODIR,EGO_CONFDIR,NNTPSERVER,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_PASS,GASNET_ROUTE_OUTPUT,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCREQ
,XKEYSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,LSB_BATCH_JID,SHELL,HOST,CHPL_TASKS,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LSB_
TRAPSIGS,LS_JOBPID,LSB_ERRORFILE,CHPL_GASNET_SEGMENT,LSB_JOBRES_CALLBACK,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_SE
RVERDIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,WORK_LOCAL,LSF_LIBDIR,CHPL_COMM,USER,SSH_OPTIONS,LD_LIBRA
RY_PATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,ENV,HOSTTYPE,CPATH,LSF_INVOKE_CMD,CHPL_MAKE_COMM_SEGMENT,L
S_EXEC_T,FROM_HEADER,CHPL_HOST_PLATFORM,CHPL_COMM_SUBSTRATE,LSF_EAUTH_SERVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVAL,
LSB_HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENAME,MINICOM,LSB_JOBINDEX,PATH,MAIL,CPU,SSH_SERVERS,LSB_EX
IT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_MAKE_GASNET_SEGMENT,LSB_JOBEXIT_STAT,PWD,INPUTRC,JAVA_HOME,_LMFILES_,LANG,WORK_SCRATCH,LSB_CHKFILEN
AME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,LSB_JOB_STARTER,X10_NTHREADS,SSH_CMD,LSB_DJOB_NUMPROC,LS
B_EXEC_HOSTTYPE,LSF_BINDIR,HOME,SHLVL,QT_SYSTEM_DIR,CHPL_HOME,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB_ACCT_FILE,XCURSOR_THEME,BINARY_TYP
E_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,LSB_MCPU_HOSTS,LSB_OUTPUTFILE,LOGNAM
E,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,XDG_DATA_DIRS,LSB_ECHKPNT_RSH_CMD,MO
DULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_HOME,INFOPATH,GASNET_IBV_SPAWNER,LSB_EEXEC_REAL_GID,CHPL_MEM,LSF_ENVDIR,GASNET_SPAWNFN,LE
SSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_
/home/mp27lupo/prog/bin/example_rea
l -nl 1 -v
executing on node 0 of 1 node(s): hpa0036
warning: max threads per locale is 255, but unable to create more than 1 threads
------------------> KILLED LSF JOB MANUALLY <---------------------------
Cleaning up orphaned processes...
--------------------------------------------------------------------------------------------------
------------------------------------- FAIL BECAUSE OF MEM
---------------------------------------
/home/mp27lupo/opt/chapel-1.10.0-ibv/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/bin/gasnetrun_ibv
-n 1 -E LESSKEY
,MANPATH,INFODIR,EGO_CONFDIR,NNTPSERVER,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_PASS,GASNET_ROUTE_OUTPUT,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCREQ
,XKEYSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,LSB_BATCH_JID,SHELL,HOST,CHPL_TASKS,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LSB_
TRAPSIGS,LS_JOBPID,LSB_ERRORFILE,CHPL_GASNET_SEGMENT,LSB_JOBRES_CALLBACK,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_SE
RVERDIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,WORK_LOCAL,LSF_LIBDIR,CHPL_COMM,USER,SSH_OPTIONS,LD_LIBRA
RY_PATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,ENV,HOSTTYPE,CPATH,LSF_INVOKE_CMD,CHPL_MAKE_COMM_SEGMENT,L
S_EXEC_T,FROM_HEADER,CHPL_HOST_PLATFORM,CHPL_COMM_SUBSTRATE,LSF_EAUTH_SERVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVAL,
LSB_HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENAME,MINICOM,LSB_JOBINDEX,PATH,MAIL,CPU,SSH_SERVERS,LSB_EX
IT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_MAKE_GASNET_SEGMENT,LSB_JOBEXIT_STAT,PWD,INPUTRC,JAVA_HOME,_LMFILES_,LANG,WORK_SCRATCH,LSB_CHKFILEN
AME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,LSB_JOB_STARTER,X10_NTHREADS,SSH_CMD,LSB_DJOB_NUMPROC,LS
B_EXEC_HOSTTYPE,LSF_BINDIR,HOME,SHLVL,QT_SYSTEM_DIR,CHPL_HOME,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB_ACCT_FILE,XCURSOR_THEME,BINARY_TYP
E_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,LSB_MCPU_HOSTS,LSB_OUTPUTFILE,LOGNAM
E,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,XDG_DATA_DIRS,LSB_ECHKPNT_RSH_CMD,MO
DULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_HOME,INFOPATH,GASNET_IBV_SPAWNER,LSB_EEXEC_REAL_GID,CHPL_MEM,LSF_ENVDIR,GASNET_SPAWNFN,LE
SSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_
/home/mp27lupo/prog/bin/example_rea
l -nl 1 -v
executing on node 0 of 1 node(s): hpa0041
A: 0
---------------------------- ... LEAVING OUT NUM_LOCALES=2 OUTPUT ...
-------------------------------------
/home/mp27lupo/opt/chapel-1.10.0-ibv/third-party/gasnet/install/linux64-gnu-unknown/seg-large/nodbg/bin/gasnetrun_ibv
-n 3 -E LESSKEY
,MANPATH,INFODIR,EGO_CONFDIR,NNTPSERVER,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_PASS,GASNET_ROUTE_OUTPUT,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCREQ
,XKEYSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,LSB_BATCH_JID,SHELL,HOST,CHPL_TASKS,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LSB_
TRAPSIGS,LS_JOBPID,LSB_ERRORFILE,CHPL_GASNET_SEGMENT,LSB_JOBRES_CALLBACK,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_SE
RVERDIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,WORK_LOCAL,LSF_LIBDIR,CHPL_COMM,USER,SSH_OPTIONS,LD_LIBRA
RY_PATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,ENV,HOSTTYPE,CPATH,LSF_INVOKE_CMD,CHPL_MAKE_COMM_SEGMENT,L
S_EXEC_T,FROM_HEADER,CHPL_HOST_PLATFORM,CHPL_COMM_SUBSTRATE,LSF_EAUTH_SERVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVAL,
LSB_HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENAME,MINICOM,LSB_JOBINDEX,PATH,MAIL,CPU,SSH_SERVERS,LSB_EX
IT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_MAKE_GASNET_SEGMENT,LSB_JOBEXIT_STAT,PWD,INPUTRC,JAVA_HOME,_LMFILES_,LANG,WORK_SCRATCH,LSB_CHKFILEN
AME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,LSB_JOB_STARTER,X10_NTHREADS,SSH_CMD,LSB_DJOB_NUMPROC,LS
B_EXEC_HOSTTYPE,LSF_BINDIR,HOME,SHLVL,QT_SYSTEM_DIR,CHPL_HOME,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB_ACCT_FILE,XCURSOR_THEME,BINARY_TYP
E_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,LSB_MCPU_HOSTS,LSB_OUTPUTFILE,LOGNAM
E,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,XDG_DATA_DIRS,LSB_ECHKPNT_RSH_CMD,MO
DULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_HOME,INFOPATH,GASNET_IBV_SPAWNER,LSB_EEXEC_REAL_GID,CHPL_MEM,LSF_ENVDIR,GASNET_SPAWNFN,LE
SSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_
/home/mp27lupo/prog/bin/example_rea
l -nl 3 -v
------------------> LSF KILLED JOB: MEM USAGE <---------------------------
Cleaning up orphaned processes...
-------------------------------------------------------------------------
Statistic output for the run that were killed by LSF batch system:
-------------------------------------------------------------------------
TERM_MEMLIMIT: job killed after reaching LSF memory usage limit.
Exited with exit code 143.
Resource usage summary:
CPU time : 10.00 sec.
Max Memory : 20103 MB
Average Memory : 10202.00 MB
Total Requested Memory : -
Delta Memory : -
(Delta: the difference between total requested memory and actual max usage.)
Max Swap : 20342 MB
Max Processes : 7
Max Threads : 8
The output (if any) follows:
------------------------------------------------
Chapel environment variables used for this:
Available Servers: hpa0041 hpa0041 hpa0041 hpa0041 hpa0041 hpa0041 hpa0041
hpa0041
machine info: Linux hpa0041 3.0.101-0.40-default #1 SMP Thu Sep 18 13:09:38 UTC
2014 (44b8c95) x86_64
CHPL_HOME: /home/mp27lupo/opt/chapel-1.10.0-ibv
script location: /home/mp27lupo/opt/chapel-1.10.0-ibv/util
CHPL_HOST_PLATFORM: linux64
CHPL_HOST_COMPILER: gnu
CHPL_TARGET_PLATFORM: linux64
CHPL_TARGET_COMPILER: gnu
CHPL_TARGET_ARCH: unknown
CHPL_LOCALE_MODEL: flat
CHPL_COMM: gasnet
CHPL_COMM_SUBSTRATE: ibv
CHPL_GASNET_SEGMENT: large
CHPL_TASKS: fifo
CHPL_LAUNCHER: gasnetrun_ibv
CHPL_TIMERS: generic
CHPL_MEM: dlmalloc
CHPL_MAKE: gmake
CHPL_ATOMICS: intrinsics
CHPL_NETWORK_ATOMICS: none
CHPL_GMP: gmp
CHPL_HWLOC: none
CHPL_REGEXP: re2
CHPL_WIDE_POINTERS: struct
CHPL_LLVM: none
CHPL_AUX_FILESYS: none
SSH_SERVERS=hpa0041 hpa0041 hpa0041 hpa0041 hpa0041 hpa0041 hpa0041 hpa0041
> IBV CHPL_TASKS=fifo CHPL_GASNET_SEGMENT=everything Cyclic test ?
hmm ... it looks like it works with fifo and everything. I've done totally 80
runs ... where numLocales € [1, 2, ..., 16]... each one executed 5 times (so
first run nl=1, 2nd nl=1, ... 80th nl=16). I haven't got the warning, no
suspicious memory usage or anything. I will try it with some more runs because
I can't believe that the warning that no more threads can be created is now
gone in this combination. So if the error occurs then I write another mail.
NOTE: I don't know if it matters but this is how the example is compiled:
chpl -I/shared/apps/openmpi/1.8.2_gcc/include
-L/shared/apps/openmpi/1.8.2_gcc/lib -lmpi -o bin/example example.chpl
Soo... in the end I would have one last (off topic) question:
- What should I set to get the best benchmark results? I've seen using the
--fast flag is popular (what I'm already doing) but are there more what I can
do? Furthermore I've read that disabling stack overflow checks will also help
(will do that next time) so I'm wondering if there are some more flags hidden
in your doc / GASNets doc that I may have missed. Is there maybe a document
available with tips/tricks for optimizing programs (compiling, runtime checks,
maybe popular "programming-mistakes"-pitfalls)
Thanks in advance!
Best regards,
Marco
> Gesendet: Dienstag, 03. März 2015 um 17:14 Uhr
> Von: "Michael Ferguson" <[email protected]>
> An: "Marco Postigo" <[email protected]>
> Cc: "[email protected]" <[email protected]>
> Betreff: Re: Aw: Re: Problem with compiling chapel programs if ibv conduit is
> used
>
> Hi Marco -
>
> Unfortunately most of the Chapel developers are working hard to
> prepare the next release of Chapel, so we might be slow in
> getting back to you for now.
>
> I haven't seen before the problem with the sample Cyclic program not
> working like that. Could you try it with CHPL_TASKS=fifo? Also,
> I had a little bit of trouble understanding whether or not
> the Cyclic example works with large or everything? (I understand
> it is not working intermittently, but which configuration is
> that? Does it always work with the other configuration?)
>
> It might help to make a little table to summarize. I've
> started with my understanding of your situation, can you
> complete it?
>
> IBV CHPL_GASNET_SEGMENT=? Cyclic test intermittently fails with SEGV
>
> IBV CHPL_GASNET_SEGMENT=everything Queue fails with firehose error
>
> IBV CHPL_TASKS=fifo CHPL_GASNET_SEGMENT=large Cyclic test ?
> IBV CHPL_TASKS=fifo CHPL_GASNET_SEGMENT=everything Cyclic test ?
>
>
> Someone will probably need to ask the GASNet team about the firehose
> error...
>
> Regarding the "Conflicting CPU frequencies detected" error,
> I've seen errors similar to that one when using a cluster
> with CPU frequency scaling. I ended up configuring SLURM so
> that it disabled frequency scaling while the job was running.
> I bet that the MXM conduit has some kind of problem when
> used on nodes with different CPU frequencies.
>
> Thanks,
>
> -michael
>
>
> On 3/3/15, 9:47 AM, "Marco Postigo" <[email protected]> wrote:
>
> >Hi again,
> >
> >it took me some time to get everything working now ... let me explain
> >what problems I've faced:
> >
> >At the beginning everything worked fine although I had to use
> >CHPL_GASNET_SEGMENT=everything instead of the default "large" with the
> >IBV conduit. Trying to use the large segmentation algorithm ended up with
> >segfaults when running with 1 or 2 locales ...
> > more than 2 locales worked but it had used much more memory than usual.
> >After that I've debugged my program but couldn't find anything wrong so
> >I've built Chapel and GASNet with lots of different settings (ibv
> >spawner, segmentation, tasking layer, lots of
> > different gasnet options, etc ...) but unfortunately without success.
> >Then I noticed that the program just runs fine with MPI (and also UDP)
> >regardless of how many locales are used.
> >Just to sum up what happened when using CHPL_GASNET_SEGMENT=everything:
> >The problems started after I've replaced a LinkedList with a Queue that
> >was based on an array ... elements from this queue have to be send to
> >other locales at runtime (on every locale exactly one such queue is
> >instantiated). If there were a lot of data to sent
> > around between the locales (-> workers stealing/retrieving tasks) then
> >an error were thrown:
> >"ibv_reg_mr failed in firehose_move_callback errno=14" followed by a
> >sigsegv 6
> >So I started to try out the large segmentation algorithm again with
> >different options (as described in your post and some stuff I found on
> >the GASNet mailing list). But then again, my program didn't worked for nl
> ><= 2...
> >I was able to reproduce the error when it crashs with 1 locale with a
> >slightly modified example from the Chapel spec... the funny thing is that
> >it won't fail always. Most times I get a SIGSEGV 11 and sometimes there
> >seems to be a memory leak sometimes causing
> > that my job is canceled by the batch system after a few runs.
> >
> >Sample program:
> >use CyclicDist;
> >const Space = {0..numLocales-1};
> >const D : domain(1) dmapped Cyclic(startIdx=Space.low) = Space;
> >var A : [D] int;
> >forall a in A do
> > a = a.locale.id;
> >writeln("A: ", A);
> >
> >Output (2 runs - 1 fail, 1 success):
> >------------------------------------------------- OUTPUT success
> >-------------------------------------------------
> >/home/mp27lupo/opt/chapel-1.10.0/third-party/gasnet/install/linux64-gnu-un
> >known/seg-large/nodbg/bin/gasnetrun_ibv -n 1 -E LESSKEY,MAN
> >PATH,INFODIR,EGO_CONFDIR,NNTPSERVER,GASNET_ROUTE_OUTPUT,LSB_EXEC_CLUSTER,L
> >SF_EAUTH_AUX_PASS,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCREQ,XKE
> >YSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,CHPL_TASKS,LSB_BATCH_JID,SHELL,HOST,H
> >ISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LSB_TRAP
> >SIGS,CHPL_GASNET_SEGMENT,LS_JOBPID,LSB_ERRORFILE,LSB_JOBRES_CALLBACK,LSB_M
> >AX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_SERVER
> >DIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_TYPE,W
> >ORK_LOCAL,CHPL_COMM,LSF_LIBDIR,SSH_OPTIONS,USER,LD_LIBRARY_P
> >ATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIBDIR,EN
> >V,HOSTTYPE,CPATH,CHPL_MAKE_COMM_SEGMENT,LSF_INVOKE_CMD,CHPL_
> >COMM_SUBSTRATE,CHPL_HOST_PLATFORM,LS_EXEC_T,FROM_HEADER,LSF_EAUTH_SERVER,P
> >AGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVAL,LSB_
> >HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOBFILENA
> >ME,MINICOM,GASNET_BACKTRACE,LSB_JOBINDEX,PATH,MAIL,CPU,SSH_S
> >ERVERS,CHPL_MAKE_GASNET_SEGMENT,LSB_EXIT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_C
> >OMM_DEBUG,LSB_JOBEXIT_STAT,PWD,INPUTRC,JAVA_HOME,_LMFILES_,L
> >ANG,WORK_SCRATCH,LSB_CHKFILENAME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONSTARTUP
> >,LSB_DJOB_HOSTFILE,LOADEDMODULES,SSH_CMD,LSB_JOB_STARTER,X10
> >_NTHREADS,LSB_DJOB_NUMPROC,LSB_EXEC_HOSTTYPE,LSF_BINDIR,CHPL_HOME,HOME,SHL
> >VL,QT_SYSTEM_DIR,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB_ACCT_
> >FILE,XCURSOR_THEME,BINARY_TYPE_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL_CONFD
> >IR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,LSB_MCP
> >U_HOSTS,LSB_OUTPUTFILE,LOGNAME,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP,CVS_R
> >SH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,XDG_DAT
> >A_DIRS,LSB_ECHKPNT_RSH_CMD,MODULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPEN,X10_
> >HOME,GASNET_IBV_SPAWNER,INFOPATH,LSB_EEXEC_REAL_GID,GASNET_S
> >PAWNFN,LSF_ENVDIR,LESSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILENAMES,C
> >OLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_
> >/home/mp27lupo/prog/bin/example_real -nl 1 -v
> >QTHREADS: Using 16 Shepherds
> >QTHREADS: Using 1 Workers per Shepherd
> >QTHREADS: Guard Pages Enabled
> >QTHREADS: Using 8388608 byte stack size.
> >executing on node 0 of 1 node(s): hpa0021
> >hello 1
> >
> >------------------------------------------------- OUTPUT fail
> >-------------------------------------------------
> >/home/mp27lupo/opt/chapel-1.10.0-debug/third-party/gasnet/install/linux64-
> >gnu-unknown/seg-large/debug/bin/gasnetrun_ibv -n 1 -E LESSK
> >EY,MANPATH,INFODIR,EGO_CONFDIR,NNTPSERVER,LSB_EXEC_CLUSTER,LSF_EAUTH_AUX_P
> >ASS,GASNET_ROUTE_OUTPUT,HOSTNAME,EGO_TOP,LSB_EFFECTIVE_RSRCR
> >EQ,XKEYSYMDB,LSF_LIM_API_NTRIES,LSF_LOGDIR,LSB_BATCH_JID,SHELL,HOST,CHPL_T
> >ASKS,HISTSIZE,PROFILEREAD,TMPDIR,SSH_CLIENT,EGO_SERVERDIR,LS
> >B_TRAPSIGS,LS_JOBPID,LSB_ERRORFILE,CHPL_GASNET_SEGMENT,LSB_JOBRES_CALLBACK
> >,LSB_MAX_NUM_PROCESSORS,MORE,LSB_JOB_EXECUSER,LSB_JOBID,LSF_
> >SERVERDIR,LSB_JOBRES_PID,LSB_JOBNAME,SSH_TTY,BSUB_BLOCK_EXEC_HOST,LSF_PJL_
> >TYPE,WORK_LOCAL,LSF_LIBDIR,CHPL_COMM,USER,SSH_OPTIONS,LD_LIB
> >RARY_PATH,LS_COLORS,SBD_KRB5CCNAME_VAL,LSB_EEXEC_REAL_UID,XNLSPATH,EGO_LIB
> >DIR,ENV,HOSTTYPE,CPATH,LSF_INVOKE_CMD,CHPL_MAKE_COMM_SEGMENT
> >,LS_EXEC_T,FROM_HEADER,CHPL_HOST_PLATFORM,CHPL_COMM_SUBSTRATE,LSF_EAUTH_SE
> >RVER,PAGER,LS_SUBCWD,CSHEDIT,LSF_VERSION,LSB_DJOB_RU_INTERVA
> >L,LSB_HOSTS,LSB_UNIXGROUP_INT,XDG_CONFIG_DIRS,LSB_DJOB_HB_INTERVAL,LSB_JOB
> >FILENAME,MINICOM,GASNET_BACKTRACE,LSB_JOBINDEX,PATH,MAIL,CPU
> >,SSH_SERVERS,LSB_EXIT_PRE_ABORT,SSH_SENDS_LOCALE,CHPL_MAKE_GASNET_SEGMENT,
> >LSB_JOBEXIT_STAT,CHPL_COMM_DEBUG,PWD,INPUTRC,JAVA_HOME,_LMFI
> >LES_,LANG,WORK_SCRATCH,LSB_CHKFILENAME,LSF_EAUTH_CLIENT,MODULEPATH,PYTHONS
> >TARTUP,LSB_DJOB_HOSTFILE,LOADEDMODULES,LSB_JOB_STARTER,X10_N
> >THREADS,SSH_CMD,LSB_DJOB_NUMPROC,LSB_EXEC_HOSTTYPE,LSF_BINDIR,HOME,SHLVL,Q
> >T_SYSTEM_DIR,CHPL_HOME,OSTYPE,LESS_ADVANCED_PREPROCESSOR,LSB
> >_ACCT_FILE,XCURSOR_THEME,BINARY_TYPE_HPC,LS_OPTIONS,LSB_SUB_HOST,EGO_LOCAL
> >_CONFDIR,__LSF_JOB_TMPDIR__,LSB_SUB_USER,LSFUSER,LSB_QUEUE,L
> >SB_MCPU_HOSTS,LSB_OUTPUTFILE,LOGNAME,MACHTYPE,LESS,G_FILENAME_ENCODING,TMP
> >,CVS_RSH,OMPI_MCA_hwloc_base_binding_policy,SSH_CONNECTION,X
> >DG_DATA_DIRS,LSB_ECHKPNT_RSH_CMD,MODULESHOME,LSF_CGROUP_TOPDIR_KEY,LESSOPE
> >N,X10_HOME,INFOPATH,GASNET_IBV_SPAWNER,LSB_EEXEC_REAL_GID,LS
> >F_ENVDIR,GASNET_SPAWNFN,LESSCLOSE,LSF_EGO_ENVDIR,EGO_BINDIR,G_BROKEN_FILEN
> >AMES,COLORTERM,BASH_FUNC_module(),BASH_FUNC_mc(),_ /home/mp2
> >7lupo/prog/bin/example_real -nl 1 -v
> >QTHREADS: Using 16 Shepherds
> >QTHREADS: Using 1 Workers per Shepherd
> >QTHREADS: Guard Pages Enabled
> >QTHREADS: Using 8388608 byte stack size.
> >executing on node 0 of 1 node(s): hpa0010
> >0: enter barrier for 'barrier before main'
> >0: enter barrier for 'barrier before broadcasting globals'
> >0: enter barrier for 'barrier after broadcasting globals'
> >0: enter barrier for 'pre-user-code hook begin'
> >0: enter barrier for 'pre-user-code hook end'
> >*** Caught a fatal signal: SIGSEGV(11) on node 0/1
> >*** Caught a fatal signal: SIGSEGV(11) on node 0/1
> >Cleaning up orphaned processes...
> >--------------------------------------------------------------------------
> >------------------------------
> >
> >The memory usage is also very interesting:
> >------------------------------------------------------------
> >Exited with exit code 11.
> >Resource usage summary:
> >CPU time : 8.12 sec.
> >Max Memory : 1284 MB
> >Average Memory : 1284.00 MB
> >Total Requested Memory : -
> >Delta Memory : -
> >(Delta: the difference between total requested memory and actual max
> >usage.)
> >Max Swap : 1388 MB
> >Max Processes : 6
> >Max Threads : 7
> >The output (if any) follows:
> >------------------------------------------------------------
> >
> >For comparison running the same program with the same configuration
> >except that CHPL_COMM_SEGMENT=everything:
> >------------------------------------------------------------
> >Successfully completed.
> >Resource usage summary:
> >CPU time : 0.97 sec.
> >Max Memory : 5 MB
> >Average Memory : 5.00 MB
> >Total Requested Memory : -
> >Delta Memory : -
> >(Delta: the difference between total requested memory and actual max
> >usage.)
> >Max Swap : 166 MB
> >Max Processes : 6
> >Max Threads : 7
> >The output (if any) follows:
> >------------------------------------------------------------
> >
> >
> >Chapel environment variables that were set:
> >
> >export CHPL_HOME=~/opt/chapel
> >export CHPL_HOST_PLATFORM=`$CHPL_HOME/util/chplenv/chpl_platform.py`
> >export GASNET_ROUTE_OUTPUT=0
> >export GASNET_SPAWNFN=S
> >export CHPL_COMM=gasnet
> >export CHPL_COMM_SUBSTRATE=ibv
> >export CHPL_GASNET_SEGMENT=large
> >export CHPL_MAKE_GASNET_SEGMENT=large
> >#export CHPL_MAKE_COMM_SEGMENT=large
> >export MANPATH=$MANPATH:$CHPL_HOME/man
> >export PATH=$PATH:$CHPL_HOME/bin/$CHPL_HOST_PLATFORM/
> >export SSH_CMD=ssh
> >export SSH_OPTIONS=
> >export CHPL_TASKS=qthreads
> >--------- default variables that were printed by
> >$CHPL_HOME/util/printchplenv ------
> >CHPL_MEM=dlmalloc
> >CHPL_GMP=gmp
> >CHPL_HWLOC=hwloc
> >CHPL_REGEXP=re2
> >
> >Unfortunately I don't have the exact logs any more otherwise I would have
> >copied the whole output of the printchplenv script... if you need them
> >just let me know and I will run with IBV again. I had also GASNet tracing
> >enabled so I could send this logs
> > as well if you're interested.
> >Furthermore I've never found out why my program is crashing for 2 locales
> >... I wasn't able to create a simple program that reproduces this error.
> >
> >So finally I've ended up with replacing the GASNet 1.22.0 version with
> >the newest 1.24.0 release... sadly the ibv error still occured but
> >luckily now I was able to build the MXM conduit. Now everything seems to
> >work fine but I'm getting some warnings where
> > I don't know how to fix them:
> >[1425347486.475355] sys.c:719 MXM WARN Conflicting CPU
> >frequencies detected, using: 2601.00
> >...
> >[1425347486.909148] [hpa0273:3056 :0] mxm.c:185 MXM WARN The
> >'ulimit -s' on the system is set to 'unlimited'. This may have
> > negative performance implications. Please set the heap size to the
> >default value (10240)
> >
> >Both warnings are printed for each locale that I'm starting. I couldn't
> >find anything yet how to get rid of these warnings / set at least the
> >heap size myself so I would appreciate any help anyone could offer :) The
> >output of printchplenv with mxm conduit:
> >machine info: Linux hpa0273 3.0.101-0.40-default #1 SMP Thu Sep 18
> >13:09:38 UTC 2014 (44b8c95) x86_64
> >CHPL_HOME: /home/mp27lupo/opt/chapel
> >script location: /home/mp27lupo/opt/chapel-1.10.0/util
> >CHPL_HOST_PLATFORM: linux64
> >CHPL_HOST_COMPILER: gnu
> >CHPL_TARGET_PLATFORM: linux64
> >CHPL_TARGET_COMPILER: gnu
> >CHPL_TARGET_ARCH: unknown
> >CHPL_LOCALE_MODEL: flat
> >CHPL_COMM: gasnet
> > CHPL_COMM_SUBSTRATE: mxm
> > CHPL_GASNET_SEGMENT: everything
> >CHPL_TASKS: qthreads
> >CHPL_LAUNCHER: gasnetrun_ibv
> >CHPL_TIMERS: generic
> >CHPL_MEM: cstdlib
> >CHPL_MAKE: gmake
> >CHPL_ATOMICS: intrinsics
> > CHPL_NETWORK_ATOMICS: none
> >CHPL_GMP: gmp
> >CHPL_HWLOC: hwloc
> >CHPL_REGEXP: re2
> >CHPL_WIDE_POINTERS: struct
> >CHPL_LLVM: none
> >CHPL_AUX_FILESYS: none
> >
> >System info:
> >Operating System: SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11,
> >PATCHLEVEL = 3
> >Other (maybe) useful information:
> >gcc (GCC) 4.8.3
> >openmpi/gcc/1.8.2
> >python/3.4.2
> >chapel/1.10.0
> >Used batch system: LSF
> >
> >
> >Best regards,
> >Marco
> >
> >
> >Gesendet: Mittwoch, 18. Februar 2015 um 17:57 Uhr
> >Von: "Michael Ferguson" <[email protected]>
> >An: "Marco Postigo" <[email protected]>
> >Cc: "[email protected]"
> ><[email protected]>
> >Betreff: Re: Problem with compiling chapel programs if ibv conduit is used
> >Hi Marco -
> >
> >I understand that you got the IBV conduit working, but I wanted to leave
> >a pointer to my earlier experience with Chapel on IBV and MXM conduits:
> >
> >http://permalink.gmane.org/gmane.comp.lang.chapel.devel/230
> >
> >I was able to get both IBV and MXM conduits to work with
> >CHPL_GASNET_SEGMENT=fast ... but had to --disable-pshm when configuring
> >GASNet - although in a current Chapel, GASNet should build with
> >--disable-pshm for these conduits by default.
> >
> >Cheers,
> >
> >-michael
> >
> >
> >On Tue, Feb 10, 2015 at 7:19 PM, Marco Postigo <[email protected]>
> >wrote:
> >
> >Hello again :)
> >
> >I've just installed Chapel on a computing cluster of a partner university
> >to make some benchmarks of my GLB implementation. Unfortunately I wasn't
> >able to use ibv or mxm conduit (only udp works).
> >
> >When I've tried to simply build Chapel (with make / gmake in Chapels home
> >directory) it already failed while trying to building Gasnet because of
> >the mxm conduit... so I've checked the Gasnet doc and edited the Makefile
> >provided from Chapel in third-party/gasnet
> > to disable mxm support (and prevent it from being built). I don't have
> >the error log any more but if you're interested I will send it as well.
> >
> >The main problem is when I'm trying to compile my program having ibv
> >activated... the error log will follow at the end of this mail. I have
> >the following environment variables set:
> >
> >export CHPL_HOME=~/opt/chapel
> >export CHPL_HOST_PLATFORM=`$CHPL_HOME/util/chplenv/chpl_platform.py`
> >export GASNET_ROUTE_OUTPUT=0
> >export GASNET_SPAWNFN=S
> >export CHPL_COMM=gasnet
> >export CHPL_COMM_SUBSTRATE=ibv
> >export CHPL_GASNET_SEGMENT=everything
> >export MANPATH=$MANPATH:$CHPL_HOME/man
> >export PATH=$PATH:$CHPL_HOME/bin/$CHPL_HOST_PLATFORM/
> >export SSH_CMD=ssh
> >export SSH_OPTIONS=-x
> >export CHPL_TASKS=fifo
> >
> >Operating System: SUSE Linux Enterprise Server 11 (x86_64), VERSION = 11,
> >PATCHLEVEL = 3
> >
> >Other (maybe) useful information:
> >gcc (GCC) 4.8.3
> >openmpi/gcc/1.8.2
> >python/3.4.2
> >chapel/1.10.0
> >
> >Compilation error occuring when trying to compile any program (example is
> >here hello2-module.chpl)... I've given the --print-commands parameter as
> >well because it may help.
> >
> >~/opt/chapel> chpl --print-commands -o hello examples/hello2-module.chpl
> >/home/mp27lupo/opt/chapel/util/chplenv/check_huge_pages.py
> >/shared/apps/gcc/4.8.3/bin/gcc -DGASNET_PAR -D_REENTRANT -D_GNU_SOURCE
> >
> >-I/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown
> >/seg-large/nodbg/include
> >-I/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown
> >/seg-large/nodbg/include/ibv-conduit
> > -I/opt/mellanox/fca/include -std=c99 -Wno-strict-overflow
> >-I/home/mp27lupo/opt/chapel/third-party/dlmalloc/include
> >-I/home/mp27lupo/opt/chapel/runtime//include/hwloc/none -std=c99
> >-DCHPL_TASKS_MODEL_H=\"tasks-fifo.h\"
> >-DCHPL_THREADS_MODEL_H=\"threads-pthreads.h\"
> > -DCHPL_WIDE_POINTER_STRUCT -DUSE_LOCKS -DONLY_MSPACES -DMSPACES
> >-DCHPL_HAS_GMP -c -o /tmp/chpl-mp27lupo-27468.deleteme/hello.tmp.o -I.
> >-I/home/mp27lupo/opt/chapel/runtime//include/localeModels/flat
> >-I/home/mp27lupo/opt/chapel/runtime//include/localeModels
> > -I/home/mp27lupo/opt/chapel/runtime//include/comm/gasnet
> >-I/home/mp27lupo/opt/chapel/runtime//include/comm
> >-I/home/mp27lupo/opt/chapel/runtime//include/tasks/fifo
> >-I/home/mp27lupo/opt/chapel/runtime//include/threads/pthreads
> >-I/home/mp27lupo/opt/chapel/runtime//include/comp-gnu
> > -I/home/mp27lupo/opt/chapel/runtime//include/linux64
> >-I/home/mp27lupo/opt/chapel/runtime//include
> >-I/home/mp27lupo/opt/chapel/runtime//include/qio
> >-I/home/mp27lupo/opt/chapel/runtime//include/atomics/intrinsics
> >-I/home/mp27lupo/opt/chapel/runtime//include/mem/dlmalloc
> > -I/home/mp27lupo/opt/chapel/third-party/utf8-decoder
> >-I/home/mp27lupo/opt/chapel/third-party/dlmalloc/include
> >-I/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/in
> >clude -I/home/mp27lupo/opt/chapel/runtime//include/hwloc/none
> >/tmp/chpl-mp27lupo-27468.deleteme/_main.c
> >g++
> >-L/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/li
> >b -Wl,-rpath
> >-Wl,/home/mp27lupo/opt/chapel/third-party/gmp/install/linux64-gnu-unknown/
> >lib
> >-L/home/mp27lupo/opt/chapel/third-party/re2/install/linux64-gnu-unknown/li
> >b -Wl,-rpath
> >-Wl,/home/mp27lupo/opt/chapel/third-party/re2/install/linux64-gnu-unknown/
> >lib
> > -o /tmp/chpl-mp27lupo-27468.deleteme/hello.tmp
> >-L/home/mp27lupo/opt/chapel/lib/linux64.gnu.arch-unknown.loc-flat.comm-gas
> >net.ibv.large.tasks-fifo.tmr-generic.mem-dlmalloc.atomics-intrinsics.none.
> >gmp.hwloc-none.re2.wide-struct.fs-none
> >/tmp/chpl-mp27lupo-27468.deleteme/hello.tmp.o
> >
> >/home/mp27lupo/opt/chapel/lib/linux64.gnu.arch-unknown.loc-flat.comm-gasne
> >t.ibv.large.tasks-fifo.tmr-generic.mem-dlmalloc.atomics-intrinsics.none.gm
> >p.hwloc-none.re2.wide-struct.fs-none/main.o -lchpl -lm -lpthread
> >-L/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown
> >/seg-large/nodbg/lib
> > -L/usr/lib64 -L/opt/mellanox/fca/lib -lgasnet-ibv-par -libverbs
> >-lfca -lpthread
> >-L/shared/apps/gcc/4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -lgcc -lm
> >
> >-L/home/mp27lupo/opt/chapel/third-party/dlmalloc/install/linux64-gnu-unkno
> >wn -ldlmalloc -lgmp -lre2
> > -lpthread
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapExchange_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0xcd): undefined reference to
> >`ompi_mpi_byte'
> >gasnet_bootstrap_mpi.c:(.text+0xf9): undefined reference to
> >`MPI_Allgather'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapBroadcast_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x132): undefined reference to
> >`ompi_mpi_byte'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapInit_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x17c): undefined reference to
> >`MPI_Initialized'
> >gasnet_bootstrap_mpi.c:(.text+0x1a9): undefined reference to `MPI_Init'
> >gasnet_bootstrap_mpi.c:(.text+0x1b5): undefined reference to
> >`ompi_mpi_comm_world'
> >gasnet_bootstrap_mpi.c:(.text+0x1ba): undefined reference to
> >`MPI_Comm_group'
> >gasnet_bootstrap_mpi.c:(.text+0x1c8): undefined reference to
> >`ompi_mpi_comm_world'
> >gasnet_bootstrap_mpi.c:(.text+0x1cd): undefined reference to
> >`MPI_Comm_create'
> >gasnet_bootstrap_mpi.c:(.text+0x1d5): undefined reference to
> >`MPI_Group_free'
> >gasnet_bootstrap_mpi.c:(.text+0x1e6): undefined reference to
> >`MPI_Comm_size'
> >gasnet_bootstrap_mpi.c:(.text+0x20a): undefined reference to
> >`MPI_Comm_rank'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapFini_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x24a): undefined reference to
> >`MPI_Comm_free'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapAbort_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x27e): undefined reference to `MPI_Abort'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapAlltoall_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x2c9): undefined reference to
> >`ompi_mpi_byte'
> >gasnet_bootstrap_mpi.c:(.text+0x2db): undefined reference to
> >`MPI_Alltoall'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapBroadcast_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x13a): undefined reference to `MPI_Bcast'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapFini_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x265): undefined reference to
> >`MPI_Finalize'
> >/home/mp27lupo/opt/chapel/third-party/gasnet/install/linux64-gnu-unknown/s
> >eg-large/nodbg/lib/libgasnet-ibv-par.a(gasnet_bootstrap_mpi-PAR.o): In
> >function `gasneti_bootstrapBarrier_mpi':
> >gasnet_bootstrap_mpi.c:(.text+0x2a8): undefined reference to `MPI_Barrier'
> >collect2: Fehler: ld gab 1 als Ende-Status zurück
> >gmake: *** [/tmp/chpl-mp27lupo-27468.deleteme/hello.tmp] Fehler 1
> >error: compiling generated source
> >
> >Am I doing anything wrong? I've tried to make a hello world mpi program
> >which worked fine... I have no clue why it can't find the MPI references
> >in gasnet.
> >
> >Thanks in advance and best regards,
> >Marco
> >
> >--------------------------------------------------------------------------
> >----
> >Dive into the World of Parallel Programming. The Go Parallel Website,
> >sponsored by Intel and developed in partnership with Slashdot Media, is
> >your
> >hub for all things parallel software development, from weekly thought
> >leadership blogs to news, videos, case studies, tutorials and more. Take a
> >look and join the conversation now.
> >http://goparallel.sourceforge.net/ <http://goparallel.sourceforge.net/>
> >_______________________________________________
> >Chapel-users mailing list
> >[email protected]
> >https://lists.sourceforge.net/lists/listinfo/chapel-users
> >
> >
> >
> >
> >
> >
> >
> >
>
>
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
