Hi Jintack --
Great, glad to hear it! The only thing that gives me slight pause here is
that both of your messages seem to be printing the same node ID. This may
be correct depending on how you've specified your compute nodes to GASNet
(e.g., GASNet has mechanisms for oversubscribing compute nodes, so if
you're doing that, things look correct. If you meant for each locale to
be mapped to its own compute node, it looks like something may not be
working correctly yet...). Usually the compute nodes are specified to the
GASNet launcher ('gasnetrun_ofi' here) via a file or command-line
argument, or sometimes implicitly through job queueing environment
variables.
Note that the SPMD images that make up a Chapel program tend to assume
that they own all of the cores of a compute node, so may tend not to
perform so well when run oversubscribed.
-Brad
On Fri, 22 Jul 2016, Lim, Jintack wrote:
Thanks Brad.
With the Makefile.mpi-gnu, I'm able to compile the hello6-taskpar-dist.chpl
without link errors.
Since I didn't set CHPL_LAUNCHER, I just got one binary. I ran the following
command and it looks like it runs well.. right??
[VM-45
~/chapel_4/chapel-1.13.1]$third-party/gasnet/install/linux64-mpi-gnu-unknown/seg-everything/nodbg/bin/gasnetrun_ofi
-np 2 ./hello6-taskpar-dist -nl 2
Hello, world! (from locale 0 of 2 named simpl-l4tm-vm45)
Hello, world! (from locale 1 of 2 named simpl-l4tm-vm45)
Best,
Jintack
-----Original Message-----
From: Brad Chamberlain [mailto:[email protected]]
Sent: Thursday, July 21, 2016 5:07 PM
To: Lim, Jintack <[email protected]>
Cc: Paul Hargrove <[email protected]>; [email protected]; Michael
Ferguson <[email protected]>
Subject: RE: [upc-devel] GASNet ofi-conduit in the context of Chapel
Hi Jintack --
Michael's suggestion to use 'mpi-gnu' as the target compiler unfortunately only
works from the master branch of our GitHub repository, as that option was added
after the 1.13 releases. That said, I wouldn't be surprised if you could grab
the following one file and drop it in $CHPL_HOME/make/compiler/ to enable it
for 1.13:
ub.com/chapel-lang/chapel/blob/master/make/compiler/Makefile.mpi-gnu
Meanwhile, I'm going to see if I can determine why our Makefiles aren't picking
up GASNET_LD as their link step as I expected.
-Brad
On Thu, 21 Jul 2016, Lim, Jintack wrote:
Hi Brad, Michael and Paul,
I really appreciate your helpful comments.
This is the information with make –f <dir>/Makefile print-commands.
CC is /usr/bin/gcc
LD is g++
GASNET_LD is /usr/bin/mpicc
GASNET_INC_MAKEFILE is
/home/limjint/chapel_4/chapel-1.13.1/third-party/gasnet/install/linux6
4-gnu-unknown/seg-everything/nodbg/include/ofi-conduit/ofi-par.mak
This is the <dir>/Makefile.
http://pastebin.ubuntu.com/20377401/
I can see that CHPL_TARGET_COMPILER=gnu.
I also set CHPL_TARGET_COMPILER=mpi-gnu and compiled Chapel again.
I’m not sure MPI is configured to use gcc, but what I got while compiling
Chapel is something like this.
Warning: Unknown compiler: "mpi-gnu"
Warning: No valid option found: arch="native" compiler="mpi-gnu"
version="CompVersion(major=4, minor=9, revision=2, build=0)"
http://pastebin.ubuntu.com/20380446/
I also got some trivial compile errors and permission errors before above
error, and just worked around.
http://pastebin.ubuntu.com/20377624/
http://pastebin.ubuntu.com/20378961/
Best,
Jintack
From: Paul Hargrove [mailto:[email protected]]
Sent: Thursday, July 21, 2016 3:59 PM
To: Michael Ferguson <[email protected]>
Cc: Lim, Jintack <[email protected]>;
[email protected]
Subject: Re: [upc-devel] GASNet ofi-conduit in the context of Chapel
Brad and Michael,
It is not clear from your answers if you fully understood Jintack's situation.
However, *I* may be mistaken if the details you skipped over simply aren't
relevant for reasons unknown to me.
Jintack is *not* using mpi-conduit nor is GASNET_CC an MPI compiler.
The conduit in question is ofi-conduit, which is using mpirun as its default
spawner.
At least ibv, mxm, ofi and psm conduits support launching via ssh or by mpirun.
By default the configure script enables both of these assuming an mpicc is
found.
If mpi-based spawning is enabled at configure time then *one* .c file is
compiled with mpicc.
All the rest of the GASNet conduit code is compiled with a non-MPI C compiler.
However, due to the one MPI object file the .mak files for those conduits will
set their GASNET_LD=mpicc.
I am fairly confident that Chapel must already deal with $(GASNET_LD) as
distinct from $(GASNET_CC) since udp-conduit needs a C++ linker.
I suspect one or more of your suggestions may lead to a work-around
(CHPL_TARGET_COMPILER=mpi-gnu in particular).
However, I am suspicious of the handling of the Makefile fragments.
So, I think Brad's make command with all the print-SOMETHINGs is important to
root out the problem.
Is it possible that the problem is as simple as Chapel needing "ofi" added to
some list of conduits somewhere?
-Paul
On Thu, Jul 21, 2016 at 3:28 PM, Michael Ferguson
<[email protected]<mailto:[email protected]>> wrote:
Hi -
You could also try using
export CHPL_TARGET_COMPILER=mpi-gnu
Assuming you gave MPI configured to use GCC. If that's the case, if
CHPL_TARGET_COMPILER=mpi-gnu when you build Chapel and use it, you
should see mpicc calls as the linker.
Brad had lots of other good ideas for how to troubleshoot this problem.
Cheers,
-michael
On 7/20/16, 6:10 PM, "Lim, Jintack"
<[email protected]<mailto:[email protected]>> wrote:
Hi GASNet and Chapel community,
I'm trying to run Chapel with GASNet ofi-conduit.
I got some link errors last time while I was running "make check"
after compiled Chapel.
I got feedback from Paul Hargrove and Greg Titus but I still can't
resolve it.
http://paste.ubuntu.com/19312464/
I set CHPL_LAUNCHER=gasnetrun_mpi as Greg suggested.
(Paul suggested to try to configure using without-mpi-cc GASNet
option, but it looks like we need mpi based on the chapel lanucher
setting above?)
Paul said it looks like Chapel is not using mpicc as its linker.
I checked that gasnet_bootstrap_mpi.c was compiled with mpicc while
building Chapel.
When building libgasnet-ofi-par.a, gcc is used, but this is the same
when compiling GASNet only.
So I wonder if I need to configure Chapel with some flags or should I
put some link option somewhere?
Best,
Jintack
From: Paul Hargrove
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, July 13, 2016 6:09 PM
To: Lim, Jintack <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>; Jeff Hammond
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]
rge.net>
Subject: Re: [upc-devel] GASNet ofi-conduit in the context of Chapel
Jintack,
The environment variable GASNET_OFI_SPAWNER is used at
application-launch to select MPI vs SSH if both were available at
build time. So, it has no connection to the link failure. Your
environment variable settings seem to me to be correct.
The linkage errors seem to indicate that Chapel is not using mpicc as
its linker (ignoring the $(GASNET_LD) setting).
That is something you will need to take up with the Chapel team if
you require mpi-based spawning.
The simplest approach to getting past the link problem would be to
configure using --without-mpi-cc.
Then SSH-based spawning will be your only option and setting of
GASNET_OFI_SPAWNER will not be needed.
If you have a PMI-based spawner (including hydra from mpich or srun
from SLURM), then you may also be able to launch GASNet applications
that way (not using the gasnetrun_ofi script). You might need some
additional configure options to help GASNet locate libpmi - let me
know if you need help pursing that option.
-Paul
On Wed, Jul 13, 2016 at 5:42 PM, Lim, Jintack
<[email protected]<mailto:[email protected]>>
wrote:
Hi all,
When I was compiling a Chapel hello world application on top of
GASNet ofi-conduit, I got some GASNet link errors related to MPI.
http://paste.ubuntu.com/19312464/
I got this either I set GASNET_OFI_SPAWNER to mpi or ssh.
GASNet version that is included in the Chapel source tree is 1.26.0.
There are instructions to set GASNET environment variables for
GASNet-udp and ibv conduit in the Chapel documentation
(http://chapel.cray.com/docs/1.13/usingchapel/multilocale.html), but
I couldn’t find one for ofi-conduit.
Could somebody shed some light on how to fix this problem? (such as
how to set CHPL_LAUNCHER or any other GASNET environment variables to
fix the link error?)
What I did is
- CHPL_COMM=gasnet
- CHPL_COMM_SUBSTRATE=ofi
- GASNET_OFI_SPAWNER= mpi or ssh
-GASNET_SSH_SERVERS=”list of my servers”
Please let me know if you need further information.
Best,
Jintack
From: Jeff Hammond
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, July 13, 2016 5:19 PM
To: Lim, Jintack <[email protected]<mailto:[email protected]>>
Cc:
[email protected]<mailto:[email protected]
rge.net>
Subject: Re: Chapel on GASNet ofi-conduit
I had trouble with Gasnet OFI in the context of UPC. You might want
post this question to the Berkeley UPC list so the Gasnet team sees
it. They will know the launcher issues better than the Chapel team
(no disrespect intended).
Jeff
On Wednesday, July 13, 2016, Lim, Jintack
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I’m trying to install Chapel using GASNet ofi-conduit. I was able to
compile Chapel itself, however I couldn’t compile
hello6-taskpar-dist.chpl because of link errors related to MPI.
Here’s the error message.
http://paste.ubuntu.com/19312464/
There are only instructions for GASNet-udp and ibv conduit in the
documentation
(http://chapel.cray.com/docs/1.13/usingchapel/multilocale.html),
so I wonder how to set environment variables (such as launcher) for
ofi conduit.
Any help would be greatly appreciated.
Best,
Jintack
--
Jeff Hammond
[email protected]<mailto:[email protected]>
http://jeffhammond.github.io/
_______________________________________________
upc-devel mailing list
[email protected]<mailto:[email protected]>
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/upc-devel
--
Paul H. Hargrove
[email protected]<mailto:[email protected]>
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department Tel:
+1-510-495-2352<tel:%2B1-510-495-2352>
Lawrence Berkeley National Laboratory Fax:
+1-510-486-6900<tel:%2B1-510-486-6900>
--
Paul H. Hargrove
[email protected]<mailto:[email protected]>
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity planning
reports.http://sdm.link/zohodev2dev
_______________________________________________
Chapel-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-users
