Hi Bilge and Kale --
Adding on to Greg's response: Another difference between Chapel targeting
Crays via ugni vs. GASnet is that in our ugni runtime we make use of
network atomics while our gasnet port does not since neither GASnet nor
SMSG support an interface to them. I don't think that we've put any
effort into trying to understand how much of our performance deltas are
due to atomics vs. more traditional data transfers -- so I can't quantify
the impact of this for you here -- but it's another factor to keep in
mind.
-Brad
On Thu, 5 Jan 2017, Greg Titus wrote:
Hello Bilge —
We do performance testing nightly using, among other things, 16-node Cray XC
runs. You can see the results here:
http://chapel.sourceforge.net/perf/16-node-xc/
In the left column you can select the configurations and which tests you want
to see. The configurations aren’t fully descriptive, but ‘gnu+ugni-qthreads’
and ‘gnu+gasnet-aries’ differ only in the Chapel runtime communication layer
implementation: the former uses a comm implementation based directly on ugni
and the latter uses a comm implementation based on GASNet and its ‘aries’
conduit. (Both use a gcc target compiler and Qthreads-based tasking.) So
those are the two you want to look at. Just select a suite or individual tests
you want to see and then click the ‘Display’ button. The ‘HPCC (time)’ suite
might be a good one to start with.
Generally you’ll find that the native ugni-based comm layer outperforms the
GASNet-based one, as you mentioned. Which one you’ll want to use depends on a
performance vs. usability tradeoff, as is so often the case in our business.
:-) The Chapel native one uses the ugni RDMA capability which can do both
GETs and PUTs, but requires us to register with the NIC any memory it will
reference directly (remote targets for PUTs and both remote sources and local
targets for GETs). The GASNet+aries one is based on the ugni SMSG (short
message) capability which can only do PUTs but doesn’t require registered
memory. It’s therefore a better match for GASNet’s communication model. The
ugni RDMA capability has better performance, which is why the Chapel native
ugni comm layer beats the GASNet-based one, but it has a more complicated
implementation and even user interface because of the need to register memory.
For simplicity, our expected situation is to pre-register the static data and a
fixed-size heap, both on hugepages (using a Cray PE hugepages shell module) and
bounce any (hopefully unlikely) references to unregistered memory through
trampoline buffers in registered memory. This in turn requires knowing the
heap size up front, so we guess at that but allow the user to set an
environment variable to override our guess. The GASNet-based layer doesn’t
have any of this complication with user-visible registration, so it’s
potentially easier on the users and certainly simpler to implement.
I can fill in more details but this should be enough to get you started on your
own decision process. Let me know if you have other questions!
greg
On Jan 5, 2017, at 11:34 AM, Bilge Acun
<[email protected]<mailto:[email protected]>> wrote:
Hello Chapel team,
I am from the Charm++ team and we would like to learn more about Chapel's
GasNet port as we are also considering implementing a GasNet communication
interface for Charm++. I have read your Reddit discussion where David writes
native ugni performs better than GasNet on Cray systems. Can you share with us
how much performance difference there is? What are the reasons for this
performance difference?
For Charm++, we have implementations of native interfaces for ugni, verbs,
pami, and some others to achieve the best performance. GasNet's portability is
very appealing, however we do not want to sacrifice from performance. Your
insights would be valuable for us.
Thanks,
--
Bilge Acun
PhD Candidate, Computer Science Department
University of Illinois at Urbana-Champaign
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org<http://SlashDot.org>!
http://sdm.link/slashdot_______________________________________________
Chapel-developers mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/chapel-developers
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
