Hi Brad and Greg,
Thank you for sharing the performance results and your comments, that's
helpful.
I see nearly 6x better performance with native ugni on FFT, 5x on miniMD,
2x on Lulesh, HPL; that's quite significant. I will forward your comments
to our group for discussion.
Sincerely,
~Bilge
On 5 January 2017 at 16:05, Brad Chamberlain <[email protected]> wrote:
>
> Hi Bilge and Kale --
>
> Adding on to Greg's response: Another difference between Chapel targeting
> Crays via ugni vs. GASnet is that in our ugni runtime we make use of
> network atomics while our gasnet port does not since neither GASnet nor
> SMSG support an interface to them. I don't think that we've put any
> effort into trying to understand how much of our performance deltas are
> due to atomics vs. more traditional data transfers -- so I can't quantify
> the impact of this for you here -- but it's another factor to keep in
> mind.
>
> -Brad
>
>
> On Thu, 5 Jan 2017, Greg Titus wrote:
>
> > Hello Bilge —
> >
> > We do performance testing nightly using, among other things, 16-node
> Cray XC runs. You can see the results here:
> > http://chapel.sourceforge.net/perf/16-node-xc/
> >
> > In the left column you can select the configurations and which tests you
> want to see. The configurations aren’t fully descriptive, but
> ‘gnu+ugni-qthreads’ and ‘gnu+gasnet-aries’ differ only in the Chapel
> runtime communication layer implementation: the former uses a comm
> implementation based directly on ugni and the latter uses a comm
> implementation based on GASNet and its ‘aries’ conduit. (Both use a gcc
> target compiler and Qthreads-based tasking.) So those are the two you want
> to look at. Just select a suite or individual tests you want to see and
> then click the ‘Display’ button. The ‘HPCC (time)’ suite might be a good
> one to start with.
> >
> > Generally you’ll find that the native ugni-based comm layer outperforms
> the GASNet-based one, as you mentioned. Which one you’ll want to use
> depends on a performance vs. usability tradeoff, as is so often the case in
> our business. :-) The Chapel native one uses the ugni RDMA capability
> which can do both GETs and PUTs, but requires us to register with the NIC
> any memory it will reference directly (remote targets for PUTs and both
> remote sources and local targets for GETs). The GASNet+aries one is based
> on the ugni SMSG (short message) capability which can only do PUTs but
> doesn’t require registered memory. It’s therefore a better match for
> GASNet’s communication model. The ugni RDMA capability has better
> performance, which is why the Chapel native ugni comm layer beats the
> GASNet-based one, but it has a more complicated implementation and even
> user interface because of the need to register memory. For simplicity, our
> expected situation is to pre-register the static data and a fixed-size
> heap, both on hugepages (using a Cray PE hugepages shell module) and bounce
> any (hopefully unlikely) references to unregistered memory through
> trampoline buffers in registered memory. This in turn requires knowing the
> heap size up front, so we guess at that but allow the user to set an
> environment variable to override our guess. The GASNet-based layer doesn’t
> have any of this complication with user-visible registration, so it’s
> potentially easier on the users and certainly simpler to implement.
> >
> > I can fill in more details but this should be enough to get you started
> on your own decision process. Let me know if you have other questions!
> >
> > greg
> >
> >
> > On Jan 5, 2017, at 11:34 AM, Bilge Acun <[email protected]<mailto:acu
> [email protected]>> wrote:
> >
> > Hello Chapel team,
> >
> > I am from the Charm++ team and we would like to learn more about
> Chapel's GasNet port as we are also considering implementing a GasNet
> communication interface for Charm++. I have read your Reddit discussion
> where David writes native ugni performs better than GasNet on Cray systems.
> Can you share with us how much performance difference there is? What are
> the reasons for this performance difference?
> >
> > For Charm++, we have implementations of native interfaces for ugni,
> verbs, pami, and some others to achieve the best performance. GasNet's
> portability is very appealing, however we do not want to sacrifice from
> performance. Your insights would be valuable for us.
> >
> > Thanks,
> >
> > --
> > Bilge Acun
> > PhD Candidate, Computer Science Department
> > University of Illinois at Urbana-Champaign
> > ------------------------------------------------------------
> ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org<http://SlashDot.org>!
> http://sdm.link/slashdot_______________________________________________
> > Chapel-developers mailing list
> > [email protected]<mailto:Chapel-
> [email protected]>
> > https://lists.sourceforge.net/lists/listinfo/chapel-developers
> >
> >
--
*Bilge Acun*
*PhD Candidate, **Computer Science Department*
*University of Illinois at Urbana-Champaign*
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
