Thanks, Greg.
It looks like I just got into this little pitfall based on my setting. I
actually tried to set "CHPL_RT_NUM_THREADS_PER_LOCALE" to be the number of
physical cores on each node. But that caused the program to hang and get
terminated due to time expiration.
On Tue, Jul 25, 2017 at 11:59 AM, Greg Titus <g...@cray.com> wrote:
> Hi Michael, Hui --
>
> There's a little bit of interface (Elliot is familiar with it) through
> which comm layers can tell tasking layers to create no more than a certain
> number of pthreads. The gasnet comm layer uses this to limit the tasking
> layer to no more than 256 pthreads, and tasks=fifo is supposed to pay
> attention to this. It sounds like this little bit of interface has gotten
> broken at some point. (Because here we have GASNet barking that the fifo
> tasking layer is trying to create more.) Perhaps with help from Elliot you
> can look into what's gone wrong there -- user code shouldn't be able to
> drive the runtime into producing this error message.
>
> I can't help much beyond this, sorry, I'm traveling and the Nashville
> airport wifi is horrid.
>
> greg
>
> ________________________________________
> From: Michael Ferguson <mfergu...@cray.com>
> Sent: Tuesday, July 25, 2017 9:36 AM
> To: Hui Zhang; Chapel Sourceforge Developers List
> Subject: Re: [Chapel-developers] Too many simultaneous local client threads
>
> Hi -
>
> >Hello,
> >
> >
> >
> >I'm running the hpl benchmark with the latest Chapel release 1.15 on
> >multi-locale. It's running OK on 2,4,8,16 nodes, until 32 nodes, where it
> >reports the error:
> >*** FATAL ERROR: GASNet Extended API: Too many simultaneous local client
> >threads (limit=256). To raise this limit, configure GASNet using
> >--with-max-pthreads-per-node=N.
> >
> >
> >Is it reasonable to have this error when running on 32 nodes?
>
> I'm not familiar with that error but it sounds like something to do
> with the number of threads you are running per locale. How many
> cores do the compute nodes have? Is it possible that all of the
> tasks are accidentally running on 1 node?
>
> Have you tried running the Chapel program with the -v option
> to observe the job launcher? Does hello6-taskpar-dist.chpl print
> out different machine names for each locale when you run it?
>
> > I read this
> >
> >http://chapel.cray.com/docs/1.15/usingchapel/tasks.html#id2
> ><http://chapel.cray.com/docs/1.15/usingchapel/tasks.html#id2> In order
> >to run it on 32 nodes, shall I set
> >CHPL_RT_NUM_THREADS_PER_LOCALE
> >and rebuild Chapel? Or in what other way? Thanks
> >
> >
> >Here's my env:
> >
> >CHPL_TARGET_PLATFORM: linux64
> >CHPL_TARGET_COMPILER: gnu
> >CHPL_TARGET_ARCH: native *
> >CHPL_LOCALE_MODEL: flat
> >CHPL_COMM: gasnet *
> > CHPL_COMM_SUBSTRATE: ibv *
> > CHPL_GASNET_SEGMENT: large
> >CHPL_TASKS: fifo *
> >CHPL_LAUNCHER: gasnetrun_ibv *
> >CHPL_TIMERS: generic
> >CHPL_UNWIND: none
> >CHPL_MEM: jemalloc
> >CHPL_MAKE: gmake
> >CHPL_ATOMICS: intrinsics
> > CHPL_NETWORK_ATOMICS: none
> >CHPL_GMP: gmp
> >CHPL_HWLOC: none
> >CHPL_REGEXP: re2
> >CHPL_WIDE_POINTERS: struct
> >CHPL_AUX_FILESYS: none
> >
> >
> >
> >​Besides, I'm a little confused of "local" statement. According to
> >http://chapel.cray.com/docs/1.15/technotes/local.html?highlight=local, is
> >it just used to assert the communication free statements in the block? Or
> >it's used as an performance
> > optimization?
>
> local blocks are a performance optimization. They do assert that
> the operations are local but might not do so with --fast.
>
> -michael
>
> >
> >
> >
> >Thanks
> >
> >
> >--
> >Best regards
> >
> >
> >Hui Zhang
> >
> >
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Chapel-developers mailing list
> Chapel-developers@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/chapel-developers
>
--
Best regards
Hui Zhang
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Chapel-developers mailing list
Chapel-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/chapel-developers
