Gilles, Sorry, I do not have any more details about NERSC’s usage.
Scott > On Mar 24, 2016, at 9:49 AM, Gilles Gouaillardet > <gilles.gouaillar...@gmail.com> wrote: > > Scott, > > out of curiosity ... > > generally speaking, and when HT is more efficient, how is it used ? > - flat MPI, with one task per thread > - Hybrid MPI+OpenMP, a task is bound to a core or socket, but never to a > thread > > Cheers, > > Gilles > > On Thursday, March 24, 2016, Atchley, Scott <atchle...@ornl.gov> wrote: > Hi Aurélien, > > I have said the same thing to many users over the years. Our colleagues at > NERSC, however, have found that 20% of their codes work better when using HT. > Some codes benefit from SMT2 (i.e. HT) and even SMT4 (available on Power8) in > order to provide enough latency hiding of memory accesses. > > As with everything in computer science, the answer is “It depends”. Try with > and without for each new generation of hardware. > > Scott > > > On Mar 23, 2016, at 4:32 PM, Aurélien Bouteiller <boute...@icl.utk.edu> > > wrote: > > > > To add to what Ralf said, you probably do not want to use Hyper Threads for > > HPC workloads, as that generally results in very poor performance (as you > > noticed). Set the number of slots to the number of real cores (not HT), > > that would yield optimal results 95% of the time. > > > > Aurélien > > > > -- > > Aurélien Bouteiller, Ph.D. ~~ https://icl.cs.utk.edu/~bouteill/ > > > >> Le 23 mars 2016 à 16:24, Ralph Castain <r...@open-mpi.org> a écrit : > >> > >> “Slots” are an abstraction commonly used by schedulers as a way of > >> indicating how many processes are allowed to run on a given node. It has > >> nothing to do with hardware, either cores or HTs. > >> > >> MPI programmers frequently like to bind a process to one or more hardware > >> assets (cores or HTs). Thus, you will see confusion in the community where > >> people mix the term “slot” with “cores” or “cpus”. This is unfortunate as > >> it the terms really do mean very different things. > >> > >> In OMPI, we chose to try and “help” the user by not requiring them to > >> specify detailed info in a hostfile. So if you don’t specify the number of > >> “slots” for a given node, we will sense the number of cores on that node > >> and set the slots to match that number. This best matches user > >> expectations today. > >> > >> If you do specify the number of slots, then we use that to guide the > >> desired number of processes assigned to each node. We then bind each of > >> those processes according to the user-provided guidance. > >> > >> HTH > >> Ralph > >> > >>> On Mar 23, 2016, at 9:35 AM, Federico Reghenzani > >>> <federico1.reghenz...@mail.polimi.it> wrote: > >>> > >>> Ok, I've investigated further today, it seems "--map-by hwthread" does > >>> not remove the problem. However, if I specified in the hostfile "node0 > >>> slots=32" it runs really slower than specifying only "node0". In both > >>> cases I run mpirun with -np 32. So I'm quite sure I didn't understand > >>> what slots are. > >>> > >>> __ > >>> Federico Reghenzani > >>> M.Eng. Student @ Politecnico di Milano > >>> Computer Science and Engineering > >>> > >>> > >>> > >>> 2016-03-22 18:56 GMT+01:00 Federico Reghenzani > >>> <federico1.reghenz...@mail.polimi.it>: > >>> Hi guys, > >>> > >>> I'm really confused about slots in resource allocation: I thought that > >>> slots are the number of processes spawnable in a certain node, so it > >>> should correspond to the number of Processing Elements of the node. For > >>> example, on each of my nodes I have 2 processors, total 16 cores with > >>> hyperthreading, so a total of 32 processing elements per node (i.e. 32 hw > >>> threads). However, considering a single node, passing in the hostfile 32 > >>> slots and requesting "-np 32" results is a performance degradation of 20x > >>> slower than using only "-np 16". The problem disappears specifing > >>> --map-by hwthread. > >>> > >>> Investigating on the problem I found these counterintuitive things: > >>> - here is stated "slots are Open MPI's representation of how many > >>> processors are available" > >>> - here is stated "Slots indicate how many processes can potentially > >>> execute on a node. For best performance, the number of slots may be > >>> chosen to be the number of cores on the node or the number of processor > >>> sockets" > >>> - I tried to remove the slots information from the hostfile, so according > >>> to this should be interpreted as "1", but it spawns anyway 32 processes > >>> - I'm not sure what --map-by and --rank-by do > >>> > >>> In custom RAS we are developing, what we have to send to mpirun? The > >>> number of processor sockets, the number of cores or the number of > >>> hwthread available? How --map-by and --rank-by affect the spawn policy? > >>> > >>> > >>> Thank you! > >>> > >>> > >>> OFFTOPIC: is someone going to EuroMPI 2016 in September? We will be there > >>> to present our migration technique. > >>> > >>> > >>> Cheers, > >>> Federico > >>> > >>> __ > >>> Federico Reghenzani > >>> M.Eng. Student @ Politecnico di Milano > >>> Computer Science and Engineering > >>> > >>> > >>> > >>> _______________________________________________ > >>> devel mailing list > >>> de...@open-mpi.org > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > >>> Link to this post: > >>> http://www.open-mpi.org/community/lists/devel/2016/03/18723.php > >> > >> _______________________________________________ > >> devel mailing list > >> de...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > >> Link to this post: > >> http://www.open-mpi.org/community/lists/devel/2016/03/18724.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > > http://www.open-mpi.org/community/lists/devel/2016/03/18725.php > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/03/18726.php > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2016/03/18727.php
