Not really - just consistent with the other cmd line options.
> On Jan 6, 2016, at 12:58 PM, Nick Papior <nickpap...@gmail.com> wrote:
>
> It was just that when I started using map-by I didn't get why:
> ppr:2
> but
> PE=2
> I would at least have expected:
> ppr=2:PE=2
> or
> ppr:2:PE:2
> ?
> Does this have a reason?
>
> 2016-01-06 21:54 GMT+01:00 Ralph Castain <r...@open-mpi.org
> <mailto:r...@open-mpi.org>>:
> <LOL> ah yes, “r” = “resource”!! Thanks for the reminder :-)
>
> The difference in delimiter is just to simplify parsing - we can “split” the
> string on colons to separate out the options, and then use “=“ to set the
> value. Nothing particularly significant about the choice.
>
>
>> On Jan 6, 2016, at 12:48 PM, Nick Papior <nickpap...@gmail.com
>> <mailto:nickpap...@gmail.com>> wrote:
>>
>> Your are correct. "socket" means that the resource is socket, "ppr:2" means
>> 2 processes per resource.
>> PE=<n> is Processing Elements per process.
>>
>> Perhaps the dev's can shed some light on why PE uses "=" and ppr has ":" as
>> delimiter for resource request?
>>
>> This "old" slide show from Jeff shows the usage (although the input have
>> changed since 1.7):
>> http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation
>>
>> <http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation>
>>
>>
>> 2016-01-06 21:33 GMT+01:00 Matt Thompson <fort...@gmail.com
>> <mailto:fort...@gmail.com>>:
>> A ha! The Gurus know all. The map-by was the magic sauce:
>>
>> (1176) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 -map-by
>> ppr:2:socket:pe=7 ./hello-hybrid.x | sort -g -k 18
>> Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU 0
>> Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU 1
>> Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU 2
>> Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU 3
>> Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU 4
>> Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU 5
>> Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU 6
>> Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU 7
>> Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU 8
>> Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU 9
>> Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU 10
>> Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU 11
>> Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU 12
>> Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU 13
>> Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU 14
>> Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU 15
>> Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU 16
>> Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU 17
>> Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU 18
>> Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU 19
>> Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU 20
>> Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU 21
>> Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU 22
>> Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU 23
>> Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU 24
>> Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU 25
>> Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU 26
>> Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU 27
>>
>> So, a question: what does "ppr" mean? The man page seems to accept it as an
>> axiom of Open MPI:
>>
>> --map-by <foo>
>> Map to the specified object, defaults to socket. Supported
>> options include slot, hwthread, core, L1cache, L2cache, L3cache, socket,
>> numa,
>> board, node, sequential, distance, and ppr. Any object can
>> include modifiers by adding a : and any combination of PE=n (bind n
>> processing
>> elements to each proc), SPAN (load balance the processes
>> across the allocation), OVERSUBSCRIBE (allow more processes on a node than
>> pro‐
>> cessing elements), and NOOVERSUBSCRIBE. This includes PPR,
>> where the pattern would be terminated by another colon to separate it from
>> the
>> modifiers.
>>
>> Is it an acronym/initialism? From some experimenting it seems to be
>> ppr:2:socket means 2 processes per socket? And pe=7 means leave 7 processes
>> between them? Is that about right?
>>
>> Matt
>>
>> On Wed, Jan 6, 2016 at 3:19 PM, Ralph Castain <r...@open-mpi.org
>> <mailto:r...@open-mpi.org>> wrote:
>> I believe he wants two procs/socket, so you’d need ppr:2:socket:pe=7
>>
>>
>>> On Jan 6, 2016, at 12:14 PM, Nick Papior <nickpap...@gmail.com
>>> <mailto:nickpap...@gmail.com>> wrote:
>>>
>>> I do not think KMP_AFFINITY should affect anything in OpenMPI, it is an MKL
>>> env setting? Or am I wrong?
>>>
>>> Note that these are used in an environment where openmpi automatically gets
>>> the host-file. Hence they are not present.
>>> With intel mkl and openmpi I got the best performance using these, rather
>>> long flags:
>>>
>>> export KMP_AFFINITY=verbose,compact,granularity=core
>>> export KMP_STACKSIZE=62M
>>> export KMP_SETTINGS=1
>>>
>>> def_flags="--bind-to core -x OMP_PROC_BIND=true --report-bindings"
>>> def_flags="$def_flags -x KMP_AFFINITY=$KMP_AFFINITY"
>>>
>>> # in your case 7:
>>> ONP=7
>>> flags="$def_flags -x MKL_NUM_THREADS=$ONP -x MKL_DYNAMIC=FALSE"
>>> flags="$flags -x OMP_NUM_THREADS=$ONP -x OMP_DYNAMIC=FALSE"
>>> flags="$flags -x KMP_STACKSIZE=$KMP_STACKSIZE"
>>> flags="$flags --map-by ppr:1:socket:pe=7"
>>>
>>> then run your program:
>>>
>>> mpirun $flags <app>
>>>
>>> A lot of the option flags are duplicated (and strictly not needed), but I
>>> provide them for easy testing changes.
>>> Surely this is application dependent, but for my case it was performing
>>> really well.
>>>
>>>
>>> 2016-01-06 20:48 GMT+01:00 Erik Schnetter <schnet...@gmail.com
>>> <mailto:schnet...@gmail.com>>:
>>> Setting KMP_AFFINITY will probably override anything that OpenMPI
>>> sets. Can you try without?
>>>
>>> -erik
>>>
>>> On Wed, Jan 6, 2016 at 2:46 PM, Matt Thompson <fort...@gmail.com
>>> <mailto:fort...@gmail.com>> wrote:
>>> > Hello Open MPI Gurus,
>>> >
>>> > As I explore MPI-OpenMP hybrid codes, I'm trying to figure out how to do
>>> > things to get the same behavior in various stacks. For example, I have a
>>> > 28-core node (2 14-core Haswells), and I'd like to run 4 MPI processes
>>> > and 7
>>> > OpenMP threads. Thus, I'd like the processes to be 2 processes per socket
>>> > with the OpenMP threads laid out on them. Using a "hybrid Hello World"
>>> > program, I can achieve this with Intel MPI (after a lot of testing):
>>> >
>>> > (1097) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4
>>> > ./hello-hybrid.x | sort -g -k 18
>>> > srun.slurm: cluster configuration lacks support for cpu binding
>>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU 0
>>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU 1
>>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU 2
>>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU 3
>>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU 4
>>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU 5
>>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU 6
>>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU 7
>>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU 8
>>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU 9
>>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU 10
>>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU 11
>>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU 12
>>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU 13
>>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU 14
>>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU 15
>>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU 16
>>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU 17
>>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU 18
>>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU 19
>>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU 20
>>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU 21
>>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU 22
>>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU 23
>>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU 24
>>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU 25
>>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU 26
>>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU 27
>>> >
>>> > Other than the odd fact that Process #0 seemed to start on Socket #1 (this
>>> > might be an artifact of how I'm trying to detect the CPU I'm on), this
>>> > looks
>>> > reasonable. 14 threads on each socket and each process is laying out its
>>> > threads in a nice orderly fashion.
>>> >
>>> > I'm trying to figure out how to do this with Open MPI (version 1.10.0) and
>>> > apparently I am just not quite good enough to figure it out. The closest
>>> > I've gotten is:
>>> >
>>> > (1155) $ env OMP_NUM_THREADS=7 KMP_AFFINITY=compact mpirun -np 4 -map-by
>>> > ppr:2:socket ./hello-hybrid.x | sort -g -k 18
>>> > Hello from thread 0 out of 7 from process 0 out of 4 on borgo035 on CPU 0
>>> > Hello from thread 0 out of 7 from process 1 out of 4 on borgo035 on CPU 0
>>> > Hello from thread 1 out of 7 from process 0 out of 4 on borgo035 on CPU 1
>>> > Hello from thread 1 out of 7 from process 1 out of 4 on borgo035 on CPU 1
>>> > Hello from thread 2 out of 7 from process 0 out of 4 on borgo035 on CPU 2
>>> > Hello from thread 2 out of 7 from process 1 out of 4 on borgo035 on CPU 2
>>> > Hello from thread 3 out of 7 from process 0 out of 4 on borgo035 on CPU 3
>>> > Hello from thread 3 out of 7 from process 1 out of 4 on borgo035 on CPU 3
>>> > Hello from thread 4 out of 7 from process 0 out of 4 on borgo035 on CPU 4
>>> > Hello from thread 4 out of 7 from process 1 out of 4 on borgo035 on CPU 4
>>> > Hello from thread 5 out of 7 from process 0 out of 4 on borgo035 on CPU 5
>>> > Hello from thread 5 out of 7 from process 1 out of 4 on borgo035 on CPU 5
>>> > Hello from thread 6 out of 7 from process 0 out of 4 on borgo035 on CPU 6
>>> > Hello from thread 6 out of 7 from process 1 out of 4 on borgo035 on CPU 6
>>> > Hello from thread 0 out of 7 from process 2 out of 4 on borgo035 on CPU 14
>>> > Hello from thread 0 out of 7 from process 3 out of 4 on borgo035 on CPU 14
>>> > Hello from thread 1 out of 7 from process 2 out of 4 on borgo035 on CPU 15
>>> > Hello from thread 1 out of 7 from process 3 out of 4 on borgo035 on CPU 15
>>> > Hello from thread 2 out of 7 from process 2 out of 4 on borgo035 on CPU 16
>>> > Hello from thread 2 out of 7 from process 3 out of 4 on borgo035 on CPU 16
>>> > Hello from thread 3 out of 7 from process 2 out of 4 on borgo035 on CPU 17
>>> > Hello from thread 3 out of 7 from process 3 out of 4 on borgo035 on CPU 17
>>> > Hello from thread 4 out of 7 from process 2 out of 4 on borgo035 on CPU 18
>>> > Hello from thread 4 out of 7 from process 3 out of 4 on borgo035 on CPU 18
>>> > Hello from thread 5 out of 7 from process 2 out of 4 on borgo035 on CPU 19
>>> > Hello from thread 5 out of 7 from process 3 out of 4 on borgo035 on CPU 19
>>> > Hello from thread 6 out of 7 from process 2 out of 4 on borgo035 on CPU 20
>>> > Hello from thread 6 out of 7 from process 3 out of 4 on borgo035 on CPU 20
>>> >
>>> > Obviously not right. Any ideas on how to help me learn? The man mpirun
>>> > page
>>> > is a bit formidable in the pinning part, so maybe I've missed an obvious
>>> > answer.
>>> >
>>> > Matt
>>> > --
>>> > Matt Thompson
>>> >
>>> > Man Among Men
>>> > Fulcrum of History
>>> >
>>> >
>>> > _______________________________________________
>>> > users mailing list
>>> > us...@open-mpi.org <mailto:us...@open-mpi.org>
>>> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> > <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>> > Link to this post:
>>> > http://www.open-mpi.org/community/lists/users/2016/01/28217.php
>>> > <http://www.open-mpi.org/community/lists/users/2016/01/28217.php>
>>>
>>>
>>>
>>> --
>>> Erik Schnetter <schnet...@gmail.com <mailto:schnet...@gmail.com>>
>>> http://www.perimeterinstitute.ca/personal/eschnetter/
>>> <http://www.perimeterinstitute.ca/personal/eschnetter/>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/01/28218.php
>>> <http://www.open-mpi.org/community/lists/users/2016/01/28218.php>
>>>
>>>
>>>
>>> --
>>> Kind regards Nick
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/01/28219.php
>>> <http://www.open-mpi.org/community/lists/users/2016/01/28219.php>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/01/28221.php
>> <http://www.open-mpi.org/community/lists/users/2016/01/28221.php>
>>
>>
>>
>> --
>> Matt Thompson
>> Man Among Men
>> Fulcrum of History
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/01/28223.php
>> <http://www.open-mpi.org/community/lists/users/2016/01/28223.php>
>>
>>
>>
>> --
>> Kind regards Nick
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org <mailto:us...@open-mpi.org>
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/01/28224.php
>> <http://www.open-mpi.org/community/lists/users/2016/01/28224.php>
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org <mailto:us...@open-mpi.org>
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> <http://www.open-mpi.org/mailman/listinfo.cgi/users>
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/01/28226.php
> <http://www.open-mpi.org/community/lists/users/2016/01/28226.php>
>
>
>
> --
> Kind regards Nick
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/01/28227.php
