Re: [OMPI devel] [OMPI] issue with mpirun

2019-07-12 Thread Ralph Castain via devel 
Again, I have no knowledge of what this program is supposed to do. I would have 
thought it would only print once as there is only one answer, but I don't know 
the code. I'd suggest looking to see where it prints.


On Jul 12, 2019, at 6:32 AM, Dangeti Tharun kumar mailto:cs15mtech11...@iith.ac.in> > wrote:

Program is a hybrid OpenMP and OpenMPI matrix multiplication.
Time is 0.004174
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
This is the program output. If you see, it is just printed only once by the 
second case.

On Fri, Jul 12, 2019 at 6:00 PM Ralph Castain via devel 
mailto:devel@lists.open-mpi.org> > wrote:
Afraid I don't know anything about that program, but it looks like it is 
printing the same number of times in both cases. It only appears to be more in 
the first case because the line wraps due to the number of PUs in the list


On Jul 12, 2019, at 3:00 AM, Dangeti Tharun kumar mailto:cs15mtech11...@iith.ac.in> > wrote:

Thanks, Ralph.

Why is the output of the program(mm-llvm.out) being run is printed only once, 
while the mpirun from intel prints as many times as mentioned in the command 
line? 


On Thu, Jul 11, 2019 at 11:08 PM Ralph Castain via devel 
mailto:devel@lists.open-mpi.org> > wrote:
Because OMPI binds to core by default when np=2. If you have an OpenMP process, 
you want to add “--bind-to numa" to your mpirun cmd line.


On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel 
mailto:devel@lists.open-mpi.org> > wrote:


Hi Devs,

I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2 
socket machine.
The following is the output of mpirun from ICC(intel c compiler package) and 
mpirun from openmpi:

Why the CPU topology(highlighted below) identified by both of them are 
different? Not sure, if this behavior is correct.

$>intel/bin/mpirun -np 2 ./mm-llvm.out

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc set

Re: [OMPI devel] [OMPI] issue with mpirun

2019-07-12 Thread Dangeti Tharun kumar via devel 
Program is a hybrid OpenMP and OpenMPI matrix multiplication.
Time is 0.004174
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
This is the program output. If you see, it is just printed only once by the
second case.

On Fri, Jul 12, 2019 at 6:00 PM Ralph Castain via devel <
devel@lists.open-mpi.org> wrote:

> Afraid I don't know anything about that program, but it looks like it is
> printing the same number of times in both cases. It only appears to be more
> in the first case because the line wraps due to the number of PUs in the
> list
>
>
> On Jul 12, 2019, at 3:00 AM, Dangeti Tharun kumar <
> cs15mtech11...@iith.ac.in> wrote:
>
> Thanks, Ralph.
>
> Why is the output of the program(mm-llvm.out) being run is printed only
> once, while the mpirun from intel prints as many times as mentioned in the
> command line?
>
>
> On Thu, Jul 11, 2019 at 11:08 PM Ralph Castain via devel <
> devel@lists.open-mpi.org> wrote:
>
>> Because OMPI binds to core by default when np=2. If you have an OpenMP
>> process, you want to add “--bind-to numa" to your mpirun cmd line.
>>
>>
>> On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel <
>> devel@lists.open-mpi.org> wrote:
>>
>>
>> Hi Devs,
>>
>> I have build openmpi with LLVM-8 compiler, I tried a simple example on a
>> 2 socket machine.
>> The following is the output of mpirun from ICC(intel c compiler package)
>> and mpirun from openmpi:
>>
>> Why the CPU topology(highlighted below) identified by both of them are
>> different? Not sure, if this behavior is correct.
>>
>> $>intel/bin/mpirun -np 2 ./mm-llvm.out
>>
>> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
>> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
>> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf
>> 11 info
>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
>> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
>> OMP: Info #156: KMP_AFFINITY: 56 available OS procs
>> OMP: Info #157: KMP_AFFINITY: Uniform topology
>> *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
>> (28 total cores)*
>> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS
>> proc set
>> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
>> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS
>> proc set
>> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
>> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS
>> proc set
>> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
>> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS
>> proc set
>> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
>> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf
>> 11 info
>> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
>> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
>> OMP: Info #156: KMP_AFFINITY: 56 available OS procs
>> OMP: Info #157: KMP_AFFINITY: Uniform topology
>> *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
>> (28 total cores)*
>> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS
>> proc set
>> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
>> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS
>> proc set
>> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
>> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS
>> proc set
>> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
>> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS
>> proc set
>> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
>> Time is 0.004174
>> 4.00 4.00 4.00 4.00
>> 4.00 4.00 4.00 4.00
>>

Re: [OMPI devel] [OMPI] issue with mpirun

2019-07-12 Thread Ralph Castain via devel 
Afraid I don't know anything about that program, but it looks like it is 
printing the same number of times in both cases. It only appears to be more in 
the first case because the line wraps due to the number of PUs in the list


On Jul 12, 2019, at 3:00 AM, Dangeti Tharun kumar mailto:cs15mtech11...@iith.ac.in> > wrote:

Thanks, Ralph.

Why is the output of the program(mm-llvm.out) being run is printed only once, 
while the mpirun from intel prints as many times as mentioned in the command 
line? 


On Thu, Jul 11, 2019 at 11:08 PM Ralph Castain via devel 
mailto:devel@lists.open-mpi.org> > wrote:
Because OMPI binds to core by default when np=2. If you have an OpenMP process, 
you want to add “--bind-to numa" to your mpirun cmd line.


On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel 
mailto:devel@lists.open-mpi.org> > wrote:


Hi Devs,

I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2 
socket machine.
The following is the output of mpirun from ICC(intel c compiler package) and 
mpirun from openmpi:

Why the CPU topology(highlighted below) identified by both of them are 
different? Not sure, if this behavior is correct.

$>intel/bin/mpirun -np 2 ./mm-llvm.out

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
Time is 0.004174
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
Time is 0.004542
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00

$>openmpi/bin/mpirun -np 2 ./mm-llvm.out 

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56
OMP: Info #156: KMP_AFFINITY: 2 available OS procs

Re: [OMPI devel] [OMPI] issue with mpirun

2019-07-12 Thread Dangeti Tharun kumar via devel 
Thanks, Ralph.

Why is the output of the program(mm-llvm.out) being run is printed only
once, while the mpirun from intel prints as many times as mentioned in the
command line?


On Thu, Jul 11, 2019 at 11:08 PM Ralph Castain via devel <
devel@lists.open-mpi.org> wrote:

> Because OMPI binds to core by default when np=2. If you have an OpenMP
> process, you want to add “--bind-to numa" to your mpirun cmd line.
>
>
> On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel <
> devel@lists.open-mpi.org> wrote:
>
>
> Hi Devs,
>
> I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2
> socket machine.
> The following is the output of mpirun from ICC(intel c compiler package)
> and mpirun from openmpi:
>
> Why the CPU topology(highlighted below) identified by both of them are
> different? Not sure, if this behavior is correct.
>
> $>intel/bin/mpirun -np 2 ./mm-llvm.out
>
> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
> OMP: Info #156: KMP_AFFINITY: 56 available OS procs
> OMP: Info #157: KMP_AFFINITY: Uniform topology
> *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
> (28 total cores)*
> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS
> proc set
> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS
> proc set
> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS
> proc set
> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
> OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS
> proc set
> 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
> OMP: Info #156: KMP_AFFINITY: 56 available OS procs
> OMP: Info #157: KMP_AFFINITY: Uniform topology
> *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
> (28 total cores)*
> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS
> proc set
> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS
> proc set
> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS
> proc set
> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
> OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS
> proc set
> 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
> Time is 0.004174
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
> Time is 0.004542
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
> 4.00 4.00 4.00 4.00
>
> $>openmpi/bin/mpirun -np 2 ./mm-llvm.out
>
> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56
> OMP: Info #156: KMP_AFFINITY: 2 available OS procs
> OMP: Info #157: KMP_AFFINITY: Uniform topology
> OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 4,60
> *OMP: Info

Re: [OMPI devel] [OMPI] issue with mpirun

2019-07-11 Thread Ralph Castain via devel 
Because OMPI binds to core by default when np=2. If you have an OpenMP process, 
you want to add “--bind-to numa" to your mpirun cmd line.


On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel 
mailto:devel@lists.open-mpi.org> > wrote:


Hi Devs,

I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2 
socket machine.
The following is the output of mpirun from ICC(intel c compiler package) and 
mpirun from openmpi:

Why the CPU topology(highlighted below) identified by both of them are 
different? Not sure, if this behavior is correct.

$>intel/bin/mpirun -np 2 ./mm-llvm.out

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc set 
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc set 
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
Time is 0.004174
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
Time is 0.004542
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00 
4.00 4.00 4.00 4.00

$>openmpi/bin/mpirun -np 2 ./mm-llvm.out 

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56
OMP: Info #156: KMP_AFFINITY: 2 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 4,60
OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 
total cores)
OMP: Info #156: KMP_AFFINITY: 2 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 
total cores)
OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89768 thread 0 bound to OS proc set 
4,60
OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89767

[OMPI devel] [OMPI] issue with mpirun

2019-07-11 Thread Dangeti Tharun kumar via devel 
Hi Devs,

I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2
socket machine.
The following is the output of mpirun from ICC(intel c compiler package)
and mpirun from openmpi:

Why the CPU topology(highlighted below) identified by both of them are
different? Not sure, if this behavior is correct.

$>intel/bin/mpirun -np 2 ./mm-llvm.out

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
*OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
(28 total cores)*
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc
set
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc
set
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc
set
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc
set
0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected:
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #156: KMP_AFFINITY: 56 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
*OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core
(28 total cores)*
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc
set
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc
set
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc
set
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc
set
2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111
Time is 0.004174
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
Time is 0.004542
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00
4.00 4.00 4.00 4.00

$>openmpi/bin/mpirun -np 2 ./mm-llvm.out

OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56
OMP: Info #156: KMP_AFFINITY: 2 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 4,60
*OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1
total cores)*
OMP: Info #156: KMP_AFFINITY: 2 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
*OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1
total cores)*
OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89768 thread 0 bound to OS proc
set 4,60
OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89767 thread 0 bound to OS proc
set 0,56
OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89773 thread 1 bound to OS proc
set 0,56
OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89774 thread 1 bound to OS proc
set 4,60
OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89777 thread 3 bound