Re: [OMPI devel] [OMPI] issue with mpirun
Because OMPI binds to core by default when np=2. If you have an OpenMP process, you want to add “--bind-to numa" to your mpirun cmd line. On Jul 11, 2019, at 10:28 AM, Dangeti Tharun kumar via devel mailto:devel@lists.open-mpi.org> > wrote: Hi Devs, I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2 socket machine. The following is the output of mpirun from ICC(intel c compiler package) and mpirun from openmpi: Why the CPU topology(highlighted below) identified by both of them are different? Not sure, if this behavior is correct. $>intel/bin/mpirun -np 2 ./mm-llvm.out OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #156: KMP_AFFINITY: 56 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 total cores) OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #156: KMP_AFFINITY: 56 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 total cores) OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 Time is 0.004174 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 Time is 0.004542 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 $>openmpi/bin/mpirun -np 2 ./mm-llvm.out OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56 OMP: Info #156: KMP_AFFINITY: 2 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 4,60 OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 total cores) OMP: Info #156: KMP_AFFINITY: 2 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 total cores) OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89768 thread 0 bound to OS proc set 4,60 OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89767
[OMPI devel] [OMPI] issue with mpirun
Hi Devs, I have build openmpi with LLVM-8 compiler, I tried a simple example on a 2 socket machine. The following is the output of mpirun from ICC(intel c compiler package) and mpirun from openmpi: Why the CPU topology(highlighted below) identified by both of them are different? Not sure, if this behavior is correct. $>intel/bin/mpirun -np 2 ./mm-llvm.out OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #156: KMP_AFFINITY: 56 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 total cores)* OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89736 thread 0 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89752 thread 1 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89753 thread 2 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #249: KMP_AFFINITY: pid 89736 tid 89754 thread 3 bound to OS proc set 0,1,4,5,8,9,12,13,16,17,20,21,24,25,28,29,32,33,36,37,40,41,44,45,48,49,52,53,56,57,60,61,64,65,68,69,72,73,76,77,80,81,84,85,88,89,92,93,96,97,100,101,104,105,108,109 OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #156: KMP_AFFINITY: 56 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology *OMP: Info #179: KMP_AFFINITY: 2 packages x 14 cores/pkg x 2 threads/core (28 total cores)* OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89737 thread 0 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89755 thread 1 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89756 thread 2 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 OMP: Info #249: KMP_AFFINITY: pid 89737 tid 89757 thread 3 bound to OS proc set 2,3,6,7,10,11,14,15,18,19,22,23,26,27,30,31,34,35,38,39,42,43,46,47,50,51,54,55,58,59,62,63,66,67,70,71,74,75,78,79,82,83,86,87,90,91,94,95,98,99,102,103,106,107,110,111 Time is 0.004174 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 Time is 0.004542 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 4.00 $>openmpi/bin/mpirun -np 2 ./mm-llvm.out OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #211: KMP_AFFINITY: decoding x2APIC ids. OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0,56 OMP: Info #156: KMP_AFFINITY: 2 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology OMP: Info #209: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 4,60 *OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 total cores)* OMP: Info #156: KMP_AFFINITY: 2 available OS procs OMP: Info #157: KMP_AFFINITY: Uniform topology *OMP: Info #179: KMP_AFFINITY: 1 packages x 1 cores/pkg x 2 threads/core (1 total cores)* OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89768 thread 0 bound to OS proc set 4,60 OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89767 thread 0 bound to OS proc set 0,56 OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89773 thread 1 bound to OS proc set 0,56 OMP: Info #249: KMP_AFFINITY: pid 89768 tid 89774 thread 1 bound to OS proc set 4,60 OMP: Info #249: KMP_AFFINITY: pid 89767 tid 89777 thread 3 bound