Hello Ralph, I've some questions on placement and -cpus-per-rank.
First, use the --cpus-per-rank option to separate the ranks from each other. In other words, instead of --bind-to-socket -bysocket, you do:-bind-to-core -cpus-per-rank NThis will take each rank and bind it to a unique set of N cores, thereby cleanly separating them on the node.
Yes, it helps a lot, but the placement arranged in this way is still not optimal, I believe. The cores are assigned from 0 on in incremental order. On a 2-socket, 12-core machine:
(0,1,2,3,4, 5[,12,13,14,15,16,17]) (6,7,8,9,10,11[,18,19,20,21,22,23]) ^cores^ ^hypercores^ running 2 processes with 5 threads lead to this: 0 <#> linuxbdc07.rz.RWTH-Aachen.DE <#> physcpubind: 0 1 2 3 4 1 <#> linuxbdc07.rz.RWTH-Aachen.DE <#> physcpubind: 5 6 7 8 9 (not used cores: 10, 11; not used hypercores: 12-23)That is, there is an MPI Process bound to core 0 (which is sweet-pot for may kernel things), and the threads of the 2nd process are spawned over both sockets.
- is there a way to say to the system to spawn the processes (= slot chunks defined by -cpus-per-rank N) over the sockets in round-robin model? - is there a way to say "do not use this core number!" in order to add some alignment in core numbering?
- in there a way to use the hypercores in company with the real cores?And last but not least, I found out that the starting and running the program on differing hardware is problematic.
Trying to start 2-rank, 5-thread on 2x6 core computer from my 4-core workstation, I get the below error message.
Seem that the calculation of core numbers / pinning determination is a part of *mpiexec* process instead of being run on the target node? *puzzled*
$ mpiexec -np 1 -H linuxbdc01 -bind-to-core -cpus-per-rank 5 ompi_testpin.sh MPI_FastTest.exe
-------------------------------------------------------------------------- Your job has requested more cpus per process(rank) than there are cpus in a socket: Cpus/rank: 5 #cpus/socket: 4 Please correct one or both of these values and try again. -------------------------------------------------------------------------- $ ssh linuxbdc01 cat /proc/cpuinfo | grep processor | wc -l 24 $ cat /proc/cpuinfo | grep processor | wc -l 4 Best, Paul P.S. Using Open MPI 1.5.3, waiting for 1.5.5 :o)P.S.2. any update on this? http://www.open-mpi.org/community/lists/users/2012/01/18240.php
P.S.3. on the same 16-way, 128 core hardware as in P.S.2, also -cpus-per-rank goes crazy:
$ mpiexec -mca btl_openib_warn_default_gid_prefix=0 -np 2 -H linuxbcsc21 -bind-to-core -cpus-per-rank 5 --report-bindings ompi_testpin.sh MPI_FastTest.exe [linuxbcsc21.rz.RWTH-Aachen.DE:106342] [[55934,0],1] odls:default:fork binding child [[55934,1],0] to cpus 1000100010001 [linuxbcsc21.rz.RWTH-Aachen.DE:106342] [[55934,0],1] odls:default:fork binding child [[55934,1],1] to cpus 20002
0 <#> linuxbcsc21.rz.RWTH-Aachen.DE <#> physcpubind: 0 16 32 48 1 <#> linuxbcsc21.rz.RWTH-Aachen.DE <#> physcpubind: 1 17 So, -cpus-per-rank 5, but one process get 4 cores, the other one - only two..
What you can do is "entice" it away from your processes by leaving 1-2 cores for its own use. For example: -npernode 2 -bind-to-core -cpus-per-rank 3 would run two MPI ranks on each node, each rank exclusively bound to 3 cores.This leaves 2 cores on each node for Linux. When the scheduler sees the 6 cores of your MPI/MP procs working hard, and 2 cores sitting idle, it will tend to use those 2 cores for everything else - and not be tempted to push you aside to gain access to "your" cores.HTH Ralph On Feb 29, 2012, at 3:08 AM, Auclair Francis wrote:Dear Open-MPI users, Our code is currently running Open-MPI (1.5.4) with SLURM on a NUMA machine (2 sockets by nodes and 4 cores by socket) with basically two levels of implementation for Open-MPI: - at lower level n "Master" MPI-processes (one by socket) are simultaneously runned by dividing classically the physical domain into n sub-domains - while at higher level 4n MPI-processes are spawn to run a sparse Poisson solver. At each time step, the code is thus going back and forth between these two levels of implementation using two MPI communicators. This also means that during about half of the computation time, 3n cores are at best sleeping (if not 'waiting' at a barrier) when not inside "Solver routines". We consequently decided to implement OpenMP functionality in our code when solver was not running (we declare one single "parallel" region and use the omp "master" command when OpenMP threads are not active). We however face several difficulties: a) It seems that both the 3n-MPI processes and the OpenMP threads 'consume processor cycles while waiting'. We consequently tried: mpirun -mpi_yield_when_idle 1 , export OMP_WAIT_POLICY=passive or export KMP_BLOCKTIME=0 ... The latest finally leads to an interesting reduction of computing time but worsens the second problem we have to face (see bellow). b) We managed to have a "correct" (?) implementation of our MPI-processeson our sockets by using: mpirun -bind-to-socket -bysocket -np 4n However if OpenMP threads initially seem to scatter on each socket (onethread by core) they slowly migrate to the same core as their 'Master MPI process' or gather on one or two cores by socket We play around with the environment variable KMP_AFFINITY but the best we could obtain was a pinning of the OpenMP threads to their own core... disorganizing at the same time the implementation of the 4n Level-2 MPI processes. When added, neither the specification of a rankfile nor the mpirun option -x IPATH_NO_CPUAFFINITY=1 seem to change significantly the situation. This comportment looks rather inefficient but so far we did not manage to prevent the migration of the 4 threads to at most a couple of cores ! Is there something wrong in our "Hybrid" implementation? Do you have any advices? Thanks for your help, Francis _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
-- Dipl.-Inform. Paul Kapinos - High Performance Computing, RWTH Aachen University, Center for Computing and Communication Seffenter Weg 23, D 52074 Aachen (Germany) Tel: +49 241/80-24915
smime.p7s
Description: S/MIME Cryptographic Signature
