On Feb 14, 2011, at 9:35 AM, Siew Yin Chan wrote: > 1. I tried Open MPI 1.5.1 before turning to hwloc-bind. Yep. Open MPI 1.5.1 > does provide the --bycore and --bind-to-core option, but this option seems to > bind processes to cores on my machine according to the *physical* indexes:
FWIW, you might want to try one of the OMPI 1.5.2 nightly tarballs -- we switched the process affinity stuff to hwloc in 1.5.2 (the 1.5.1 stuff uses a different mechanism). > FYI, my testing environment and application imposes these requirements for > optimum performance: > > i. Different binaries optimized for heterogeneous machines. This necessitates > MIMD, and can be done in OMPI using the -app option (providing an > application context file). > ii. The application is communication-sensitive. Thus, fine-grained process > mapping on *machines* and on *cores* is required to minimize inter-machine > and inter-socket communication costs occurring on the network and on the > system bus. Specifically, processes should be mapped onto successive cores of > one socket before the next socket is considered, i.e., socket.0:core0-3, then > socket.1:core0-3. In this case, the communication among neighboring rank 0-3 > will be confined to socket 0 without going through the system bus. Same for > rank 4-7 on socket 1. As such, the order of the cores should follow the > *logical* indexes. I think that OMPI 1.5.2 should do this for you -- rather than following and logical/physical ordering, it does what you describe: traverses successive cores on a socket before going to the next socket (which happens to correspond to hwloc's logical ordering, but that was not the intent). FWIW, we have a huge revamp of OMPI's affinity support on the mpirun command line that will offer much more flexible binding choices. > Initially, I tried combining the features of rankfile and appfile, e.g., > > $ cat rankfile8np4 > rank 0=compute-0-8 slot=0:0 > rank 1=compute-0-8 slot=0:1 > rank 2=compute-0-8 slot=0:2 > rank 3=compute-0-8 slot=0:3 > $ cat rankfile9np4 > rank 0=compute-0-9 slot=0:0 > rank 1=compute-0-9 slot=0:1 > rank 2=compute-0-9 slot=0:2 > rank 3=compute-0-9 slot=0:3 > $ cat my_appfile_rankfile > --host compute-0-8 -rf rankfile8np4 -np 4 ./test1 > --host compute-0-9 -rf rankfile9np4 -np 4 ./test2 > $ mpirun -app my_appfile_rankfile > > but found out that only the rankfile stated on the first line took effect; > the second was ignored completely. After some time of googling and trial and > error, I decided to try an external binder, and this direction led me to > hwloc-bind. > > Maybe I should bring the issue of rankfile + appfile to the OMPI mailing list. Yes. I'd have to look at it more closely, but it's possible that we only allow one rankfile per job -- i.e., that the rankfile should specify all the procs in the job, not on a per-host basis. But perhaps we don't warn/error if multiple rankfiles are used; I would consider that a bug. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
