Also, was this with HT's enabled? I'm wondering if the print code is incorrectly computing the core because it isn't correctly accounting for HT cpus.
On Mon, Apr 20, 2015 at 3:49 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > wrote: > Ralph's the authority on this one, but just to be sure: are all nodes the > same topology? E.g., does adding "--hetero-nodes" to the mpirun command > line fix the problem? > > > > On Apr 20, 2015, at 9:29 AM, Elena Elkina <elena.elk...@itseez.com> > wrote: > > > > Hi guys, > > > > I faced with an issue on our cluster related to mapping & binding > policies on 1.8.5. > > > > The matter is that --report-bindings output doesn't correspond to the > locale. It looks like there is a mistake on the output itself, because it > just puts serial core number while that core can be on another socket. For > example, > > > > mpirun -np 2 --display-devel-map --report-bindings --map-by socket > hostname > > Data for JOB [43064,1] offset 0 > > > > Mapper requested: NULL Last mapper: round_robin Mapping policy: > BYSOCKET Ranking policy: SOCKET > > Binding policy: CORE Cpu set: NULL PPR: NULL Cpus-per-rank: 1 > > Num new daemons: 0 New daemon starting vpid INVALID > > Num nodes: 1 > > > > Data for node: clx-orion-001 Launch id: -1 State: 2 > > Daemon: [[43064,0],0] Daemon launched: True > > Num slots: 28 Slots in use: 2 Oversubscribed: FALSE > > Num slots allocated: 28 Max slots: 0 > > Username on node: NULL > > Num procs: 2 Next node_rank: 2 > > Data for proc: [[43064,1],0] > > Pid: 0 Local rank: 0 Node rank: 0 App rank: 0 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 0-6,14-20 Bind location: 0 Binding: 0 > > Data for proc: [[43064,1],1] > > Pid: 0 Local rank: 1 Node rank: 1 App rank: 1 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 7-13,21-27 Bind location: 7 Binding: 7 > > [clx-orion-001:26951] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B/././././././././././././.][./././././././././././././.] > > [clx-orion-001:26951] MCW rank 1 bound to socket 1[core 14[hwt 0]]: > [./././././././././././././.][B/././././././././././././.] > > > > The second process should be bound at core 7 (not core 14). > > > > > > Another example: > > mpirun -np 8 --display-devel-map --report-bindings --map-by core hostname > > Data for JOB [43202,1] offset 0 > > > > Mapper requested: NULL Last mapper: round_robin Mapping policy: > BYCORE Ranking policy: CORE > > Binding policy: CORE Cpu set: NULL PPR: NULL Cpus-per-rank: 1 > > Num new daemons: 0 New daemon starting vpid INVALID > > Num nodes: 1 > > > > Data for node: clx-orion-001 Launch id: -1 State: 2 > > Daemon: [[43202,0],0] Daemon launched: True > > Num slots: 28 Slots in use: 8 Oversubscribed: FALSE > > Num slots allocated: 28 Max slots: 0 > > Username on node: NULL > > Num procs: 8 Next node_rank: 8 > > Data for proc: [[43202,1],0] > > Pid: 0 Local rank: 0 Node rank: 0 App rank: 0 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 0 Bind location: 0 Binding: 0 > > Data for proc: [[43202,1],1] > > Pid: 0 Local rank: 1 Node rank: 1 App rank: 1 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 1 Bind location: 1 Binding: 1 > > Data for proc: [[43202,1],2] > > Pid: 0 Local rank: 2 Node rank: 2 App rank: 2 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 2 Bind location: 2 Binding: 2 > > Data for proc: [[43202,1],3] > > Pid: 0 Local rank: 3 Node rank: 3 App rank: 3 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 3 Bind location: 3 Binding: 3 > > Data for proc: [[43202,1],4] > > Pid: 0 Local rank: 4 Node rank: 4 App rank: 4 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 4 Bind location: 4 Binding: 4 > > Data for proc: [[43202,1],5] > > Pid: 0 Local rank: 5 Node rank: 5 App rank: 5 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 5 Bind location: 5 Binding: 5 > > Data for proc: [[43202,1],6] > > Pid: 0 Local rank: 6 Node rank: 6 App rank: 6 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 6 Bind location: 6 Binding: 6 > > Data for proc: [[43202,1],7] > > Pid: 0 Local rank: 7 Node rank: 7 App rank: 7 > > State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 14 Bind location: 14 Binding: 14 > > [clx-orion-001:27069] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B/././././././././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 1 bound to socket 0[core 1[hwt 0]]: > [./B/./././././././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 2 bound to socket 0[core 2[hwt 0]]: > [././B/././././././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 3 bound to socket 0[core 3[hwt 0]]: > [./././B/./././././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 4 bound to socket 0[core 4[hwt 0]]: > [././././B/././././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 5 bound to socket 0[core 5[hwt 0]]: > [./././././B/./././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 6 bound to socket 0[core 6[hwt 0]]: > [././././././B/././././././.][./././././././././././././.] > > [clx-orion-001:27069] MCW rank 7 bound to socket 0[core 7[hwt 0]]: > [./././././././B/./././././.][./././././././././././././.] > > > > Rank 7 should be bound at core 14 instead of core 7 since core 7 is at > another socket. > > > > Best regards, > > Elena > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17273.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17282.php >
