Guess I should have emphasized the "IF" more - I'm talking about setting the slots ONLY in the case where the user didn't provide that information. If the user provides it, then that is what we use. In OMPI, we *always* accept the user as the ultimate decider.
As things stand, we assume slots=1. This is just as arbitrary as you can get. My point was that now that we use hwloc to discover the number of cpus in the system, we could do something more intelligent than just set it to 1 *in the case where we are given no info*. Note that there is no intention to set binding policy here. All this impacts is how many procs are mapped to a given node when we use the byslot mapping algo. Does that help? On Sep 2, 2012, at 6:20 AM, Kenneth A. Lloyd <kenneth.ll...@wattsys.com> wrote: > I should note that we only virtualize the private cloud / management nodes > over our HPC. The HPC is not virtualized as such. > > Ken > > -----Original Message----- > From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On > Behalf Of Kenneth A. Lloyd > Sent: Sunday, September 02, 2012 7:14 AM > To: 'Open MPI Developers' > Subject: Re: [OMPI devel] RFC: hostfile setting of #slots > > This is a tricky issue, isn't it? With the differences between AMD & Intel, > and between base operating systems "touching" & heaps (betw. Linux & > Windows), and various virtual machines schemes- we have opted for an > "outside the main code path" solution to get deterministic results. But that > is as things are now. Who knows how stuff like AVX2 / memory mapping - or > the next new thing - will affect this? This is similar to issues we've > found with CPU/GPU memory & affinity mapping over IB. The basis of the > decision is (as is often) how much do you trust the user to do the right > thing? What happens if you are wrong? > > Only my opinion. > > Ken > > -----Original Message----- > From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On > Behalf Of Ralph Castain > Sent: Saturday, September 01, 2012 3:54 AM > To: Open MPI Developers > Subject: [OMPI devel] RFC: hostfile setting of #slots > > This is not a notice of intended change, but rather a solicitation for > comment. > > We currently default the number of slots on a host provided via hostfile or > -host to 1. This is a historical "feature" driven by the fact that our > initial launch system spawned daemons on the remote nodes after we had > already mapped the processes to them - so we had no way of guessing a > reasonable value for the number of slots on any node. > > However, the "vm" launch method starts daemons on every node prior to doing > the mapping, precisely so we can use the topology in the mapping algorithm. > This creates the possibility of setting the number of slots on a node to the > number of cpus (either cores or hardware threads, depending on how that flag > is set) IF it wasn't provided in the hostfile. > > This would have an impact on the default "byslot" mapping algorithm. With > only one slot/node, byslot essentially equates to bynode mapping. So a > user-provided hostfile that doesn't give any info on the number of slots on > a node effectively changes the default mapping algorithm to "bynode". This > change would alter that behavior and retain a "byslot" algorithm, with the > number of slots being the number of cpus. > > I have a use-case that would benefit from making the change, but can handle > it outside of the main code path. However, if others would also find it of > use, I can add it to the main code path, either as the default or via MCA > param. > > Any thoughts? > Ralph > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
