Yes, I missed that point before - too early in the morning :-/ As I said in my last note, it would be nice to either have a flag indicating we are bound, or see all the cpu info so we can compute that we are bound. Either way, we still need to have a complete picture of all I/O devices so you can compute the distance.
On Feb 9, 2012, at 6:01 AM, nadia.der...@bull.net wrote: > > > devel-boun...@open-mpi.org wrote on 02/09/2012 01:32:31 PM: > > > De : Ralph Castain <r...@open-mpi.org> > > A : Open MPI Developers <de...@open-mpi.org> > > Date : 02/09/2012 01:32 PM > > Objet : Re: [OMPI devel] btl/openib: get_ib_dev_distance doesn't see > > processes as bound if the job has been launched by srun > > Envoyé par : devel-boun...@open-mpi.org > > > > Hi Nadia > > > > I'm wondering what value there is in showing the full topology, or > > using it in any of our components, if the process is restricted to a > > specific set of cpus? Does it really help to know that there are > > other cpus out there that are unreachable? > > Ralph, > > The intention here is not to show cpus that are unreachable, but to fix an > issue we have at least in get_ib_dev_distance() in the openib btl. > > The problem is that if a process is restricted to a single CPU, the algorithm > used in get_ib_dev_distance doesn't work at all: > I have 2 ib interfaces on my victim (say mlx4_0 and mlx4_1), and I want the > openib btl to select the one that is the closest to my rank. > > As I said in my first e-mail, here is what is done today: > . opal_paffinity_base_get_processor_info() is called to get the number of > logical processors (we get 1 due to the singleton cpuset) > . we loop over that # of processors to check whether our process is bound > to one of them. In our case the loop will be executed only once and we will > never get the correct binding information. > . if the process is bound actually get the distance to the device. > in our case, the distance won't be computed and mlx4_0 will be seen as > "equivalent" to mlx4_1 in terms of distances. This is what I definitely want > to avoid. > > Regards, > Nadia > > > > > On Feb 9, 2012, at 5:15 AM, nadia.der...@bull.net wrote: > > > > > > > > devel-boun...@open-mpi.org wrote on 02/09/2012 12:20:41 PM: > > > > > De : Brice Goglin <brice.gog...@inria.fr> > > > A : Open MPI Developers <de...@open-mpi.org> > > > Date : 02/09/2012 12:20 PM > > > Objet : Re: [OMPI devel] btl/openib: get_ib_dev_distance doesn't see > > > processes as bound if the job has been launched by srun > > > Envoyé par : devel-boun...@open-mpi.org > > > > > > By default, hwloc only shows what's inside the current cpuset. There's > > > an option to show everything instead (topology flag). > > > > So may be using that flag inside > > opal_paffinity_base_get_processor_info() would be a better fix than > > the one I'm proposing in my patch. > > > > I found a bunch of other places where things are managed as in > > get_ib_dev_distance(). > > > > Just doing a grep in the sources, I could find: > > . init_maffinity() in btl/sm/btl_sm.c > > . vader_init_maffinity() in btl/vader/btl_vader.c > > . get_ib_dev_distance() in btl/wv/btl_wv_component.c > > > > So I think the flag Brice is talking about should definitely be the fix. > > > > Regards, > > Nadia > > > > > > > > Brice > > > > > > > > > > > > Le 09/02/2012 12:18, Jeff Squyres a écrit : > > > > Just so that I understand this better -- if a process is bound in > > > a cpuset, will tools like hwloc's lstopo only show the Linux > > > processors *in that cpuset*? I.e., does it not have any visibility > > > of the processors outside of its cpuset? > > > > > > > > > > > > On Jan 27, 2012, at 11:38 AM, nadia.derbey wrote: > > > > > > > >> Hi, > > > >> > > > >> If a job is launched using "srun --resv-ports --cpu_bind:..." and slurm > > > >> is configured with: > > > >> TaskPlugin=task/affinity > > > >> TaskPluginParam=Cpusets > > > >> > > > >> each rank of that job is in a cpuset that contains a single CPU. > > > >> > > > >> Now, if we use carto on top of this, the following happens in > > > >> get_ib_dev_distance() (in btl/openib/btl_openib_component.c): > > > >> . opal_paffinity_base_get_processor_info() is called to get the > > > >> number of logical processors (we get 1 due to the singleton cpuset) > > > >> . we loop over that # of processors to check whether our process is > > > >> bound to one of them. In our case the loop will be executed only > > > >> once and we will never get the correct binding information. > > > >> . if the process is bound actually get the distance to the device. > > > >> in our case we won't execute that part of the code. > > > >> > > > >> The attached patch is a proposal to fix the issue. > > > >> > > > >> Regards, > > > >> Nadia > > > >> > > <get_ib_dev_distance.patch>_______________________________________________ > > > >> devel mailing list > > > >> de...@open-mpi.org > > > >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > > > > > > > _______________________________________________ > > > devel mailing list > > > de...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel_______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
