Got it. Thanks! On Apr 21, 2012, at 8:05 AM, Brice Goglin wrote:
> On 21/04/2012 13:15, Jeffrey Squyres wrote: >> On Apr 21, 2012, at 7:09 AM, Brice Goglin wrote: >> >>> I assume you have the entire distance (latency) matrix between all NUMA >>> nodes as usually reported by the BIOS. >>> >>> const struct hwloc_distance_s *distances = >>> hwloc_get_whole_distance_matrix_by_type(topology, HWLOC_OBJ_NODE); >>> assert(distances); >>> assert(distances->latency); >> Is this stored on the topology object? > > No it's stored in the object that covers all objects connected by the > distance matrix. So usually in the root object. > >> Hence, if this distance data is already covered by the XML export/import, >> then I should have this data. > > Yes it should be there. > >>> Now distances->latency[a+b*distances->nbobjs] contains the latency between >>> NUMA nodes whose *logical* indexes are a and b (it may be asymmetrical). >>> >>> Now get the NUMA node object close to your PUs and the NUMA objects close >>> to each OFED device, take their ->logical_index and you'll get the >>> latencies. >> Ah, ok. This is what I didn't understand from the docs -- is there no >> distance to actual PCI devices? I.e., distance is only measured between >> NUMA nodes? >> >> I ask because the functions allow measuring distance by depth and type -- >> are those effectively ignored, and really all you can check is the distance >> between NUMA nodes? > > You can have distance matrices between any object sets of any type/depth. > Depends what the BIOS reports or what the user adds. The BIOS usually only > reports NUMA node distances. > > We could extend them by saying that the distance between any child of NUMA > node X and any child of NUMA node Y is equal to the distance between NUMA > nodes X and Y, but we don't do that. > > One reason is that the current distance stuff lets the user add a distance > matrix between NUMA nodes and another one between sockets, even if they are > incompatible. When this happens, which one do you use to generate the > distance between two cores? > > There are some tickets in trac they will help clarifying all this mess. > > Brice > > > _______________________________________________ > hwloc-users mailing list > hwloc-us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
