Le 05/07/2011 22:04, Jiri Hladky a écrit : > Well, this is interesting. numactl --hardware shows the number of > hops, regarding to the information from that private BZ.
I think this is wrong. numactl takes everything from sysfs as far as I can tell. On x86, sysfs distances are ACPI SLIT latencies (memory latencies that are normalized to 10 for latencies from one cpu to its local memory). A couple months ago, I checked all Linux ports that show distances in sysfs. All of them report memory latencies, except the SGI IP27 as mentioned previously (this one indeed shows number of hops (0 when local) and it makes a lot of sense for this architecture). That said, the latency that is reported in sysfs is definitely related to the number of hops, and I assume that BIOS vendors look at the number of hops when deciding what to put in the ACPI SLIT distances. It's not something measured at boot. It's kind of hardcoded by/in the BIOS and decided as the same time the routing is computed. But still the ACPI spec says that distances are normalized latencies. One problem I see with the number of hops is that it doesn't make sense on some machines. On some 8-socket AMD machines (such as 8amd64-4n2c.tar.bz2 below) , the hypertransport route between some sockets varies with the type of packet (response or request) and the direction. So the number of hops ends up being asymmetric, depends on read/write, and can be half of an integer. > could me send me the output of hwloc-gather-topology for some NUMA > box? I don't have access to any NUMA running a recent version of kernel. Look at tests/linux/ in the hwloc SVN. The following tarballs contain NUMA architectures. Some of these were gathered while running old kernels, but I don't think it matters because Linux/sysfs reports what the BIOS without changing much of it. 8amd64-4n2c.tar.bz2 (2 hops, but distance is reported as local(10) or remote (20)) 16amd64-8n2c.tar.bz2 (up to 3 hops, but local or remote too) 48amd64-4d2n6c-sparse.tar.bz2 (2 hops IIRC, local(10), remote(16) or "very remote"(22)) 96em64t-4n4d3ca2co.tar.bz2 (1 hop, local(10) or remote(22)) 16ia64-8n2s.tar.bz2 (2 hops, 10/25/29) 256ia64-64n2s2c.tar.bz2 (big SGI Altix machine with several hops, 10/22/26/30/34) Brice
