Hello,
After much hacking I have some code, which I present here for
comment/query/criticism, with some ramblings and queries of my own.
Code can be found here:
http://xenbits.xen.org/gitweb/?p=people/andrewcoop/hwloc.git;a=commitdiff;h=6c37406dae887386205124ab9151b9be5812b56a
For anyone wishing to try it out for themselves, there is an extra patch
required for libxc, available here:
http://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=commitdiff;h=3585994405b6a73c137309dd4be91f48c71e4903
(Basically, the existing xc_{topology,numa}info() library calls were
practically useless, and required the library user to perform the
hypercall bounce buffering themselves, without access to half the bounce
infrastructure. I have rewritten them in such a way that libxc does all
the appropriate bounce buffering.)
First of all, I have hacked at the m4, by copying surrounding code, and
it now appears to work sanely for me, including --{en,dis}able-xen
configure options. I have no idea whether what I have done is appropriate.
Xen support itself is only usable if explicitly requested, via the
presence of the HWLOC_XEN environment variable. The Xen backend has a
higher priority than native, and excludes all other CPU topology
gathering modules, as the native OS will see the fake topology, and the
x86 module has no idea which pcpus its vcpus are currently executing on,
so can't map cpuid results back to reality.
As for developing the backend, the documentation was a little lacking.
It would have vastly helped had there been a sentence describing how one
is expected to build the topology. What I realised, after far too long
staring at the spaghetti code in other backends, was that
hwloc_insert_object_by_cpuset() appears to be my friend, and
subsequently made the process very simple.
At the moment, topology-xen appears capable of constructing the PUs,
cores, sockets and numa nodes. I have a stack of queries and bugfixes
against Xen which I shall bring up on xen-devel in due course. Chief
among them is that there the way Xen currently signals an offline PU's
is to trash its location information in the system topology. This means
that I can identify a specific PU as being offline, but can only infer
its position in the topology as I happen to know Xen writes the records
sequentially. Another is that Xen enumerates the cores per-socket,
rather than on an absolute numbering scheme. There is a gross hack to
"fix" this in hwloc_get_xen_info() but is only valid for homogeneous
systems.
Somewhat expectedly, Xen has interface for gathering the cache
information. dom0 using the cpuid instruction can gather cache
information that way, but cant map a particular cpuid call to a specific
PU. It would certainly be useful to get, so I have half a mind to add a
new Xen sysctl hypercall of "please execute cpuid with these parameters
on that specific cpu and hand me the results". That would at least
provide an ability to get the information.
Completely expectedly (as this is still an open problem upstream), Xen
has no ability to work out the NUMA affinity of IO devices.
Dom0 in Xen is a bit of a strange system. It has almost
completely-unrestricted access to all the PCI devices in the system, and
access to the real ACPI tables and special BIOS areas in low RAM. Xen
controls all interrupts, the virtualisation hardware and the serial
UART; dom0 has the rest. Dom0 has an explicitly faked up cpu topology
via cpuid, and a non-contiguous address space for its RAM, yet real
cache information via cpuid, and a mostly non-faked feature set.
Most importantly, Xen does not have an AML VM; it can read the static
ACPI tables, but relies on dom0 to provide any information required by
executing AML, which includes any _PXM attributes for IO devices. Xen
itself has little/no use for the _PXM information as it has no device
drivers, yet dom0 doesn't have an accurate view of the CPU/RAM topology
with which to associate the _PXM information. (It is all quite a mess)
Attached are two (non-io) xml outputs from the most interesting AMD and
Intel servers I could easily put my hands on from our testing pool. As
far as I am aware, the drawn diagrams matches my understanding of how
the server topology is organised.
You might notice that the xml is a little thin on details. One problem
I have is how to integrate things like the DMI information? I know for
certain that the linux component will get the correct DMI information
(as dom0 gets the real SMBios tables), but it is not in an easy form to
consume from outside the linux module. Then comes the question of how
to use the *BSD DMI information on BSD systems which can use the xen
component as-is? One idea would be to have a "native dmi information"
function pointer which can be optionally implemented, but that would be
somewhat of an architecture overhaul. Also I suspect it would require
access to the native components private data which doesn't appears to
exist for the duration of Xen's topology gathering.
One thing I noticed was about allowed_{cpu,node}set. From what I can
gather, this is to do with process binding, which is meaningless in the
context of the Xen system topology. What is the approved way of setting
these all to 0?
A problem I encountered was the difference between cpuset, online_cpuset
and complete_cpuset. I can see that online_cpuset is likely to be
subset of the others, but I cant find a definitive difference between
the cpuset and complete_cpuset. Can anyone enlighten me?
What is the canonical way of stating that a certain PU is offline? Xen
doesn't really do offline cpus of its own accord at the moment but does
have a hypercalls to explicitly online and offline cpus. In the case
that one is offline, I suspect my current code will cause the PU to fall
out of the rest of the topology, rather than stay within it and marked
differently.
Anyway - I think I should stop rambling. I would appreciate code
review/comments on the code itself, particularly with whether I am using
the API correctly (its all backwards WRT how the docs are written).
Thanks,
~Andrew
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
<object type="Machine" os_index="0" cpuset="0x00ffffff,0xffffffff" complete_cpuset="0x00ffffff,0xffffffff" online_cpuset="0x00ffffff,0xffffffff" allowed_cpuset="0x00ffffff,0xffffffff">
<info name="Backend" value="Xen"/>
<object type="NUMANode" os_index="0" cpuset="0x0fffffff" complete_cpuset="0x0fffffff" online_cpuset="0x0fffffff" allowed_cpuset="0x0fffffff">
<object type="Socket" os_index="0" cpuset="0x0fffffff" complete_cpuset="0x0fffffff" online_cpuset="0x0fffffff" allowed_cpuset="0x0fffffff">
<object type="Core" os_index="0" cpuset="0x00000003" complete_cpuset="0x00000003" online_cpuset="0x00000003" allowed_cpuset="0x00000003">
<object type="PU" os_index="0" cpuset="0x00000001" complete_cpuset="0x00000001" online_cpuset="0x00000001" allowed_cpuset="0x00000001"/>
<object type="PU" os_index="1" cpuset="0x00000002" complete_cpuset="0x00000002" online_cpuset="0x00000002" allowed_cpuset="0x00000002"/>
</object>
<object type="Core" os_index="1" cpuset="0x0000000c" complete_cpuset="0x0000000c" online_cpuset="0x0000000c" allowed_cpuset="0x0000000c">
<object type="PU" os_index="2" cpuset="0x00000004" complete_cpuset="0x00000004" online_cpuset="0x00000004" allowed_cpuset="0x00000004"/>
<object type="PU" os_index="3" cpuset="0x00000008" complete_cpuset="0x00000008" online_cpuset="0x00000008" allowed_cpuset="0x00000008"/>
</object>
<object type="Core" os_index="2" cpuset="0x00000030" complete_cpuset="0x00000030" online_cpuset="0x00000030" allowed_cpuset="0x00000030">
<object type="PU" os_index="4" cpuset="0x00000010" complete_cpuset="0x00000010" online_cpuset="0x00000010" allowed_cpuset="0x00000010"/>
<object type="PU" os_index="5" cpuset="0x00000020" complete_cpuset="0x00000020" online_cpuset="0x00000020" allowed_cpuset="0x00000020"/>
</object>
<object type="Core" os_index="3" cpuset="0x000000c0" complete_cpuset="0x000000c0" online_cpuset="0x000000c0" allowed_cpuset="0x000000c0">
<object type="PU" os_index="6" cpuset="0x00000040" complete_cpuset="0x00000040" online_cpuset="0x00000040" allowed_cpuset="0x00000040"/>
<object type="PU" os_index="7" cpuset="0x00000080" complete_cpuset="0x00000080" online_cpuset="0x00000080" allowed_cpuset="0x00000080"/>
</object>
<object type="Core" os_index="4" cpuset="0x00000300" complete_cpuset="0x00000300" online_cpuset="0x00000300" allowed_cpuset="0x00000300">
<object type="PU" os_index="8" cpuset="0x00000100" complete_cpuset="0x00000100" online_cpuset="0x00000100" allowed_cpuset="0x00000100"/>
<object type="PU" os_index="9" cpuset="0x00000200" complete_cpuset="0x00000200" online_cpuset="0x00000200" allowed_cpuset="0x00000200"/>
</object>
<object type="Core" os_index="5" cpuset="0x00000c00" complete_cpuset="0x00000c00" online_cpuset="0x00000c00" allowed_cpuset="0x00000c00">
<object type="PU" os_index="10" cpuset="0x00000400" complete_cpuset="0x00000400" online_cpuset="0x00000400" allowed_cpuset="0x00000400"/>
<object type="PU" os_index="11" cpuset="0x00000800" complete_cpuset="0x00000800" online_cpuset="0x00000800" allowed_cpuset="0x00000800"/>
</object>
<object type="Core" os_index="6" cpuset="0x00003000" complete_cpuset="0x00003000" online_cpuset="0x00003000" allowed_cpuset="0x00003000">
<object type="PU" os_index="12" cpuset="0x00001000" complete_cpuset="0x00001000" online_cpuset="0x00001000" allowed_cpuset="0x00001000"/>
<object type="PU" os_index="13" cpuset="0x00002000" complete_cpuset="0x00002000" online_cpuset="0x00002000" allowed_cpuset="0x00002000"/>
</object>
<object type="Core" os_index="8" cpuset="0x0000c000" complete_cpuset="0x0000c000" online_cpuset="0x0000c000" allowed_cpuset="0x0000c000">
<object type="PU" os_index="14" cpuset="0x00004000" complete_cpuset="0x00004000" online_cpuset="0x00004000" allowed_cpuset="0x00004000"/>
<object type="PU" os_index="15" cpuset="0x00008000" complete_cpuset="0x00008000" online_cpuset="0x00008000" allowed_cpuset="0x00008000"/>
</object>
<object type="Core" os_index="9" cpuset="0x00030000" complete_cpuset="0x00030000" online_cpuset="0x00030000" allowed_cpuset="0x00030000">
<object type="PU" os_index="16" cpuset="0x00010000" complete_cpuset="0x00010000" online_cpuset="0x00010000" allowed_cpuset="0x00010000"/>
<object type="PU" os_index="17" cpuset="0x00020000" complete_cpuset="0x00020000" online_cpuset="0x00020000" allowed_cpuset="0x00020000"/>
</object>
<object type="Core" os_index="10" cpuset="0x000c0000" complete_cpuset="0x000c0000" online_cpuset="0x000c0000" allowed_cpuset="0x000c0000">
<object type="PU" os_index="18" cpuset="0x00040000" complete_cpuset="0x00040000" online_cpuset="0x00040000" allowed_cpuset="0x00040000"/>
<object type="PU" os_index="19" cpuset="0x00080000" complete_cpuset="0x00080000" online_cpuset="0x00080000" allowed_cpuset="0x00080000"/>
</object>
<object type="Core" os_index="11" cpuset="0x00300000" complete_cpuset="0x00300000" online_cpuset="0x00300000" allowed_cpuset="0x00300000">
<object type="PU" os_index="20" cpuset="0x00100000" complete_cpuset="0x00100000" online_cpuset="0x00100000" allowed_cpuset="0x00100000"/>
<object type="PU" os_index="21" cpuset="0x00200000" complete_cpuset="0x00200000" online_cpuset="0x00200000" allowed_cpuset="0x00200000"/>
</object>
<object type="Core" os_index="12" cpuset="0x00c00000" complete_cpuset="0x00c00000" online_cpuset="0x00c00000" allowed_cpuset="0x00c00000">
<object type="PU" os_index="22" cpuset="0x00400000" complete_cpuset="0x00400000" online_cpuset="0x00400000" allowed_cpuset="0x00400000"/>
<object type="PU" os_index="23" cpuset="0x00800000" complete_cpuset="0x00800000" online_cpuset="0x00800000" allowed_cpuset="0x00800000"/>
</object>
<object type="Core" os_index="13" cpuset="0x03000000" complete_cpuset="0x03000000" online_cpuset="0x03000000" allowed_cpuset="0x03000000">
<object type="PU" os_index="24" cpuset="0x01000000" complete_cpuset="0x01000000" online_cpuset="0x01000000" allowed_cpuset="0x01000000"/>
<object type="PU" os_index="25" cpuset="0x02000000" complete_cpuset="0x02000000" online_cpuset="0x02000000" allowed_cpuset="0x02000000"/>
</object>
<object type="Core" os_index="14" cpuset="0x0c000000" complete_cpuset="0x0c000000" online_cpuset="0x0c000000" allowed_cpuset="0x0c000000">
<object type="PU" os_index="26" cpuset="0x04000000" complete_cpuset="0x04000000" online_cpuset="0x04000000" allowed_cpuset="0x04000000"/>
<object type="PU" os_index="27" cpuset="0x08000000" complete_cpuset="0x08000000" online_cpuset="0x08000000" allowed_cpuset="0x08000000"/>
</object>
</object>
</object>
<object type="NUMANode" os_index="1" cpuset="0x00ffffff,0xf0000000" complete_cpuset="0x00ffffff,0xf0000000" online_cpuset="0x00ffffff,0xf0000000" allowed_cpuset="0x00ffffff,0xf0000000">
<object type="Socket" os_index="1" cpuset="0x00ffffff,0xf0000000" complete_cpuset="0x00ffffff,0xf0000000" online_cpuset="0x00ffffff,0xf0000000" allowed_cpuset="0x00ffffff,0xf0000000">
<object type="Core" os_index="28" cpuset="0x30000000" complete_cpuset="0x30000000" online_cpuset="0x30000000" allowed_cpuset="0x30000000">
<object type="PU" os_index="28" cpuset="0x10000000" complete_cpuset="0x10000000" online_cpuset="0x10000000" allowed_cpuset="0x10000000"/>
<object type="PU" os_index="29" cpuset="0x20000000" complete_cpuset="0x20000000" online_cpuset="0x20000000" allowed_cpuset="0x20000000"/>
</object>
<object type="Core" os_index="29" cpuset="0xc0000000" complete_cpuset="0xc0000000" online_cpuset="0xc0000000" allowed_cpuset="0xc0000000">
<object type="PU" os_index="30" cpuset="0x40000000" complete_cpuset="0x40000000" online_cpuset="0x40000000" allowed_cpuset="0x40000000"/>
<object type="PU" os_index="31" cpuset="0x80000000" complete_cpuset="0x80000000" online_cpuset="0x80000000" allowed_cpuset="0x80000000"/>
</object>
<object type="Core" os_index="30" cpuset="0x00000003,0x0" complete_cpuset="0x00000003,0x0" online_cpuset="0x00000003,0x0" allowed_cpuset="0x00000003,0x0">
<object type="PU" os_index="32" cpuset="0x00000001,0x0" complete_cpuset="0x00000001,0x0" online_cpuset="0x00000001,0x0" allowed_cpuset="0x00000001,0x0"/>
<object type="PU" os_index="33" cpuset="0x00000002,0x0" complete_cpuset="0x00000002,0x0" online_cpuset="0x00000002,0x0" allowed_cpuset="0x00000002,0x0"/>
</object>
<object type="Core" os_index="31" cpuset="0x0000000c,0x0" complete_cpuset="0x0000000c,0x0" online_cpuset="0x0000000c,0x0" allowed_cpuset="0x0000000c,0x0">
<object type="PU" os_index="34" cpuset="0x00000004,0x0" complete_cpuset="0x00000004,0x0" online_cpuset="0x00000004,0x0" allowed_cpuset="0x00000004,0x0"/>
<object type="PU" os_index="35" cpuset="0x00000008,0x0" complete_cpuset="0x00000008,0x0" online_cpuset="0x00000008,0x0" allowed_cpuset="0x00000008,0x0"/>
</object>
<object type="Core" os_index="32" cpuset="0x00000030,0x0" complete_cpuset="0x00000030,0x0" online_cpuset="0x00000030,0x0" allowed_cpuset="0x00000030,0x0">
<object type="PU" os_index="36" cpuset="0x00000010,0x0" complete_cpuset="0x00000010,0x0" online_cpuset="0x00000010,0x0" allowed_cpuset="0x00000010,0x0"/>
<object type="PU" os_index="37" cpuset="0x00000020,0x0" complete_cpuset="0x00000020,0x0" online_cpuset="0x00000020,0x0" allowed_cpuset="0x00000020,0x0"/>
</object>
<object type="Core" os_index="33" cpuset="0x000000c0,0x0" complete_cpuset="0x000000c0,0x0" online_cpuset="0x000000c0,0x0" allowed_cpuset="0x000000c0,0x0">
<object type="PU" os_index="38" cpuset="0x00000040,0x0" complete_cpuset="0x00000040,0x0" online_cpuset="0x00000040,0x0" allowed_cpuset="0x00000040,0x0"/>
<object type="PU" os_index="39" cpuset="0x00000080,0x0" complete_cpuset="0x00000080,0x0" online_cpuset="0x00000080,0x0" allowed_cpuset="0x00000080,0x0"/>
</object>
<object type="Core" os_index="34" cpuset="0x00000300,0x0" complete_cpuset="0x00000300,0x0" online_cpuset="0x00000300,0x0" allowed_cpuset="0x00000300,0x0">
<object type="PU" os_index="40" cpuset="0x00000100,0x0" complete_cpuset="0x00000100,0x0" online_cpuset="0x00000100,0x0" allowed_cpuset="0x00000100,0x0"/>
<object type="PU" os_index="41" cpuset="0x00000200,0x0" complete_cpuset="0x00000200,0x0" online_cpuset="0x00000200,0x0" allowed_cpuset="0x00000200,0x0"/>
</object>
<object type="Core" os_index="36" cpuset="0x00000c00,0x0" complete_cpuset="0x00000c00,0x0" online_cpuset="0x00000c00,0x0" allowed_cpuset="0x00000c00,0x0">
<object type="PU" os_index="42" cpuset="0x00000400,0x0" complete_cpuset="0x00000400,0x0" online_cpuset="0x00000400,0x0" allowed_cpuset="0x00000400,0x0"/>
<object type="PU" os_index="43" cpuset="0x00000800,0x0" complete_cpuset="0x00000800,0x0" online_cpuset="0x00000800,0x0" allowed_cpuset="0x00000800,0x0"/>
</object>
<object type="Core" os_index="37" cpuset="0x00003000,0x0" complete_cpuset="0x00003000,0x0" online_cpuset="0x00003000,0x0" allowed_cpuset="0x00003000,0x0">
<object type="PU" os_index="44" cpuset="0x00001000,0x0" complete_cpuset="0x00001000,0x0" online_cpuset="0x00001000,0x0" allowed_cpuset="0x00001000,0x0"/>
<object type="PU" os_index="45" cpuset="0x00002000,0x0" complete_cpuset="0x00002000,0x0" online_cpuset="0x00002000,0x0" allowed_cpuset="0x00002000,0x0"/>
</object>
<object type="Core" os_index="38" cpuset="0x0000c000,0x0" complete_cpuset="0x0000c000,0x0" online_cpuset="0x0000c000,0x0" allowed_cpuset="0x0000c000,0x0">
<object type="PU" os_index="46" cpuset="0x00004000,0x0" complete_cpuset="0x00004000,0x0" online_cpuset="0x00004000,0x0" allowed_cpuset="0x00004000,0x0"/>
<object type="PU" os_index="47" cpuset="0x00008000,0x0" complete_cpuset="0x00008000,0x0" online_cpuset="0x00008000,0x0" allowed_cpuset="0x00008000,0x0"/>
</object>
<object type="Core" os_index="39" cpuset="0x00030000,0x0" complete_cpuset="0x00030000,0x0" online_cpuset="0x00030000,0x0" allowed_cpuset="0x00030000,0x0">
<object type="PU" os_index="48" cpuset="0x00010000,0x0" complete_cpuset="0x00010000,0x0" online_cpuset="0x00010000,0x0" allowed_cpuset="0x00010000,0x0"/>
<object type="PU" os_index="49" cpuset="0x00020000,0x0" complete_cpuset="0x00020000,0x0" online_cpuset="0x00020000,0x0" allowed_cpuset="0x00020000,0x0"/>
</object>
<object type="Core" os_index="40" cpuset="0x000c0000,0x0" complete_cpuset="0x000c0000,0x0" online_cpuset="0x000c0000,0x0" allowed_cpuset="0x000c0000,0x0">
<object type="PU" os_index="50" cpuset="0x00040000,0x0" complete_cpuset="0x00040000,0x0" online_cpuset="0x00040000,0x0" allowed_cpuset="0x00040000,0x0"/>
<object type="PU" os_index="51" cpuset="0x00080000,0x0" complete_cpuset="0x00080000,0x0" online_cpuset="0x00080000,0x0" allowed_cpuset="0x00080000,0x0"/>
</object>
<object type="Core" os_index="41" cpuset="0x00300000,0x0" complete_cpuset="0x00300000,0x0" online_cpuset="0x00300000,0x0" allowed_cpuset="0x00300000,0x0">
<object type="PU" os_index="52" cpuset="0x00100000,0x0" complete_cpuset="0x00100000,0x0" online_cpuset="0x00100000,0x0" allowed_cpuset="0x00100000,0x0"/>
<object type="PU" os_index="53" cpuset="0x00200000,0x0" complete_cpuset="0x00200000,0x0" online_cpuset="0x00200000,0x0" allowed_cpuset="0x00200000,0x0"/>
</object>
<object type="Core" os_index="42" cpuset="0x00c00000,0x0" complete_cpuset="0x00c00000,0x0" online_cpuset="0x00c00000,0x0" allowed_cpuset="0x00c00000,0x0">
<object type="PU" os_index="54" cpuset="0x00400000,0x0" complete_cpuset="0x00400000,0x0" online_cpuset="0x00400000,0x0" allowed_cpuset="0x00400000,0x0"/>
<object type="PU" os_index="55" cpuset="0x00800000,0x0" complete_cpuset="0x00800000,0x0" online_cpuset="0x00800000,0x0" allowed_cpuset="0x00800000,0x0"/>
</object>
</object>
</object>
</object>
</topology>
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE topology SYSTEM "hwloc.dtd">
<topology>
<object type="Machine" os_index="0" cpuset="0xffffffff,0xffffffff" complete_cpuset="0xffffffff,0xffffffff" online_cpuset="0xffffffff,0xffffffff" allowed_cpuset="0xffffffff,0xffffffff">
<info name="Backend" value="Xen"/>
<object type="Socket" os_index="0" cpuset="0x0000ffff" complete_cpuset="0x0000ffff" online_cpuset="0x0000ffff" allowed_cpuset="0x0000ffff">
<object type="NUMANode" os_index="0" cpuset="0x000000ff" complete_cpuset="0x000000ff" online_cpuset="0x000000ff" allowed_cpuset="0x000000ff">
<object type="Core" os_index="0" cpuset="0x00000001" complete_cpuset="0x00000001" online_cpuset="0x00000001" allowed_cpuset="0x00000001">
<object type="PU" os_index="0" cpuset="0x00000001" complete_cpuset="0x00000001" online_cpuset="0x00000001" allowed_cpuset="0x00000001"/>
</object>
<object type="Core" os_index="1" cpuset="0x00000002" complete_cpuset="0x00000002" online_cpuset="0x00000002" allowed_cpuset="0x00000002">
<object type="PU" os_index="1" cpuset="0x00000002" complete_cpuset="0x00000002" online_cpuset="0x00000002" allowed_cpuset="0x00000002"/>
</object>
<object type="Core" os_index="2" cpuset="0x00000004" complete_cpuset="0x00000004" online_cpuset="0x00000004" allowed_cpuset="0x00000004">
<object type="PU" os_index="2" cpuset="0x00000004" complete_cpuset="0x00000004" online_cpuset="0x00000004" allowed_cpuset="0x00000004"/>
</object>
<object type="Core" os_index="3" cpuset="0x00000008" complete_cpuset="0x00000008" online_cpuset="0x00000008" allowed_cpuset="0x00000008">
<object type="PU" os_index="3" cpuset="0x00000008" complete_cpuset="0x00000008" online_cpuset="0x00000008" allowed_cpuset="0x00000008"/>
</object>
<object type="Core" os_index="4" cpuset="0x00000010" complete_cpuset="0x00000010" online_cpuset="0x00000010" allowed_cpuset="0x00000010">
<object type="PU" os_index="4" cpuset="0x00000010" complete_cpuset="0x00000010" online_cpuset="0x00000010" allowed_cpuset="0x00000010"/>
</object>
<object type="Core" os_index="5" cpuset="0x00000020" complete_cpuset="0x00000020" online_cpuset="0x00000020" allowed_cpuset="0x00000020">
<object type="PU" os_index="5" cpuset="0x00000020" complete_cpuset="0x00000020" online_cpuset="0x00000020" allowed_cpuset="0x00000020"/>
</object>
<object type="Core" os_index="6" cpuset="0x00000040" complete_cpuset="0x00000040" online_cpuset="0x00000040" allowed_cpuset="0x00000040">
<object type="PU" os_index="6" cpuset="0x00000040" complete_cpuset="0x00000040" online_cpuset="0x00000040" allowed_cpuset="0x00000040"/>
</object>
<object type="Core" os_index="7" cpuset="0x00000080" complete_cpuset="0x00000080" online_cpuset="0x00000080" allowed_cpuset="0x00000080">
<object type="PU" os_index="7" cpuset="0x00000080" complete_cpuset="0x00000080" online_cpuset="0x00000080" allowed_cpuset="0x00000080"/>
</object>
</object>
<object type="NUMANode" os_index="1" cpuset="0x0000ff00" complete_cpuset="0x0000ff00" online_cpuset="0x0000ff00" allowed_cpuset="0x0000ff00">
<object type="Core" os_index="8" cpuset="0x00000100" complete_cpuset="0x00000100" online_cpuset="0x00000100" allowed_cpuset="0x00000100">
<object type="PU" os_index="8" cpuset="0x00000100" complete_cpuset="0x00000100" online_cpuset="0x00000100" allowed_cpuset="0x00000100"/>
</object>
<object type="Core" os_index="9" cpuset="0x00000200" complete_cpuset="0x00000200" online_cpuset="0x00000200" allowed_cpuset="0x00000200">
<object type="PU" os_index="9" cpuset="0x00000200" complete_cpuset="0x00000200" online_cpuset="0x00000200" allowed_cpuset="0x00000200"/>
</object>
<object type="Core" os_index="10" cpuset="0x00000400" complete_cpuset="0x00000400" online_cpuset="0x00000400" allowed_cpuset="0x00000400">
<object type="PU" os_index="10" cpuset="0x00000400" complete_cpuset="0x00000400" online_cpuset="0x00000400" allowed_cpuset="0x00000400"/>
</object>
<object type="Core" os_index="11" cpuset="0x00000800" complete_cpuset="0x00000800" online_cpuset="0x00000800" allowed_cpuset="0x00000800">
<object type="PU" os_index="11" cpuset="0x00000800" complete_cpuset="0x00000800" online_cpuset="0x00000800" allowed_cpuset="0x00000800"/>
</object>
<object type="Core" os_index="12" cpuset="0x00001000" complete_cpuset="0x00001000" online_cpuset="0x00001000" allowed_cpuset="0x00001000">
<object type="PU" os_index="12" cpuset="0x00001000" complete_cpuset="0x00001000" online_cpuset="0x00001000" allowed_cpuset="0x00001000"/>
</object>
<object type="Core" os_index="13" cpuset="0x00002000" complete_cpuset="0x00002000" online_cpuset="0x00002000" allowed_cpuset="0x00002000">
<object type="PU" os_index="13" cpuset="0x00002000" complete_cpuset="0x00002000" online_cpuset="0x00002000" allowed_cpuset="0x00002000"/>
</object>
<object type="Core" os_index="14" cpuset="0x00004000" complete_cpuset="0x00004000" online_cpuset="0x00004000" allowed_cpuset="0x00004000">
<object type="PU" os_index="14" cpuset="0x00004000" complete_cpuset="0x00004000" online_cpuset="0x00004000" allowed_cpuset="0x00004000"/>
</object>
<object type="Core" os_index="15" cpuset="0x00008000" complete_cpuset="0x00008000" online_cpuset="0x00008000" allowed_cpuset="0x00008000">
<object type="PU" os_index="15" cpuset="0x00008000" complete_cpuset="0x00008000" online_cpuset="0x00008000" allowed_cpuset="0x00008000"/>
</object>
</object>
</object>
<object type="Socket" os_index="1" cpuset="0xffff0000" complete_cpuset="0xffff0000" online_cpuset="0xffff0000" allowed_cpuset="0xffff0000">
<object type="NUMANode" os_index="2" cpuset="0x00ff0000" complete_cpuset="0x00ff0000" online_cpuset="0x00ff0000" allowed_cpuset="0x00ff0000">
<object type="Core" os_index="16" cpuset="0x00010000" complete_cpuset="0x00010000" online_cpuset="0x00010000" allowed_cpuset="0x00010000">
<object type="PU" os_index="16" cpuset="0x00010000" complete_cpuset="0x00010000" online_cpuset="0x00010000" allowed_cpuset="0x00010000"/>
</object>
<object type="Core" os_index="17" cpuset="0x00020000" complete_cpuset="0x00020000" online_cpuset="0x00020000" allowed_cpuset="0x00020000">
<object type="PU" os_index="17" cpuset="0x00020000" complete_cpuset="0x00020000" online_cpuset="0x00020000" allowed_cpuset="0x00020000"/>
</object>
<object type="Core" os_index="18" cpuset="0x00040000" complete_cpuset="0x00040000" online_cpuset="0x00040000" allowed_cpuset="0x00040000">
<object type="PU" os_index="18" cpuset="0x00040000" complete_cpuset="0x00040000" online_cpuset="0x00040000" allowed_cpuset="0x00040000"/>
</object>
<object type="Core" os_index="19" cpuset="0x00080000" complete_cpuset="0x00080000" online_cpuset="0x00080000" allowed_cpuset="0x00080000">
<object type="PU" os_index="19" cpuset="0x00080000" complete_cpuset="0x00080000" online_cpuset="0x00080000" allowed_cpuset="0x00080000"/>
</object>
<object type="Core" os_index="20" cpuset="0x00100000" complete_cpuset="0x00100000" online_cpuset="0x00100000" allowed_cpuset="0x00100000">
<object type="PU" os_index="20" cpuset="0x00100000" complete_cpuset="0x00100000" online_cpuset="0x00100000" allowed_cpuset="0x00100000"/>
</object>
<object type="Core" os_index="21" cpuset="0x00200000" complete_cpuset="0x00200000" online_cpuset="0x00200000" allowed_cpuset="0x00200000">
<object type="PU" os_index="21" cpuset="0x00200000" complete_cpuset="0x00200000" online_cpuset="0x00200000" allowed_cpuset="0x00200000"/>
</object>
<object type="Core" os_index="22" cpuset="0x00400000" complete_cpuset="0x00400000" online_cpuset="0x00400000" allowed_cpuset="0x00400000">
<object type="PU" os_index="22" cpuset="0x00400000" complete_cpuset="0x00400000" online_cpuset="0x00400000" allowed_cpuset="0x00400000"/>
</object>
<object type="Core" os_index="23" cpuset="0x00800000" complete_cpuset="0x00800000" online_cpuset="0x00800000" allowed_cpuset="0x00800000">
<object type="PU" os_index="23" cpuset="0x00800000" complete_cpuset="0x00800000" online_cpuset="0x00800000" allowed_cpuset="0x00800000"/>
</object>
</object>
<object type="NUMANode" os_index="3" cpuset="0xff000000" complete_cpuset="0xff000000" online_cpuset="0xff000000" allowed_cpuset="0xff000000">
<object type="Core" os_index="24" cpuset="0x01000000" complete_cpuset="0x01000000" online_cpuset="0x01000000" allowed_cpuset="0x01000000">
<object type="PU" os_index="24" cpuset="0x01000000" complete_cpuset="0x01000000" online_cpuset="0x01000000" allowed_cpuset="0x01000000"/>
</object>
<object type="Core" os_index="25" cpuset="0x02000000" complete_cpuset="0x02000000" online_cpuset="0x02000000" allowed_cpuset="0x02000000">
<object type="PU" os_index="25" cpuset="0x02000000" complete_cpuset="0x02000000" online_cpuset="0x02000000" allowed_cpuset="0x02000000"/>
</object>
<object type="Core" os_index="26" cpuset="0x04000000" complete_cpuset="0x04000000" online_cpuset="0x04000000" allowed_cpuset="0x04000000">
<object type="PU" os_index="26" cpuset="0x04000000" complete_cpuset="0x04000000" online_cpuset="0x04000000" allowed_cpuset="0x04000000"/>
</object>
<object type="Core" os_index="27" cpuset="0x08000000" complete_cpuset="0x08000000" online_cpuset="0x08000000" allowed_cpuset="0x08000000">
<object type="PU" os_index="27" cpuset="0x08000000" complete_cpuset="0x08000000" online_cpuset="0x08000000" allowed_cpuset="0x08000000"/>
</object>
<object type="Core" os_index="28" cpuset="0x10000000" complete_cpuset="0x10000000" online_cpuset="0x10000000" allowed_cpuset="0x10000000">
<object type="PU" os_index="28" cpuset="0x10000000" complete_cpuset="0x10000000" online_cpuset="0x10000000" allowed_cpuset="0x10000000"/>
</object>
<object type="Core" os_index="29" cpuset="0x20000000" complete_cpuset="0x20000000" online_cpuset="0x20000000" allowed_cpuset="0x20000000">
<object type="PU" os_index="29" cpuset="0x20000000" complete_cpuset="0x20000000" online_cpuset="0x20000000" allowed_cpuset="0x20000000"/>
</object>
<object type="Core" os_index="30" cpuset="0x40000000" complete_cpuset="0x40000000" online_cpuset="0x40000000" allowed_cpuset="0x40000000">
<object type="PU" os_index="30" cpuset="0x40000000" complete_cpuset="0x40000000" online_cpuset="0x40000000" allowed_cpuset="0x40000000"/>
</object>
<object type="Core" os_index="31" cpuset="0x80000000" complete_cpuset="0x80000000" online_cpuset="0x80000000" allowed_cpuset="0x80000000">
<object type="PU" os_index="31" cpuset="0x80000000" complete_cpuset="0x80000000" online_cpuset="0x80000000" allowed_cpuset="0x80000000"/>
</object>
</object>
</object>
<object type="Socket" os_index="2" cpuset="0x0000ffff,0x0" complete_cpuset="0x0000ffff,0x0" online_cpuset="0x0000ffff,0x0" allowed_cpuset="0x0000ffff,0x0">
<object type="NUMANode" os_index="4" cpuset="0x000000ff,0x0" complete_cpuset="0x000000ff,0x0" online_cpuset="0x000000ff,0x0" allowed_cpuset="0x000000ff,0x0">
<object type="Core" os_index="32" cpuset="0x00000001,0x0" complete_cpuset="0x00000001,0x0" online_cpuset="0x00000001,0x0" allowed_cpuset="0x00000001,0x0">
<object type="PU" os_index="32" cpuset="0x00000001,0x0" complete_cpuset="0x00000001,0x0" online_cpuset="0x00000001,0x0" allowed_cpuset="0x00000001,0x0"/>
</object>
<object type="Core" os_index="33" cpuset="0x00000002,0x0" complete_cpuset="0x00000002,0x0" online_cpuset="0x00000002,0x0" allowed_cpuset="0x00000002,0x0">
<object type="PU" os_index="33" cpuset="0x00000002,0x0" complete_cpuset="0x00000002,0x0" online_cpuset="0x00000002,0x0" allowed_cpuset="0x00000002,0x0"/>
</object>
<object type="Core" os_index="34" cpuset="0x00000004,0x0" complete_cpuset="0x00000004,0x0" online_cpuset="0x00000004,0x0" allowed_cpuset="0x00000004,0x0">
<object type="PU" os_index="34" cpuset="0x00000004,0x0" complete_cpuset="0x00000004,0x0" online_cpuset="0x00000004,0x0" allowed_cpuset="0x00000004,0x0"/>
</object>
<object type="Core" os_index="35" cpuset="0x00000008,0x0" complete_cpuset="0x00000008,0x0" online_cpuset="0x00000008,0x0" allowed_cpuset="0x00000008,0x0">
<object type="PU" os_index="35" cpuset="0x00000008,0x0" complete_cpuset="0x00000008,0x0" online_cpuset="0x00000008,0x0" allowed_cpuset="0x00000008,0x0"/>
</object>
<object type="Core" os_index="36" cpuset="0x00000010,0x0" complete_cpuset="0x00000010,0x0" online_cpuset="0x00000010,0x0" allowed_cpuset="0x00000010,0x0">
<object type="PU" os_index="36" cpuset="0x00000010,0x0" complete_cpuset="0x00000010,0x0" online_cpuset="0x00000010,0x0" allowed_cpuset="0x00000010,0x0"/>
</object>
<object type="Core" os_index="37" cpuset="0x00000020,0x0" complete_cpuset="0x00000020,0x0" online_cpuset="0x00000020,0x0" allowed_cpuset="0x00000020,0x0">
<object type="PU" os_index="37" cpuset="0x00000020,0x0" complete_cpuset="0x00000020,0x0" online_cpuset="0x00000020,0x0" allowed_cpuset="0x00000020,0x0"/>
</object>
<object type="Core" os_index="38" cpuset="0x00000040,0x0" complete_cpuset="0x00000040,0x0" online_cpuset="0x00000040,0x0" allowed_cpuset="0x00000040,0x0">
<object type="PU" os_index="38" cpuset="0x00000040,0x0" complete_cpuset="0x00000040,0x0" online_cpuset="0x00000040,0x0" allowed_cpuset="0x00000040,0x0"/>
</object>
<object type="Core" os_index="39" cpuset="0x00000080,0x0" complete_cpuset="0x00000080,0x0" online_cpuset="0x00000080,0x0" allowed_cpuset="0x00000080,0x0">
<object type="PU" os_index="39" cpuset="0x00000080,0x0" complete_cpuset="0x00000080,0x0" online_cpuset="0x00000080,0x0" allowed_cpuset="0x00000080,0x0"/>
</object>
</object>
<object type="NUMANode" os_index="5" cpuset="0x0000ff00,0x0" complete_cpuset="0x0000ff00,0x0" online_cpuset="0x0000ff00,0x0" allowed_cpuset="0x0000ff00,0x0">
<object type="Core" os_index="40" cpuset="0x00000100,0x0" complete_cpuset="0x00000100,0x0" online_cpuset="0x00000100,0x0" allowed_cpuset="0x00000100,0x0">
<object type="PU" os_index="40" cpuset="0x00000100,0x0" complete_cpuset="0x00000100,0x0" online_cpuset="0x00000100,0x0" allowed_cpuset="0x00000100,0x0"/>
</object>
<object type="Core" os_index="41" cpuset="0x00000200,0x0" complete_cpuset="0x00000200,0x0" online_cpuset="0x00000200,0x0" allowed_cpuset="0x00000200,0x0">
<object type="PU" os_index="41" cpuset="0x00000200,0x0" complete_cpuset="0x00000200,0x0" online_cpuset="0x00000200,0x0" allowed_cpuset="0x00000200,0x0"/>
</object>
<object type="Core" os_index="42" cpuset="0x00000400,0x0" complete_cpuset="0x00000400,0x0" online_cpuset="0x00000400,0x0" allowed_cpuset="0x00000400,0x0">
<object type="PU" os_index="42" cpuset="0x00000400,0x0" complete_cpuset="0x00000400,0x0" online_cpuset="0x00000400,0x0" allowed_cpuset="0x00000400,0x0"/>
</object>
<object type="Core" os_index="43" cpuset="0x00000800,0x0" complete_cpuset="0x00000800,0x0" online_cpuset="0x00000800,0x0" allowed_cpuset="0x00000800,0x0">
<object type="PU" os_index="43" cpuset="0x00000800,0x0" complete_cpuset="0x00000800,0x0" online_cpuset="0x00000800,0x0" allowed_cpuset="0x00000800,0x0"/>
</object>
<object type="Core" os_index="44" cpuset="0x00001000,0x0" complete_cpuset="0x00001000,0x0" online_cpuset="0x00001000,0x0" allowed_cpuset="0x00001000,0x0">
<object type="PU" os_index="44" cpuset="0x00001000,0x0" complete_cpuset="0x00001000,0x0" online_cpuset="0x00001000,0x0" allowed_cpuset="0x00001000,0x0"/>
</object>
<object type="Core" os_index="45" cpuset="0x00002000,0x0" complete_cpuset="0x00002000,0x0" online_cpuset="0x00002000,0x0" allowed_cpuset="0x00002000,0x0">
<object type="PU" os_index="45" cpuset="0x00002000,0x0" complete_cpuset="0x00002000,0x0" online_cpuset="0x00002000,0x0" allowed_cpuset="0x00002000,0x0"/>
</object>
<object type="Core" os_index="46" cpuset="0x00004000,0x0" complete_cpuset="0x00004000,0x0" online_cpuset="0x00004000,0x0" allowed_cpuset="0x00004000,0x0">
<object type="PU" os_index="46" cpuset="0x00004000,0x0" complete_cpuset="0x00004000,0x0" online_cpuset="0x00004000,0x0" allowed_cpuset="0x00004000,0x0"/>
</object>
<object type="Core" os_index="47" cpuset="0x00008000,0x0" complete_cpuset="0x00008000,0x0" online_cpuset="0x00008000,0x0" allowed_cpuset="0x00008000,0x0">
<object type="PU" os_index="47" cpuset="0x00008000,0x0" complete_cpuset="0x00008000,0x0" online_cpuset="0x00008000,0x0" allowed_cpuset="0x00008000,0x0"/>
</object>
</object>
</object>
<object type="Socket" os_index="3" cpuset="0xffff0000,0x0" complete_cpuset="0xffff0000,0x0" online_cpuset="0xffff0000,0x0" allowed_cpuset="0xffff0000,0x0">
<object type="NUMANode" os_index="6" cpuset="0x00ff0000,0x0" complete_cpuset="0x00ff0000,0x0" online_cpuset="0x00ff0000,0x0" allowed_cpuset="0x00ff0000,0x0">
<object type="Core" os_index="48" cpuset="0x00010000,0x0" complete_cpuset="0x00010000,0x0" online_cpuset="0x00010000,0x0" allowed_cpuset="0x00010000,0x0">
<object type="PU" os_index="48" cpuset="0x00010000,0x0" complete_cpuset="0x00010000,0x0" online_cpuset="0x00010000,0x0" allowed_cpuset="0x00010000,0x0"/>
</object>
<object type="Core" os_index="49" cpuset="0x00020000,0x0" complete_cpuset="0x00020000,0x0" online_cpuset="0x00020000,0x0" allowed_cpuset="0x00020000,0x0">
<object type="PU" os_index="49" cpuset="0x00020000,0x0" complete_cpuset="0x00020000,0x0" online_cpuset="0x00020000,0x0" allowed_cpuset="0x00020000,0x0"/>
</object>
<object type="Core" os_index="50" cpuset="0x00040000,0x0" complete_cpuset="0x00040000,0x0" online_cpuset="0x00040000,0x0" allowed_cpuset="0x00040000,0x0">
<object type="PU" os_index="50" cpuset="0x00040000,0x0" complete_cpuset="0x00040000,0x0" online_cpuset="0x00040000,0x0" allowed_cpuset="0x00040000,0x0"/>
</object>
<object type="Core" os_index="51" cpuset="0x00080000,0x0" complete_cpuset="0x00080000,0x0" online_cpuset="0x00080000,0x0" allowed_cpuset="0x00080000,0x0">
<object type="PU" os_index="51" cpuset="0x00080000,0x0" complete_cpuset="0x00080000,0x0" online_cpuset="0x00080000,0x0" allowed_cpuset="0x00080000,0x0"/>
</object>
<object type="Core" os_index="52" cpuset="0x00100000,0x0" complete_cpuset="0x00100000,0x0" online_cpuset="0x00100000,0x0" allowed_cpuset="0x00100000,0x0">
<object type="PU" os_index="52" cpuset="0x00100000,0x0" complete_cpuset="0x00100000,0x0" online_cpuset="0x00100000,0x0" allowed_cpuset="0x00100000,0x0"/>
</object>
<object type="Core" os_index="53" cpuset="0x00200000,0x0" complete_cpuset="0x00200000,0x0" online_cpuset="0x00200000,0x0" allowed_cpuset="0x00200000,0x0">
<object type="PU" os_index="53" cpuset="0x00200000,0x0" complete_cpuset="0x00200000,0x0" online_cpuset="0x00200000,0x0" allowed_cpuset="0x00200000,0x0"/>
</object>
<object type="Core" os_index="54" cpuset="0x00400000,0x0" complete_cpuset="0x00400000,0x0" online_cpuset="0x00400000,0x0" allowed_cpuset="0x00400000,0x0">
<object type="PU" os_index="54" cpuset="0x00400000,0x0" complete_cpuset="0x00400000,0x0" online_cpuset="0x00400000,0x0" allowed_cpuset="0x00400000,0x0"/>
</object>
<object type="Core" os_index="55" cpuset="0x00800000,0x0" complete_cpuset="0x00800000,0x0" online_cpuset="0x00800000,0x0" allowed_cpuset="0x00800000,0x0">
<object type="PU" os_index="55" cpuset="0x00800000,0x0" complete_cpuset="0x00800000,0x0" online_cpuset="0x00800000,0x0" allowed_cpuset="0x00800000,0x0"/>
</object>
</object>
<object type="NUMANode" os_index="7" cpuset="0xff000000,0x0" complete_cpuset="0xff000000,0x0" online_cpuset="0xff000000,0x0" allowed_cpuset="0xff000000,0x0">
<object type="Core" os_index="56" cpuset="0x01000000,0x0" complete_cpuset="0x01000000,0x0" online_cpuset="0x01000000,0x0" allowed_cpuset="0x01000000,0x0">
<object type="PU" os_index="56" cpuset="0x01000000,0x0" complete_cpuset="0x01000000,0x0" online_cpuset="0x01000000,0x0" allowed_cpuset="0x01000000,0x0"/>
</object>
<object type="Core" os_index="57" cpuset="0x02000000,0x0" complete_cpuset="0x02000000,0x0" online_cpuset="0x02000000,0x0" allowed_cpuset="0x02000000,0x0">
<object type="PU" os_index="57" cpuset="0x02000000,0x0" complete_cpuset="0x02000000,0x0" online_cpuset="0x02000000,0x0" allowed_cpuset="0x02000000,0x0"/>
</object>
<object type="Core" os_index="58" cpuset="0x04000000,0x0" complete_cpuset="0x04000000,0x0" online_cpuset="0x04000000,0x0" allowed_cpuset="0x04000000,0x0">
<object type="PU" os_index="58" cpuset="0x04000000,0x0" complete_cpuset="0x04000000,0x0" online_cpuset="0x04000000,0x0" allowed_cpuset="0x04000000,0x0"/>
</object>
<object type="Core" os_index="59" cpuset="0x08000000,0x0" complete_cpuset="0x08000000,0x0" online_cpuset="0x08000000,0x0" allowed_cpuset="0x08000000,0x0">
<object type="PU" os_index="59" cpuset="0x08000000,0x0" complete_cpuset="0x08000000,0x0" online_cpuset="0x08000000,0x0" allowed_cpuset="0x08000000,0x0"/>
</object>
<object type="Core" os_index="60" cpuset="0x10000000,0x0" complete_cpuset="0x10000000,0x0" online_cpuset="0x10000000,0x0" allowed_cpuset="0x10000000,0x0">
<object type="PU" os_index="60" cpuset="0x10000000,0x0" complete_cpuset="0x10000000,0x0" online_cpuset="0x10000000,0x0" allowed_cpuset="0x10000000,0x0"/>
</object>
<object type="Core" os_index="61" cpuset="0x20000000,0x0" complete_cpuset="0x20000000,0x0" online_cpuset="0x20000000,0x0" allowed_cpuset="0x20000000,0x0">
<object type="PU" os_index="61" cpuset="0x20000000,0x0" complete_cpuset="0x20000000,0x0" online_cpuset="0x20000000,0x0" allowed_cpuset="0x20000000,0x0"/>
</object>
<object type="Core" os_index="62" cpuset="0x40000000,0x0" complete_cpuset="0x40000000,0x0" online_cpuset="0x40000000,0x0" allowed_cpuset="0x40000000,0x0">
<object type="PU" os_index="62" cpuset="0x40000000,0x0" complete_cpuset="0x40000000,0x0" online_cpuset="0x40000000,0x0" allowed_cpuset="0x40000000,0x0"/>
</object>
<object type="Core" os_index="63" cpuset="0x80000000,0x0" complete_cpuset="0x80000000,0x0" online_cpuset="0x80000000,0x0" allowed_cpuset="0x80000000,0x0">
<object type="PU" os_index="63" cpuset="0x80000000,0x0" complete_cpuset="0x80000000,0x0" online_cpuset="0x80000000,0x0" allowed_cpuset="0x80000000,0x0"/>
</object>
</object>
</object>
</object>
</topology>
