Hi Brice,
> > The box has strange NUMA layout - I will need to check why it is so.
> > Basically, there are 3 NUMA nodes - one includes 2 Sockets, other 2 have
> > one Socket associated to each of it.
>
> Seems strange to me, likely a BIOS bug. Nehalem-EX should always have
> one NUMA node per socket from what I understand.
I will check it if it's intended or a bug. It's a testing box we got from
Intel before official release of Nehalem-EX.
> I think it's expected. You have a asymmetric topology. One NUMA node
> with 16 cores and two nodes with 8 cores. It's quite a mess for
> hwloc-distrib to handle that. And we actually have a trac ticket about
> this, but I didn't think it would be that useful :/
In theory we could add weights to NUMA nodes to account for asymmetric
topology. I personally think that at least at my scenario it's better to tell
to hwloc-distrib to ignore NUMA nodes and work at Socket level. It's more
transparent.
I don't have mutli-node (multiple hosts or machines) scenario. (I guess
machine concept is intended for clusters like Beowulf cluster). I can imagine
that it can be also asymmetric scenario where different machines have different
CPU and memory. Using weights here may be useful but it can be really hard to
come up with some algorithm to automatically compute the weights.
To make long story short: IMHO, current behavior which assume symmetric NUMA
nodes seem to be right to me.
>
> > I have figured-out following way how to tell hwloc-distrib to avoid using
> > NUMANodes when computing CPU distribution:
> >
> > lstopo --ignore NUMANode No_NUMA.xml
> > for A in $(hwloc-distrib --xml No_NUMA.xml --single 8); do taskset ${A}
> > sleep 100 & done
> > lstopo --top fix.pdf
> >
> > I'm wondering if there is a better way how to make "Socket" the top
> > object. Something like:
> > hwloc-distrib --ignore NUMANode --single 8
> > or
> > hwloc-distrib --top_level Socket --single 8
> >
> > would be very useful. Is there something like this already? If not would
> > you consider this as an enhancement?
>
> Indeed, such an option would be an easy way to work around problems with
> asymmetric topologies. I don't know yet whether --ignore or --top-level
> is better. I'll think about it.
IMHO, --top-level would be better.
Thanks a lot!
Jirka
