Brock Palen wrote: > has anyone done work with hwloc on scalemp systems? They provide > their own tool numabind, but we are looking for a more generic > solution to process placement and control that works well inside our > MPI library (openMPI in most cases). > > Any input on this would be great!
Hello Brock, >From what I remember, ScaleMP uses an hypervisor on each node that virtually merges all of them into a fake big shared-memory machine. Then a vanilla Linux kernel runs on top of it. So hwloc should just see regular cores and NUMA node information, assuming the virtual "merged" hardware reports all necessary information to the OS. There's a bit of ScaleMP code in the Linux kernel, but it does pretty much nothing, it does not seem to add anything to /proc or /sys for instance. So I am not sure hwloc could get some specialized knowledge of ScaleMP machines. Maybe their custom numabind tool knows that ScaleMP machines only works on machines with some well-defined types/counts/numbering of processors and NUMA nodes, and thus uses this information to group sockets/NUMA-nodes depending on their physical distance. Brice