Re: [hwloc-devel] PCI device location in hwloc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 19/11/10 10:04, Brice Goglin wrote: > Many dual-nehalem-EP boxes have a single I/O hub which is connected to > both sockets, so no I/O affinity there. If your machine wasn't designed > to have many big PCIe slots (e.g. to plug multiple GPUs), that's > probably what's happening here. That would make sense, these are their SuperMicro based systems, nothing unusual on these. cheers! Chris - -- Christopher Samuel - Senior Systems Administrator VLSCI - Victorian Life Sciences Computational Initiative Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.unimelb.edu.au/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkzlsnEACgkQO2KABBYQAh9GBQCfVobPoqVMAFuKBtEB6bJLrj9m 9vAAnRUHkCado+u2GB8CM7bpyVUYne76 =/0sJ -END PGP SIGNATURE-
Re: [hwloc-devel] PCI device location in hwloc
Le 18/11/2010 16:58, Samuel Thibault a écrit : > Hello, > > Christopher Samuel, le Thu 18 Nov 2010 23:47:20 +0100, a écrit : > >> Does the information occur to the right of the socket with >> the closest distance to the devices ? >> > If your case, hwloc was apparently unable to decide whether the devices > where "inside" one or the other NUMA node. Could you also post the > output of gather-topology? > Many dual-nehalem-EP boxes have a single I/O hub which is connected to both sockets, so no I/O affinity there. If your machine wasn't designed to have many big PCIe slots (e.g. to plug multiple GPUs), that's probably what's happening here. Otherwise, it could be a machine with two I/O hubs (any multiple big PCIe slots) and the BIOS not saying which one is close to which socket. Brice
Re: [hwloc-devel] PCI device location in hwloc
Hello, Christopher Samuel, le Thu 18 Nov 2010 23:47:20 +0100, a écrit : > Does the information occur to the right of the socket with > the closest distance to the devices ? If your case, hwloc was apparently unable to decide whether the devices where "inside" one or the other NUMA node. Could you also post the output of gather-topology? > Well at the moment that looks rather nice to me, though it > would indeed be good to see GPUs labelled to - though I've > seen your comment in the source saying: > > /* FIXME: what about gpus? could try class "drm", but >proprietary drivers won't appear there */ > > I don't have any boxes at all with GPUs in them, so I'm not > sure what to suggest there. :-( Actually we might as well just first implement the modularized version of hwloc, so that we can dynamically load a cuda-based module which will just show a proper CUDA device. In the meanwhile, we could special-case a few things. We probably don't want to emit the whole pciids identifier (e.g. 85:00.0 VGA compatible controller: nVidia Corporation GT200GL [Quadro FX 5800] (rev a1) ), but in this case we could handle the nvidia vendorId specially: parse the pciid to extract Quadro FX 5800 and display that. Samuel
Re: [hwloc-devel] hwloc to be included in RHEL 6.1
On Thursday, November 18, 2010 03:55:35 pm Brice Goglin wrote:
> Le 18/11/2010 08:50, Jirka Hladky a écrit :
> > Hi all,
> >
> > Red Hat would like to included hwloc in the upcoming version of the Red
> > Hat Enterprise Linux 6.1. There is Bugzilla 648593
> > [RFE] Include Portable Hardware Locality (hwloc) in RHEL
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=648593
> >
> > to address this.
> >
> > I got following input from the devel:
> > =
> > There appears to be a significant drawback to using hwloc. The core #
> > shown in hwloc-ls does NOT map 1:1 with the processor id in
> > /proc/cpuinfo.
> >
> > For example, on intel-s3e36-02.lab hwloc shows the core ids in socket 0
> > as {0,1,2,3,4,5,6,7}.
> >
> > /proc/cpuinfo shows these as physically being {0,4,8,12,16,20,24,28}.
> >
> > On the cmd-line, hwloc-ls does indicate a difference between the hwloc
> > core id and the physical id:
> >
> > [root@intel-s3e36-02 ~]# hwloc-ls
> > Machine (64GB)
> >
> > NUMANode #0 (phys=0 16GB) + Socket #0 + L3 #0 (24MB)
> >
> > L2 #0 (256KB) + L1 #0 (32KB) + Core #0 + PU #0 (phys=0)
> > L2 #1 (256KB) + L1 #1 (32KB) + Core #1 + PU #1 (phys=4)
> > L2 #2 (256KB) + L1 #2 (32KB) + Core #2 + PU #2 (phys=8)
> > L2 #3 (256KB) + L1 #3 (32KB) + Core #3 + PU #3 (phys=12)
> > L2 #4 (256KB) + L1 #4 (32KB) + Core #4 + PU #4 (phys=16)
> > L2 #5 (256KB) + L1 #5 (32KB) + Core #5 + PU #5 (phys=20)
> > L2 #6 (256KB) + L1 #6 (32KB) + Core #6 + PU #6 (phys=24)
> > L2 #7 (256KB) + L1 #7 (32KB) + Core #7 + PU #7 (phys=28)
> >
> > If you use the graphical interface, it is possible that
> > customers/GSS/everyone screws up the reporting of CPU #s.
> >
> > Possible solution: Have hwloc-ls use '-p' by default.
> > =
> >
> > I'm not sure if you are open to change the default from --logical to --
> > physical. Please let me know your opinion on it. If you don't think that
> > it's a good idea perhaps you can give us arguments why you prefer
> > logical indexing over physical indexing.
Hi Brice,
> We want to keep a consistent default across the whole project. The API,
> hwloc-calc and hwloc-bind use logical by default.
I do agree.
>
> > Another point is that at the moment you cannot distinguish if the
> > graphical output (.png, X, ...) was created with lstopo --physical or
> > lstopo --logical.
> Actually, you can. Instead of "Core #0", you get "Core p#0" (this "p"
> means "physical").
Oh, you are right. I didn't notice it. For the novice user, it will be
difficult
to notice the difference.
Actually, I had the same problem when I have first started lstopo. I was
wondering how these indexes match with /proc/cpuinfo indexing.
> > Could you please add the legend to the picture explaining which index was
> > used?
>
> I guess it's possible.
Oh, this would be great! Will it make it into 1.1?
Jirka
Re: [hwloc-devel] hwloc to be included in RHEL 6.1
Le 18/11/2010 08:50, Jirka Hladky a écrit :
> Hi all,
>
> Red Hat would like to included hwloc in the upcoming version of the Red Hat
> Enterprise Linux 6.1. There is Bugzilla 648593
> [RFE] Include Portable Hardware Locality (hwloc) in RHEL
>
> https://bugzilla.redhat.com/show_bug.cgi?id=648593
>
> to address this.
>
> I got following input from the devel:
> =
> There appears to be a significant drawback to using hwloc. The core # shown
> in
> hwloc-ls does NOT map 1:1 with the processor id in /proc/cpuinfo.
>
> For example, on intel-s3e36-02.lab hwloc shows the core ids in socket 0 as
> {0,1,2,3,4,5,6,7}.
>
> /proc/cpuinfo shows these as physically being {0,4,8,12,16,20,24,28}.
>
> On the cmd-line, hwloc-ls does indicate a difference between the hwloc core id
> and the physical id:
>
> [root@intel-s3e36-02 ~]# hwloc-ls
> Machine (64GB)
> NUMANode #0 (phys=0 16GB) + Socket #0 + L3 #0 (24MB)
> L2 #0 (256KB) + L1 #0 (32KB) + Core #0 + PU #0 (phys=0)
> L2 #1 (256KB) + L1 #1 (32KB) + Core #1 + PU #1 (phys=4)
> L2 #2 (256KB) + L1 #2 (32KB) + Core #2 + PU #2 (phys=8)
> L2 #3 (256KB) + L1 #3 (32KB) + Core #3 + PU #3 (phys=12)
> L2 #4 (256KB) + L1 #4 (32KB) + Core #4 + PU #4 (phys=16)
> L2 #5 (256KB) + L1 #5 (32KB) + Core #5 + PU #5 (phys=20)
> L2 #6 (256KB) + L1 #6 (32KB) + Core #6 + PU #6 (phys=24)
> L2 #7 (256KB) + L1 #7 (32KB) + Core #7 + PU #7 (phys=28)
>
> If you use the graphical interface, it is possible that customers/GSS/everyone
> screws up the reporting of CPU #s.
>
> Possible solution: Have hwloc-ls use '-p' by default.
> =
>
> I'm not sure if you are open to change the default from --logical to --
> physical. Please let me know your opinion on it. If you don't think that it's
> a good idea perhaps you can give us arguments why you prefer logical indexing
> over physical indexing.
>
We want to keep a consistent default across the whole project. The API,
hwloc-calc and hwloc-bind use logical by default.
> Another point is that at the moment you cannot distinguish if the graphical
> output (.png, X, ...) was created with lstopo --physical or lstopo --logical.
>
Actually, you can. Instead of "Core #0", you get "Core p#0" (this "p"
means "physical").
> Could you please add the legend to the picture explaining which index was
> used?
>
I guess it's possible.
Brice
Re: [hwloc-devel] PCI device location in hwloc
Hello Chris, It's not in trunk yet. Try this branch instead: https://svn.open-mpi.org/svn/hwloc/branches/libpci We are not sure yet how we should expose all these devices. Right now, we have a new BRIDGE type, a PCI_DEVICE type, and a OS_DEVICE type (which corresponds to OS names such as eth0 or sda1 or... nothing for gpus). If you have any feeling about all this, please tell us :) Brice Le 18/11/2010 08:15, Christopher Samuel a écrit : > Hi there, > > At the hwloc talk on the Cisco booth by Brice he mentioned > work to locate PCI bridges & devices and their socket > locality and put up a nice graphic which was very interesting. > > So last night I grabbed the latest snapshot of trunk but > couldn't see how to give me that info. > > Was I being too enthusiastic ? :-) > > Is this in a private branch or was it a mockup ? > > cheers! > Chris ___ hwloc-devel mailing list hwloc-de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
Re: [hwloc-devel] hwloc-distrib --among
Hi Samuel, thanks for looking into it! I'm using hwloc_distribute to distribute parallel jobs on multi-socket systems. Usually, it gives nice results: running hwloc-distrib --single on box with sockets will ditrbitute one job per socket. This is what I want. hwloc-distrib --single <2*N> will distribute 2 jobs per socket, picking-up PU wisely. It breaks however on strange systems. Please check with lstopo --input or hwloc-distrib --input on topology I sent you with my last e-mail (hp-dl980g7-01.tar.bz2, sent on Tuesday 09:30:37 pm) This box has a broken NUMA topology - there are 7 sockets in one NUMA node and 1 socket in another NUMA node. My goal is to distribute one job per Socket with command hwloc-distrib --single 8 This is not working. So I have tried various --among and -ignore options to achieve this but without success. Please try hwloc-distrib --input hp-dl980g7-01 --single 8 with data I sent you on Tuesday (tar jxvf hp-dl980g7-01.tar.bz2). Goal is to distribute one job per one socket. Thanks! Jirka On Tuesday, November 16, 2010 10:20:38 pm Samuel Thibault wrote: > Samuel Thibault, le Tue 16 Nov 2010 22:18:54 +0100, a écrit : > > Also note that currently the hwloc_distribute() function doesn't take > > e.g. the number of PUs into account when splitting elements over the > > hierarchy. It was more a demonstration example than something to be used > > as is. We can however extend it, we just need to know what's desired. > > Reading your mail again, I guess that's where your issue actually lied. > > Samuel > ___ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel hp-dl980g7-01.tar.bz2 Description: application/bzip-compressed-tar
Re: [hwloc-devel] hwloc-distrib --among
On Tuesday, November 16, 2010 10:31:11 pm Brice Goglin wrote:
> Le 16/11/2010 15:18, Samuel Thibault a écrit :
> > Jirka Hladky, le Tue 16 Nov 2010 21:37:01 +0100, a écrit :
> >> There was some discussion about hwloc-distrib --among
> >>
> >> If I understand it correctly, --among accepts one of
> >> {pu,core,socket,node,machine}
> >
> > I actually didn't know about the --among option. It seems a bit
> > difficult to comprehend without reading the source code... Actually I'm
> > not even sure about the cases where it is useful.
>
Hi Brice,
> I don't remember the exact use case but it was basically a machine where
> the topology strange or broken and the user wanted to force the
> distribution among some given object type (the distribution of objects
> of other types was unusable).
Yes, I would need such option! Please check hp-dl980g7-01.tar.bz2 attached to
my previous e-mail for an example of strange topology (uneven NUMA
distribution).
The problem is that neither --among nor --ignore works for me. Perhaps I'm
using it in a wrong way? It seems also that --among does not check if the
input is valid. Try
hwloc-distrib --single --among blabla 4
Thanks!
Jirka
