Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Brice Goglin]

Le 24/09/2014 00:19, Pedaballe, Vineet a écrit :
> Please find my response inline.
>
> Some of the items related to hard drives & DIMM is available only with root 
> privileges. We should be able to populate objects related to these items only 
> when root privileges are available. Is this an acceptable change to the 
> HWLOC's design? 

We already have this difference for minor PCI attributes

http://www.open-mpi.org/projects/hwloc/doc/v1.9.1/a00028.php#faq_privileged
As long as it doesn't impact major features, that's fine for me.

>
>> 7. Other PCI Devices
>>  a. Device ID
>>  b. Device Serial number (if applicable)
> IIRC the serial number isn't standardized anywhere in the PCI config space, 
> this item is likely impossible.
> [VIN] : Agreed. But if we show it is possible for some devices (like the 
> coprocessors), I was hoping the vendors would eventually contribute to their 
> respective devices. I was hoping all this information is provided as 
> attributes, and these can be optional for each device.(am I correct?)

It's optional, and may be different for each device. So it's possible
for vendors to contribute, but that means a lot of different cases. I
don't expect the list ot be huge and to cover a non-negligible part of
the PCI devices. We'll see.

Brice

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Ralph Castain]

True - but we intend to collect the inventory as root anyway. :-)

On Sep 23, 2014, at 1:50 PM, Christopher Samuel  wrote:

> On 24/09/14 00:57, Ralph Castain wrote:
> 
>> Memory info is available from lshw, though they are a GPL code:
> 
> FWIW on this laptop (Intel Haswell) lshw only report DIMM info when run
> as root, which I suspect would point them to accessing DMI information
> via /dev/mem.
> 
> Using strace supports this:
> 
> 3405  open("/dev/mem", O_RDONLY)= -1 EACCES (Permission denied)
> 
> FWIW dmidecode does the same.
> 
> samuel@haswell:~$ dmidecode
> # dmidecode 2.12
> /dev/mem: Permission denied
> 
> All the best,
> Chris
> -- 
> Christopher SamuelSenior Systems Administrator
> VLSCI - Victorian Life Sciences Computation Initiative
> Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
> http://www.vlsci.org.au/  http://twitter.com/vlsci
> 
> ___
> hwloc-devel mailing list
> hwloc-de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-devel/2014/09/4238.php

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Christopher Samuel]

On 24/09/14 00:57, Ralph Castain wrote:

> Memory info is available from lshw, though they are a GPL code:

FWIW on this laptop (Intel Haswell) lshw only report DIMM info when run
as root, which I suspect would point them to accessing DMI information
via /dev/mem.

Using strace supports this:

3405  open("/dev/mem", O_RDONLY)= -1 EACCES (Permission denied)

FWIW dmidecode does the same.

samuel@haswell:~$ dmidecode
# dmidecode 2.12
/dev/mem: Permission denied

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Brice Goglin]

Le 23/09/2014 21:06, Pedaballe, Vineet a écrit :
> 4a. Network Adapters (Ethernet)
>   a. Model
>   b. Speed

Both supported and currently negociated link speed?

On Linux, we'll have to use the ethtool interface.

>   c. Serial Number (if applicable)
>   d. MAC address
> 4b. Network Adapters (Infiniband)
>   a. Model
>   b. Speed
>   c. Serial Number (if applicable)
>   d. MAC address

We'll to use the verbs or libfabric or whatever interface.

> 4. Host Bus Adapters
>   a. Manufacturer
>   b. Serial Number
>   c. MAC address

What kind of adapter are you talking about here? Hostbridge?

> 6. Coprocessors
>   a. Manufacturer
>   b. Serial Number

We'll need an explicit list of supported coprocessors/accelerators and
specific ways to retrieve the S/N for each of them.

> 7. Other PCI Devices
>   a. Device ID
>   b. Device Serial number (if applicable)

IIRC the serial number isn't standardized anywhere in the PCI config
space, this item is likely impossible.

Brice

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Pedaballe, Vineet]

Thank you Ralph and Jeff. 

This was the list I was hoping to read through hwloc.

3. Memory
a. Total memory
b. Total DIMMS
c. Individual DIMM's:
(i) Serial numbers
(ii) Vendor Name
(iii) Model
(iv) Memory Frequency
4a. Network Adapters (Ethernet)
a. Model
b. Speed
c. Serial Number (if applicable)
d. MAC address
4b. Network Adapters (Infiniband)
a. Model
b. Speed
c. Serial Number (if applicable)
d. MAC address
4. Host Bus Adapters
a. Manufacturer
b. Serial Number
c. MAC address
6. Coprocessors
a. Manufacturer
b. Serial Number
7. Other PCI Devices
a. Device ID
b. Device Serial number (if applicable)
8. HardDrive
a. Model, Form factor, etc.
b. Vendor
c. Serial Number
d. Size


Some of it is already provided by HWLOC. For some other items, a limited set of 
information is provided by HWLOC.

Most of the DIMM related data is provided by the SMBIOS tables. 'dmidecode' 
provides a lot of this information.
The application hdparm provides a lot of information about hard drives. I 
glanced over their code, and it seems like they also udev to retrieve the 
information.


* For Memory, I would suggest have a single object for each node and list the 
DIMM details as attributes for that object.
* For hard drives, we can have similar objects for each SATA0.., etc node, 
whose lanes are usually connected via the PCH to a single socket. Each hard 
drive can have its own object, and all the attributes of the hard drive can be 
stored within that object.
* For the PCI devices I understand it is not simple to read the serial number 
and UUID details, but I know how to read the serial numbers for some 
products(like the Intel coprocessors) and it would be really cool to add this 
also as an attribute to it.

I'm still not entirely familiar with the hwloc's internal architecture, I 
apologize in case I make any wrong assumptions.
How would you like me to proceed from here?

Thanks,
Vineet

-Original Message-
From: hwloc-devel [mailto:hwloc-devel-boun...@open-mpi.org] On Behalf Of Brice 
Goglin
Sent: Tuesday, September 23, 2014 8:48 AM
To: hwloc-de...@open-mpi.org
Subject: Re: [hwloc-devel] Using hwloc to detect Hard Disks

Le 23/09/2014 16:46, Brice Goglin a écrit :
> Le 23/09/2014 16:38, Guy Streeter a écrit :
>> I know that udev gathers this information:
>>
>> # ll /sys/block/sda/bdi
>> lrwxrwxrwx. 1 root root 0 Sep 23 09:33 /sys/block/sda/bdi ->
>> ../../../../../../../../virtual/bdi/8:0
>> # grep SERIAL '/run/udev/data/b8:0'
>> E:ID_SERIAL=SAMSUNG_MZ7TD256HAFV-000L9_S17LNSADC13325
>> E:ID_SERIAL_SHORT=S17LNSADC13325
>>
>> So you could get it from udev or gather it the same way udev does. If 
>> you want to know how udev does it, I can research that.
> If you can get more information, that'd be great. I wonder if they are 
> using ioctls to retrieve these, I can't find anything in sysfs even 
> though udev has similar info on my machines.
>

Indeed there's a struct hd_driveid in linux/hdreg.h that you get with ioctl 
HDIO_GET_IDENTITY on the device.

There's also a libblkid that may help.

Brice

___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
Link to this post: 
http://www.open-mpi.org/community/lists/hwloc-devel/2014/09/4235.php

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Brice Goglin]

Le 23/09/2014 16:46, Brice Goglin a écrit :
> Le 23/09/2014 16:38, Guy Streeter a écrit :
>> I know that udev gathers this information:
>>
>> # ll /sys/block/sda/bdi
>> lrwxrwxrwx. 1 root root 0 Sep 23 09:33 /sys/block/sda/bdi ->
>> ../../../../../../../../virtual/bdi/8:0
>> # grep SERIAL '/run/udev/data/b8:0'
>> E:ID_SERIAL=SAMSUNG_MZ7TD256HAFV-000L9_S17LNSADC13325
>> E:ID_SERIAL_SHORT=S17LNSADC13325
>>
>> So you could get it from udev or gather it the same way udev does. If you
>> want to know how udev does it, I can research that.
> If you can get more information, that'd be great. I wonder if they are
> using ioctls to retrieve these, I can't find anything in sysfs even
> though udev has similar info on my machines.
>

Indeed there's a struct hd_driveid in linux/hdreg.h that you get with
ioctl HDIO_GET_IDENTITY on the device.

There's also a libblkid that may help.

Brice

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Ralph Castain]

Memory info is available from lshw, though they are a GPL code:

 *-bank:0
  description: DIMM Synchronous 1333 MHz (0.8 ns)
  product: M393B1K70DH0-YH9
  vendor: 0x80CE
  physical id: 0
  serial: 0x85B5FED3
  slot: DIMM_A1
  size: 8GiB
  width: 64 bits
  clock: 1333MHz (0.8ns)

Not sure how they are getting it, but I can have someone look at the code to 
see where the info is being obtained.


On Sep 22, 2014, at 8:54 PM, Ralph Castain  wrote:

> 
> On Sep 22, 2014, at 4:58 PM, Jeff Squyres (jsquyres)  
> wrote:
> 
>> On Sep 22, 2014, at 6:55 PM, Brice Goglin  wrote:
>> 
 HWLOC already provides similar info for processors and mother boards, so 
 it seemed a natural extension of current capabilities to provide it for 
 other system elements.
>>> 
>>> Disk vendor/model is easy to add from sysfs on Linux. I don't know where
>>> to find the serial number. Spindle speed may require more than just
>>> sysfs. Do you have more info on how to get these attributes?
>>> 
>>> For memory, we currently have a single memory object for all DIMMs of a
>>> single NUMA node. Adding multiple objects may not be useful, but adding
>>> many serials to a single NUMA object may be ugly.
>>> There are some information about physical memory in
>>> /sys/devices/system/node/node0/memory* but it doesn't correspond to
>>> DIMMs (I have 135 of them on my laptop for only 2 SODIMMs). dmidecode
>>> gets DIMM info somehow.
>> 
>> Back in Nehalem days, it wasn't possible to map Linux kernel "physical" 
>> memory back to individual DIMMs (because the BIOS could/would introduce 
>> another layer of kernel<-->DIMM mapping that the kernel might not be aware 
>> of).
>> 
>> Has that changed?
> 
> I don't think so, no - at least, I'm not sure you can map a specific DIMM to 
> a specific address within a NUMA region. However, we can at least add the 
> DIMMs to the root-object attributes. In addition, you can certainly map a 
> DIMM to a specific DIMM socket, and I believe that means you can map it to a 
> given NUMA region even if you can't say *where* it is within that region. 
> Have to verify that.
> 
> 
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> ___
>> hwloc-devel mailing list
>> hwloc-de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/hwloc-devel/2014/09/4229.php

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Brice Goglin]

Le 23/09/2014 16:38, Guy Streeter a écrit :
> I know that udev gathers this information:
>
> # ll /sys/block/sda/bdi
> lrwxrwxrwx. 1 root root 0 Sep 23 09:33 /sys/block/sda/bdi ->
> ../../../../../../../../virtual/bdi/8:0
> # grep SERIAL '/run/udev/data/b8:0'
> E:ID_SERIAL=SAMSUNG_MZ7TD256HAFV-000L9_S17LNSADC13325
> E:ID_SERIAL_SHORT=S17LNSADC13325
>
> So you could get it from udev or gather it the same way udev does. If you
> want to know how udev does it, I can research that.

If you can get more information, that'd be great. I wonder if they are
using ioctls to retrieve these, I can't find anything in sysfs even
though udev has similar info on my machines.

Brice

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Ralph Castain]

On Sep 22, 2014, at 4:58 PM, Jeff Squyres (jsquyres)  wrote:

> On Sep 22, 2014, at 6:55 PM, Brice Goglin  wrote:
> 
>>> HWLOC already provides similar info for processors and mother boards, so it 
>>> seemed a natural extension of current capabilities to provide it for other 
>>> system elements.
>> 
>> Disk vendor/model is easy to add from sysfs on Linux. I don't know where
>> to find the serial number. Spindle speed may require more than just
>> sysfs. Do you have more info on how to get these attributes?
>> 
>> For memory, we currently have a single memory object for all DIMMs of a
>> single NUMA node. Adding multiple objects may not be useful, but adding
>> many serials to a single NUMA object may be ugly.
>> There are some information about physical memory in
>> /sys/devices/system/node/node0/memory* but it doesn't correspond to
>> DIMMs (I have 135 of them on my laptop for only 2 SODIMMs). dmidecode
>> gets DIMM info somehow.
> 
> Back in Nehalem days, it wasn't possible to map Linux kernel "physical" 
> memory back to individual DIMMs (because the BIOS could/would introduce 
> another layer of kernel<-->DIMM mapping that the kernel might not be aware 
> of).
> 
> Has that changed?

I don't think so, no - at least, I'm not sure you can map a specific DIMM to a 
specific address within a NUMA region. However, we can at least add the DIMMs 
to the root-object attributes. In addition, you can certainly map a DIMM to a 
specific DIMM socket, and I believe that means you can map it to a given NUMA 
region even if you can't say *where* it is within that region. Have to verify 
that.


> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> hwloc-devel mailing list
> hwloc-de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/hwloc-devel/2014/09/4229.php

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Jeff Squyres (jsquyres)]

On Sep 22, 2014, at 6:55 PM, Brice Goglin  wrote:

>> HWLOC already provides similar info for processors and mother boards, so it 
>> seemed a natural extension of current capabilities to provide it for other 
>> system elements.
> 
> Disk vendor/model is easy to add from sysfs on Linux. I don't know where
> to find the serial number. Spindle speed may require more than just
> sysfs. Do you have more info on how to get these attributes?
> 
> For memory, we currently have a single memory object for all DIMMs of a
> single NUMA node. Adding multiple objects may not be useful, but adding
> many serials to a single NUMA object may be ugly.
> There are some information about physical memory in
> /sys/devices/system/node/node0/memory* but it doesn't correspond to
> DIMMs (I have 135 of them on my laptop for only 2 SODIMMs). dmidecode
> gets DIMM info somehow.

Back in Nehalem days, it wasn't possible to map Linux kernel "physical" memory 
back to individual DIMMs (because the BIOS could/would introduce another layer 
of kernel<-->DIMM mapping that the kernel might not be aware of).

Has that changed?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [hwloc-devel] Using hwloc to detect Hard Disks

[Brice Goglin]

Le 22/09/2014 23:28, Ralph Castain a écrit :
>
> I believe that is what we are requesting - and to be clear, Intel is 
> proposing to develop and contribute this feature. We'd like to add attributes 
> to the objects to record:
>
> * memory - the serial number and model number of the DIMMs
>
> * disks - serial, model, manufacturer and any other available info (sometimes 
> they have spindle speed, for example)
>
> HWLOC already provides similar info for processors and mother boards, so it 
> seemed a natural extension of current capabilities to provide it for other 
> system elements.

Disk vendor/model is easy to add from sysfs on Linux. I don't know where
to find the serial number. Spindle speed may require more than just
sysfs. Do you have more info on how to get these attributes?

For memory, we currently have a single memory object for all DIMMs of a
single NUMA node. Adding multiple objects may not be useful, but adding
many serials to a single NUMA object may be ugly.
There are some information about physical memory in
/sys/devices/system/node/node0/memory* but it doesn't correspond to
DIMMs (I have 135 of them on my laptop for only 2 SODIMMs). dmidecode
gets DIMM info somehow.

Brice