Re: [hwloc-devel] memory size attributes

[Jeff Squyres]

On Jan 19, 2010, at 3:15 PM, Brice Goglin wrote:

> IIRC, it was changed into a pointer before the first release to reduce
> ABI changes inside the the main hwloc_obj struct when attributes are
> changed. I don't think hwloc_obj and attributes will be stable before
> hwloc 2.0 so I don't really care whether it's a pointer or not :)

Ah -- preserving ABI goodness is a Good Thing.  Keeping it a pointer won't 
guarantee us ABI compatibility, of course, but it helps.  So disregard my prior 
comments -- keep it a pointer.  Might as well give it a good college try to be 
ABI friendly.

-- 
Jeff Squyres
jsquy...@cisco.com

Re: [hwloc-devel] memory size attributes

[Samuel Thibault]

Brice Goglin, le Tue 26 Jan 2010 16:09:13 +0100, a écrit :
> = 0 ? if some OS fail to give the size of normal pages or huge pages, we
> might have count !=0 while size = 0 in some cases.

In that case the size wouldn't be advertised at all. count == 0 is
however plausible when you run out of big pages :)

Samuel

Re: [hwloc-devel] memory size attributes

[Brice Goglin]

I have pushed some changes in the memoryattrs branch. See the new memory
attributes starting at line 138 of
https://svn.open-mpi.org/trac/hwloc/browser/branches/memoryattrs/include/hwloc.h

I am still working on the XML backend. It'd be good if people could
complain about the API before I convert everything else.

Random things to think about:
* should we enforce the ordering of pages size+count structures ? I am
sorting by page size for now
* how is the pages array terminated ? size = 0 ? or both size and count
= 0 ? if some OS fail to give the size of normal pages or huge pages, we
might have count !=0 while size = 0 in some cases.
* or should we add pages_count to the memory strcuture to explictly
store the length of the pages array ?

Brice

Re: [hwloc-devel] memory size attributes

[Samuel Thibault]

Jeff Squyres, le Tue 19 Jan 2010 07:03:55 -0500, a écrit :
> On Jan 17, 2010, at 6:31 AM, Brice Goglin wrote:
> 
> > Right now we have total amount + number of hugepages + size of
> > hugepages. I was only talking about modifying the way to store all this
> > so as it'd be to easier to add multiple hugepage size x number one day.
> > I wasn't talking about adding normal page size x number.
> 
> This may be a dumb question, but can there be multiple sized pages?

Yes.

Solaris even permits to choose various page sizes at least for different
processes (through a system call), maybe even for the same process, I
can't remember.

Samuel

Re: [hwloc-devel] memory size attributes

[Jeff Squyres]

On Jan 19, 2010, at 3:20 AM, Brice Goglin wrote:

> I propose the following changes to the object attributes. struct
> hwloc_obj now contains a struct hwloc_memory pointer:

Is there any reason not to make this a sub-struct included in hwloc_obj (vs. a 
pointer that requires an additional malloc)?  I'm not saying that malloc 
performance and/or the cost of dereferencing the pointer is a big deal -- it 
just feels "un-tidy" to have another malloc for something that will be used in 
many situations.

> struct hwloc_memory {
>   unsigned long total_kB; /* Total memory in this object and its children */
>   unsigned long local_kB; /* Local memory in this object */

FWIW, I'm not wild about the capitol "B" at the end, but that's mainly because 
I'm lazy.  ;-)  How about total_mem and local_mem, and we specify that they are 
in units of bytes?

I say "bytes" instead of "kilobytes" because it might be easier to avoid 
getting into the "what does kilo mean -- 1000 or 1024?" arguments.  

Also, do we want to make them uint64_t's?  A system's total memory could get 
pretty large -- more than 4B bytes on 32 bit systems...?  Hmm.  I'm indecisive 
on this one -- I can think of reasons for and against...  I guess I just prefer 
future-proofing as much as possible.

>   struct hwloc_memory_pages {
> unsigned long size;
> unsigned long count;
>   } * local_pages; /* 0-terminated array of pages or hugepages sizes and 
> counts */
> }

In the worst case, we've added 2*sizeof(long)+sizeof(struct*) to the size of 
the hwloc_obj struct for objects that don't have memory.  But I don't think 
that's a big deal, right?

> This obj->memory could be NULL except for Node, Machine and maybe
> System.

Or maybe just total and local memory set to 0?

> > I'd say it can be valuable to support key=value pairs on any object so
> > that future object types can be extensible (e.g., vendor pci devices).
> > But common stuff should be accessible as struct members so that
> > there's no string parsing needed (I'm no doubt just voicing what we
> > all already think). I.e., esoteric stuff can start as a key=value
> > strings but as they get mature / popular, they can "graduate" to
> > become a struct member.
> 
> If we do the above, Node and System won't have any specific attribute
> anymore. Machine will still have DMI infos. But I am not sure they are
> common enough and deserve an actual struct field, they could stay in the
> array of stringified info.

I'm not sure I understand what you mean here -- are you thinking of moving the 
DMI fields to the general key=value arrays?  What is "DMI"?

> Actually, maybe obj->name should go there as well before it mixes to
> many meanings. It's mostly unused in trunk, except to store "Kerrighed"
> since very recently. We talked about using it to store the hostname of
> Machine objects. And the libpci branch uses it to store the PCI
> vendor/model string and OS device names. Maybe all these shouldn't be in
> obj->name and rather in the array of strings as OSName=Kerrighed,
> Hostname=foobar, PCIVendor=foo, PCIModel=bar, OSDevice=eth2, ...

How about both?  It can be useful to have a simple/common field to print a 
human-readable text field (even if it's only a summary and the real details are 
in key=value pairs).

-- 
Jeff Squyres
jsquy...@cisco.com

Re: [hwloc-devel] memory size attributes

[Jeff Squyres]

On Jan 17, 2010, at 6:31 AM, Brice Goglin wrote:

> Right now we have total amount + number of hugepages + size of
> hugepages. I was only talking about modifying the way to store all this
> so as it'd be to easier to add multiple hugepage size x number one day.
> I wasn't talking about adding normal page size x number.

This may be a dumb question, but can there be multiple sized pages?

E.g., can a system of multiple machines have page size X on some machines and 
page size Y on other machines?

Additionally, I'm a little concerned about saying "hugepage" today, when that 
might be the normal tomorrow (kind of like calling something "new" -- someday, 
it won't be new anymore).  I'd rather either specify the page size or not; not 
use an adjective that may or may not have meaning someday / on some platforms.

-- 
Jeff Squyres
jsquy...@cisco.com

Re: [hwloc-devel] memory size attributes

[Brice Goglin]

Jeff Squyres (jsquyres) wrote:
>
> Just so I understand - are you saying hwloc should track both the
> total amount of memory *and* the makeup of that amount, broken up by
> page size?
>

I don't know if we need this.

Right now we have total amount + number of hugepages + size of
hugepages. I was only talking about modifying the way to store all this
so as it'd be to easier to add multiple hugepage size x number one day.
I wasn't talking about adding normal page size x number.

Brice

Re: [hwloc-devel] memory size attributes

[Jeff Squyres (jsquyres)]

Just so I understand - are you saying hwloc should track both the total amount 
of memory *and* the makeup of that amount, broken up by page size?  So obj A 
may have x total memory, split across y 4k pages and z bigk hugepages (for 
example)?  And then the question becomes how to store this variable-page-sze 
information, right?

I'd say it can be valuable to support key=value pairs on any object so that 
future object types can be extensible (e.g., vendor pci devices). But common 
stuff should be accessible as struct members so that there's no string parsing 
needed (I'm no doubt just voicing what we all already think). I.e., esoteric 
stuff can start as a key=value strings but as they get mature / popular, they 
can "graduate" to become a struct member. 

As for how to store page counts as a function of page size, since we may not 
want to hard-code page sizes into fields, and I would prefer that they are not 
strings, how about an array of int[2]'s (page size and count)?  Or an array of 
structs (with fields of page size and count)?

-jms
Sent from my PDA.  No type good.

- Original Message -
From: hwloc-devel-boun...@open-mpi.org <hwloc-devel-boun...@open-mpi.org>
To: Hardware locality development list <hwloc-de...@open-mpi.org>
Sent: Sat Jan 16 07:08:46 2010
Subject: Re: [hwloc-devel] memory size attributes

Brice Goglin wrote:
> Hello,
>
> While cleaning the System/Machine root types, I wondered what we
> actually want to store in memory_kB attributes. It looks obvious for
> Caches and NUMA nodes. But I am not sure about Machines and Systems.
>
> If we have a machine with 2 NUMA nodes, should the machine memory size
> be the sum the sizes of both NUMA node sizes? Or should it be 0 since
> the machine has no memory except in NUMA nodes? Same question for a
> Kerrighed system assembling 2 machines.
>
> Then, if we have 1 Misc object grouping some NUMA nodes that are close
> to each other: Should we store the total memory size of these nodes in
> the Misc object attribute as well? We have the total memory size in the
> NUMA node object (below misc) and in the machine object (above misc),
> why not in Misc too? I am not saying that I want it, I am saying that
> it's not very consistent.
>
> So I wonder if we should just not sum anymore and let the application do
> the math when it actually needs the sum. A quick helper with
> get_next_obj_by_type( ... NODE) would work.
>
> Or we need to document memory size attributes better:
> * NUMA node: set of memory that can be accessed with the same access
> time from other objects
> * machine: set of cache-coherent memory, can be made of multiple NUMA nodes
> * system: set of memory that is virtually accessible, but may not be
> cache-coherent
>   

Aside from the memory_kB attribute, I wonder what should be done with
hugepages. I don't think we need to accumulate these at the system level
since multiple machines could well have different hugepage sizes.

And even inside a single machine, it's been pointed out that
architectures support multiple hugepage sizes. So we might have to
support several of them at the same time in the future. It could stored
in the attributes as an array of (hugepage size, hugepage number) in
numa node attributes but I don't really like that.

One way to support future random attributes could be to have an array of
stringified attributes, like foo=bar, dmiboardinfo=bar, ... and
hugepage(2M)=1024. Applications would have to parse them, but it's much
easier for us in the end.

And if we go this way, aside from stringified hugepage stuff, memory
attributes of node/machine/system would only be the unsigned long
memory_kB field. So we could even put memory_kB back into the main
hwloc_obj structure. Only cache would still have specific attributes
(its depth and maybe data/instruction/unified flag).

Brice

___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel

Re: [hwloc-devel] memory size attributes

[Jeff Squyres (jsquyres)]

Perhaps we should distinguish between memory in this object and memory that is 
accessible by this object...?

I.e., cache and numa node can have x local memory. System/machine may have 0 
local memory but (sum of children) accessible memory. 

Specifically:

1 I think it is a common enough action to want to know how much memory there is 
on a machine/system that hwloc should do the sum. 

2 maybe local-kb can be an attribute of relevant objects, but total-kb can be 
an attribute of machine/system/whatever (i.e., anything that can have children 
that have local-kb). This keeps the values and meanings separate. 


-jms
Sent from my PDA.  No type good.

- Original Message -
From: hwloc-devel-boun...@open-mpi.org 
To: hwloc development 
Sent: Wed Jan 13 08:40:49 2010
Subject: [hwloc-devel] memory size attributes

Hello,

While cleaning the System/Machine root types, I wondered what we
actually want to store in memory_kB attributes. It looks obvious for
Caches and NUMA nodes. But I am not sure about Machines and Systems.

If we have a machine with 2 NUMA nodes, should the machine memory size
be the sum the sizes of both NUMA node sizes? Or should it be 0 since
the machine has no memory except in NUMA nodes? Same question for a
Kerrighed system assembling 2 machines.

Then, if we have 1 Misc object grouping some NUMA nodes that are close
to each other: Should we store the total memory size of these nodes in
the Misc object attribute as well? We have the total memory size in the
NUMA node object (below misc) and in the machine object (above misc),
why not in Misc too? I am not saying that I want it, I am saying that
it's not very consistent.

So I wonder if we should just not sum anymore and let the application do
the math when it actually needs the sum. A quick helper with
get_next_obj_by_type( ... NODE) would work.

Or we need to document memory size attributes better:
* NUMA node: set of memory that can be accessed with the same access
time from other objects
* machine: set of cache-coherent memory, can be made of multiple NUMA nodes
* system: set of memory that is virtually accessible, but may not be
cache-coherent

Brice

___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel

Re: [hwloc-devel] memory size attributes

[Brice Goglin]

Brice Goglin wrote:
> Hello,
>
> While cleaning the System/Machine root types, I wondered what we
> actually want to store in memory_kB attributes. It looks obvious for
> Caches and NUMA nodes. But I am not sure about Machines and Systems.
>
> If we have a machine with 2 NUMA nodes, should the machine memory size
> be the sum the sizes of both NUMA node sizes? Or should it be 0 since
> the machine has no memory except in NUMA nodes? Same question for a
> Kerrighed system assembling 2 machines.
>
> Then, if we have 1 Misc object grouping some NUMA nodes that are close
> to each other: Should we store the total memory size of these nodes in
> the Misc object attribute as well? We have the total memory size in the
> NUMA node object (below misc) and in the machine object (above misc),
> why not in Misc too? I am not saying that I want it, I am saying that
> it's not very consistent.
>
> So I wonder if we should just not sum anymore and let the application do
> the math when it actually needs the sum. A quick helper with
> get_next_obj_by_type( ... NODE) would work.
>
> Or we need to document memory size attributes better:
> * NUMA node: set of memory that can be accessed with the same access
> time from other objects
> * machine: set of cache-coherent memory, can be made of multiple NUMA nodes
> * system: set of memory that is virtually accessible, but may not be
> cache-coherent
>   

Aside from the memory_kB attribute, I wonder what should be done with
hugepages. I don't think we need to accumulate these at the system level
since multiple machines could well have different hugepage sizes.

And even inside a single machine, it's been pointed out that
architectures support multiple hugepage sizes. So we might have to
support several of them at the same time in the future. It could stored
in the attributes as an array of (hugepage size, hugepage number) in
numa node attributes but I don't really like that.

One way to support future random attributes could be to have an array of
stringified attributes, like foo=bar, dmiboardinfo=bar, ... and
hugepage(2M)=1024. Applications would have to parse them, but it's much
easier for us in the end.

And if we go this way, aside from stringified hugepage stuff, memory
attributes of node/machine/system would only be the unsigned long
memory_kB field. So we could even put memory_kB back into the main
hwloc_obj structure. Only cache would still have specific attributes
(its depth and maybe data/instruction/unified flag).

Brice