Re: [hwloc-devel] memory size attributes
I pushed the last bunch of changes to the memoryattrs branch. I don't plan any other changes about this. I'll probably merge into trunk by the end of the week unless somebody complains ;) Brice
Re: [hwloc-devel] memory size attributes
On Jan 26, 2010, at 3:11 PM, Jeff Squyres (jsquyres) wrote: > It might still be nice to have a total_descendants_memory field (or a shorter > field) that represents the sum of all children memory since it is probably > not an uncommon action to just want to know how much memory I can access > *from this point in the hierarchy*. Of course, I didn't look at https://svn.open-mpi.org/trac/hwloc/browser/branches/memoryattrs/include/hwloc.h until *after* I replied to all of these -- and I see total_memory in there already. Perfect. :) -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
On Jan 26, 2010, at 10:09 AM, Brice Goglin wrote: > * should we enforce the ordering of pages size+count structures ? I am > sorting by page size for now Seems like a neighborly thing to do. If it's not hard, I say keep that functionality. > * how is the pages array terminated ? size = 0 ? or both size and count > = 0 ? if some OS fail to give the size of normal pages or huge pages, we > might have count !=0 while size = 0 in some cases. > * or should we add pages_count to the memory strcuture to explictly > store the length of the pages array ? If there's a question about what some OS's may do (e.g., report 0), then I'd be in favor of explicitly storing the pages_count. Who knows; someone may need to allocate some resources based on the length of that array (E.g., a GUI showing all the different page sizes); so if that length is available without the application needing to traverse the array just to count the length, that's another neighborly thing to do. (for the purposes of this email, "neighborly" = "nice to do and might be useful to some people, but not strictly required and I wouldn't fight if we didn't do it") -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
On Jan 19, 2010, at 3:15 PM, Brice Goglin wrote: > > I say "bytes" instead of "kilobytes" because it might be easier to avoid > > getting into the "what does kilo mean -- 1000 or 1024?" arguments. > > kilo mean 1000, otherwise it's called kibi :) Forgot to mention -- I think that comment proves the point right there. I would have never come up with "kibi" to mean "kilobytes". > But I am ok with changing all this into bytes or whatever. Perfect; I think that will be least ambiguous. > > Or maybe just total and local memory set to 0? > > Don't know. I thought having memory point to NULL when the object can > obviously contain no memory could help, but it's not a big deal. Ya, I'm still torn -- on one hand, saving a little memory is a good thing. On the other hand, it's nice to not have to put an "if" in just to calculate how much memory you have (i.e., if (ptr) size+=ptr->...). It might still be nice to have a total_descendants_memory field (or a shorter field) that represents the sum of all children memory since it is probably not an uncommon action to just want to know how much memory I can access *from this point in the hierarchy*. -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
On Jan 19, 2010, at 3:15 PM, Brice Goglin wrote: > IIRC, it was changed into a pointer before the first release to reduce > ABI changes inside the the main hwloc_obj struct when attributes are > changed. I don't think hwloc_obj and attributes will be stable before > hwloc 2.0 so I don't really care whether it's a pointer or not :) Ah -- preserving ABI goodness is a Good Thing. Keeping it a pointer won't guarantee us ABI compatibility, of course, but it helps. So disregard my prior comments -- keep it a pointer. Might as well give it a good college try to be ABI friendly. -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
Brice Goglin, le Tue 26 Jan 2010 16:09:13 +0100, a écrit : > = 0 ? if some OS fail to give the size of normal pages or huge pages, we > might have count !=0 while size = 0 in some cases. In that case the size wouldn't be advertised at all. count == 0 is however plausible when you run out of big pages :) Samuel
Re: [hwloc-devel] memory size attributes
I have pushed some changes in the memoryattrs branch. See the new memory attributes starting at line 138 of https://svn.open-mpi.org/trac/hwloc/browser/branches/memoryattrs/include/hwloc.h I am still working on the XML backend. It'd be good if people could complain about the API before I convert everything else. Random things to think about: * should we enforce the ordering of pages size+count structures ? I am sorting by page size for now * how is the pages array terminated ? size = 0 ? or both size and count = 0 ? if some OS fail to give the size of normal pages or huge pages, we might have count !=0 while size = 0 in some cases. * or should we add pages_count to the memory strcuture to explictly store the length of the pages array ? Brice
Re: [hwloc-devel] memory size attributes
Jeff Squyres wrote: > On Jan 19, 2010, at 3:20 AM, Brice Goglin wrote: > >> I propose the following changes to the object attributes. struct >> hwloc_obj now contains a struct hwloc_memory pointer: >> > > Is there any reason not to make this a sub-struct included in hwloc_obj (vs. > a pointer that requires an additional malloc)? I'm not saying that malloc > performance and/or the cost of dereferencing the pointer is a big deal -- it > just feels "un-tidy" to have another malloc for something that will be used > in many situations. > IIRC, it was changed into a pointer before the first release to reduce ABI changes inside the the main hwloc_obj struct when attributes are changed. I don't think hwloc_obj and attributes will be stable before hwloc 2.0 so I don't really care whether it's a pointer or not :) >> struct hwloc_memory { >> unsigned long total_kB; /* Total memory in this object and its children */ >> unsigned long local_kB; /* Local memory in this object */ >> > > FWIW, I'm not wild about the capitol "B" at the end, but that's mainly > because I'm lazy. ;-) How about total_mem and local_mem, and we specify > that they are in units of bytes? > > I say "bytes" instead of "kilobytes" because it might be easier to avoid > getting into the "what does kilo mean -- 1000 or 1024?" arguments. > kilo mean 1000, otherwise it's called kibi :) But I am ok with changing all this into bytes or whatever. > Also, do we want to make them uint64_t's? A system's total memory could get > pretty large -- more than 4B bytes on 32 bit systems...? Hmm. I'm > indecisive on this one -- I can think of reasons for and against... I guess > I just prefer future-proofing as much as possible. > I thought about that, but unsigned long was ok for 32bits system when storing values as kilobytes. If switching to bytes, we should use uint64_t. >> struct hwloc_memory_pages { >> unsigned long size; >> unsigned long count; >> } * local_pages; /* 0-terminated array of pages or hugepages sizes and >> counts */ >> } >> > > In the worst case, we've added 2*sizeof(long)+sizeof(struct*) to the size of > the hwloc_obj struct for objects that don't have memory. But I don't think > that's a big deal, right? > It's probably negligible against the size of cpusets since those are statically allocated so far. By the way, (I may have some students working on dynamic cpusets in the near future. > Or maybe just total and local memory set to 0? > Don't know. I thought having memory point to NULL when the object can obviously contain no memory could help, but it's not a big deal. > I'm not sure I understand what you mean here -- are you thinking of moving > the DMI fields to the general key=value arrays? What is "DMI"? > Yes. Try running dmidecode on a machine (as root) to see what DMI tells you, and see http://en.wikipedia.org/wiki/Desktop_Management_Interface for a bit of doc. It's basically a way to recognize the exact machine model and apply some hardware quirks. For instance hwloc could use that to find out the hypertransport topology since the BIOS doesn't tell you how opterons are connected (we did that in other software in the past). >> Actually, maybe obj->name should go there as well before it mixes to >> many meanings. It's mostly unused in trunk, except to store "Kerrighed" >> since very recently. We talked about using it to store the hostname of >> Machine objects. And the libpci branch uses it to store the PCI >> vendor/model string and OS device names. Maybe all these shouldn't be in >> obj->name and rather in the array of strings as OSName=Kerrighed, >> Hostname=foobar, PCIVendor=foo, PCIModel=bar, OSDevice=eth2, ... >> > > How about both? It can be useful to have a simple/common field to print a > human-readable text field (even if it's only a summary and the real details > are in key=value pairs). > Sure. Brice
Re: [hwloc-devel] memory size attributes
Jeff Squyres, le Tue 19 Jan 2010 07:03:55 -0500, a écrit : > On Jan 17, 2010, at 6:31 AM, Brice Goglin wrote: > > > Right now we have total amount + number of hugepages + size of > > hugepages. I was only talking about modifying the way to store all this > > so as it'd be to easier to add multiple hugepage size x number one day. > > I wasn't talking about adding normal page size x number. > > This may be a dumb question, but can there be multiple sized pages? Yes. Solaris even permits to choose various page sizes at least for different processes (through a system call), maybe even for the same process, I can't remember. Samuel
Re: [hwloc-devel] memory size attributes
On Jan 19, 2010, at 3:20 AM, Brice Goglin wrote: > I propose the following changes to the object attributes. struct > hwloc_obj now contains a struct hwloc_memory pointer: Is there any reason not to make this a sub-struct included in hwloc_obj (vs. a pointer that requires an additional malloc)? I'm not saying that malloc performance and/or the cost of dereferencing the pointer is a big deal -- it just feels "un-tidy" to have another malloc for something that will be used in many situations. > struct hwloc_memory { > unsigned long total_kB; /* Total memory in this object and its children */ > unsigned long local_kB; /* Local memory in this object */ FWIW, I'm not wild about the capitol "B" at the end, but that's mainly because I'm lazy. ;-) How about total_mem and local_mem, and we specify that they are in units of bytes? I say "bytes" instead of "kilobytes" because it might be easier to avoid getting into the "what does kilo mean -- 1000 or 1024?" arguments. Also, do we want to make them uint64_t's? A system's total memory could get pretty large -- more than 4B bytes on 32 bit systems...? Hmm. I'm indecisive on this one -- I can think of reasons for and against... I guess I just prefer future-proofing as much as possible. > struct hwloc_memory_pages { > unsigned long size; > unsigned long count; > } * local_pages; /* 0-terminated array of pages or hugepages sizes and > counts */ > } In the worst case, we've added 2*sizeof(long)+sizeof(struct*) to the size of the hwloc_obj struct for objects that don't have memory. But I don't think that's a big deal, right? > This obj->memory could be NULL except for Node, Machine and maybe > System. Or maybe just total and local memory set to 0? > > I'd say it can be valuable to support key=value pairs on any object so > > that future object types can be extensible (e.g., vendor pci devices). > > But common stuff should be accessible as struct members so that > > there's no string parsing needed (I'm no doubt just voicing what we > > all already think). I.e., esoteric stuff can start as a key=value > > strings but as they get mature / popular, they can "graduate" to > > become a struct member. > > If we do the above, Node and System won't have any specific attribute > anymore. Machine will still have DMI infos. But I am not sure they are > common enough and deserve an actual struct field, they could stay in the > array of stringified info. I'm not sure I understand what you mean here -- are you thinking of moving the DMI fields to the general key=value arrays? What is "DMI"? > Actually, maybe obj->name should go there as well before it mixes to > many meanings. It's mostly unused in trunk, except to store "Kerrighed" > since very recently. We talked about using it to store the hostname of > Machine objects. And the libpci branch uses it to store the PCI > vendor/model string and OS device names. Maybe all these shouldn't be in > obj->name and rather in the array of strings as OSName=Kerrighed, > Hostname=foobar, PCIVendor=foo, PCIModel=bar, OSDevice=eth2, ... How about both? It can be useful to have a simple/common field to print a human-readable text field (even if it's only a summary and the real details are in key=value pairs). -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
On Jan 17, 2010, at 6:31 AM, Brice Goglin wrote: > Right now we have total amount + number of hugepages + size of > hugepages. I was only talking about modifying the way to store all this > so as it'd be to easier to add multiple hugepage size x number one day. > I wasn't talking about adding normal page size x number. This may be a dumb question, but can there be multiple sized pages? E.g., can a system of multiple machines have page size X on some machines and page size Y on other machines? Additionally, I'm a little concerned about saying "hugepage" today, when that might be the normal tomorrow (kind of like calling something "new" -- someday, it won't be new anymore). I'd rather either specify the page size or not; not use an adjective that may or may not have meaning someday / on some platforms. -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] memory size attributes
I propose the following changes to the object attributes. struct hwloc_obj now contains a struct hwloc_memory pointer: struct hwloc_memory { unsigned long total_kB; /* Total memory in this object and its children */ unsigned long local_kB; /* Local memory in this object */ struct hwloc_memory_pages { unsigned long size; unsigned long count; } * local_pages; /* 0-terminated array of pages or hugepages sizes and counts */ } This obj->memory could be NULL except for Node, Machine and maybe System. Cache attributes are still specifically in obj->attr.cache.{size,depth} since they are different from the above semantics. > I'd say it can be valuable to support key=value pairs on any object so > that future object types can be extensible (e.g., vendor pci devices). > But common stuff should be accessible as struct members so that > there's no string parsing needed (I'm no doubt just voicing what we > all already think). I.e., esoteric stuff can start as a key=value > strings but as they get mature / popular, they can "graduate" to > become a struct member. > If we do the above, Node and System won't have any specific attribute anymore. Machine will still have DMI infos. But I am not sure they are common enough and deserve an actual struct field, they could stay in the array of stringified info. Actually, maybe obj->name should go there as well before it mixes to many meanings. It's mostly unused in trunk, except to store "Kerrighed" since very recently. We talked about using it to store the hostname of Machine objects. And the libpci branch uses it to store the PCI vendor/model string and OS device names. Maybe all these shouldn't be in obj->name and rather in the array of strings as OSName=Kerrighed, Hostname=foobar, PCIVendor=foo, PCIModel=bar, OSDevice=eth2, ... Brice
Re: [hwloc-devel] memory size attributes
Jeff Squyres (jsquyres) wrote: > > Just so I understand - are you saying hwloc should track both the > total amount of memory *and* the makeup of that amount, broken up by > page size? > I don't know if we need this. Right now we have total amount + number of hugepages + size of hugepages. I was only talking about modifying the way to store all this so as it'd be to easier to add multiple hugepage size x number one day. I wasn't talking about adding normal page size x number. Brice
Re: [hwloc-devel] memory size attributes
Just so I understand - are you saying hwloc should track both the total amount of memory *and* the makeup of that amount, broken up by page size? So obj A may have x total memory, split across y 4k pages and z bigk hugepages (for example)? And then the question becomes how to store this variable-page-sze information, right? I'd say it can be valuable to support key=value pairs on any object so that future object types can be extensible (e.g., vendor pci devices). But common stuff should be accessible as struct members so that there's no string parsing needed (I'm no doubt just voicing what we all already think). I.e., esoteric stuff can start as a key=value strings but as they get mature / popular, they can "graduate" to become a struct member. As for how to store page counts as a function of page size, since we may not want to hard-code page sizes into fields, and I would prefer that they are not strings, how about an array of int[2]'s (page size and count)? Or an array of structs (with fields of page size and count)? -jms Sent from my PDA. No type good. - Original Message - From: hwloc-devel-boun...@open-mpi.org To: Hardware locality development list Sent: Sat Jan 16 07:08:46 2010 Subject: Re: [hwloc-devel] memory size attributes Brice Goglin wrote: > Hello, > > While cleaning the System/Machine root types, I wondered what we > actually want to store in memory_kB attributes. It looks obvious for > Caches and NUMA nodes. But I am not sure about Machines and Systems. > > If we have a machine with 2 NUMA nodes, should the machine memory size > be the sum the sizes of both NUMA node sizes? Or should it be 0 since > the machine has no memory except in NUMA nodes? Same question for a > Kerrighed system assembling 2 machines. > > Then, if we have 1 Misc object grouping some NUMA nodes that are close > to each other: Should we store the total memory size of these nodes in > the Misc object attribute as well? We have the total memory size in the > NUMA node object (below misc) and in the machine object (above misc), > why not in Misc too? I am not saying that I want it, I am saying that > it's not very consistent. > > So I wonder if we should just not sum anymore and let the application do > the math when it actually needs the sum. A quick helper with > get_next_obj_by_type( ... NODE) would work. > > Or we need to document memory size attributes better: > * NUMA node: set of memory that can be accessed with the same access > time from other objects > * machine: set of cache-coherent memory, can be made of multiple NUMA nodes > * system: set of memory that is virtually accessible, but may not be > cache-coherent > Aside from the memory_kB attribute, I wonder what should be done with hugepages. I don't think we need to accumulate these at the system level since multiple machines could well have different hugepage sizes. And even inside a single machine, it's been pointed out that architectures support multiple hugepage sizes. So we might have to support several of them at the same time in the future. It could stored in the attributes as an array of (hugepage size, hugepage number) in numa node attributes but I don't really like that. One way to support future random attributes could be to have an array of stringified attributes, like foo=bar, dmiboardinfo=bar, ... and hugepage(2M)=1024. Applications would have to parse them, but it's much easier for us in the end. And if we go this way, aside from stringified hugepage stuff, memory attributes of node/machine/system would only be the unsigned long memory_kB field. So we could even put memory_kB back into the main hwloc_obj structure. Only cache would still have specific attributes (its depth and maybe data/instruction/unified flag). Brice ___ hwloc-devel mailing list hwloc-de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
Re: [hwloc-devel] memory size attributes
Perhaps we should distinguish between memory in this object and memory that is accessible by this object...? I.e., cache and numa node can have x local memory. System/machine may have 0 local memory but (sum of children) accessible memory. Specifically: 1 I think it is a common enough action to want to know how much memory there is on a machine/system that hwloc should do the sum. 2 maybe local-kb can be an attribute of relevant objects, but total-kb can be an attribute of machine/system/whatever (i.e., anything that can have children that have local-kb). This keeps the values and meanings separate. -jms Sent from my PDA. No type good. - Original Message - From: hwloc-devel-boun...@open-mpi.org To: hwloc development Sent: Wed Jan 13 08:40:49 2010 Subject: [hwloc-devel] memory size attributes Hello, While cleaning the System/Machine root types, I wondered what we actually want to store in memory_kB attributes. It looks obvious for Caches and NUMA nodes. But I am not sure about Machines and Systems. If we have a machine with 2 NUMA nodes, should the machine memory size be the sum the sizes of both NUMA node sizes? Or should it be 0 since the machine has no memory except in NUMA nodes? Same question for a Kerrighed system assembling 2 machines. Then, if we have 1 Misc object grouping some NUMA nodes that are close to each other: Should we store the total memory size of these nodes in the Misc object attribute as well? We have the total memory size in the NUMA node object (below misc) and in the machine object (above misc), why not in Misc too? I am not saying that I want it, I am saying that it's not very consistent. So I wonder if we should just not sum anymore and let the application do the math when it actually needs the sum. A quick helper with get_next_obj_by_type( ... NODE) would work. Or we need to document memory size attributes better: * NUMA node: set of memory that can be accessed with the same access time from other objects * machine: set of cache-coherent memory, can be made of multiple NUMA nodes * system: set of memory that is virtually accessible, but may not be cache-coherent Brice ___ hwloc-devel mailing list hwloc-de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
Re: [hwloc-devel] memory size attributes
Brice Goglin wrote: > Hello, > > While cleaning the System/Machine root types, I wondered what we > actually want to store in memory_kB attributes. It looks obvious for > Caches and NUMA nodes. But I am not sure about Machines and Systems. > > If we have a machine with 2 NUMA nodes, should the machine memory size > be the sum the sizes of both NUMA node sizes? Or should it be 0 since > the machine has no memory except in NUMA nodes? Same question for a > Kerrighed system assembling 2 machines. > > Then, if we have 1 Misc object grouping some NUMA nodes that are close > to each other: Should we store the total memory size of these nodes in > the Misc object attribute as well? We have the total memory size in the > NUMA node object (below misc) and in the machine object (above misc), > why not in Misc too? I am not saying that I want it, I am saying that > it's not very consistent. > > So I wonder if we should just not sum anymore and let the application do > the math when it actually needs the sum. A quick helper with > get_next_obj_by_type( ... NODE) would work. > > Or we need to document memory size attributes better: > * NUMA node: set of memory that can be accessed with the same access > time from other objects > * machine: set of cache-coherent memory, can be made of multiple NUMA nodes > * system: set of memory that is virtually accessible, but may not be > cache-coherent > Aside from the memory_kB attribute, I wonder what should be done with hugepages. I don't think we need to accumulate these at the system level since multiple machines could well have different hugepage sizes. And even inside a single machine, it's been pointed out that architectures support multiple hugepage sizes. So we might have to support several of them at the same time in the future. It could stored in the attributes as an array of (hugepage size, hugepage number) in numa node attributes but I don't really like that. One way to support future random attributes could be to have an array of stringified attributes, like foo=bar, dmiboardinfo=bar, ... and hugepage(2M)=1024. Applications would have to parse them, but it's much easier for us in the end. And if we go this way, aside from stringified hugepage stuff, memory attributes of node/machine/system would only be the unsigned long memory_kB field. So we could even put memory_kB back into the main hwloc_obj structure. Only cache would still have specific attributes (its depth and maybe data/instruction/unified flag). Brice
Re: [hwloc-devel] memory size attributes
Brice Goglin, le Wed 13 Jan 2010 14:40:49 +0100, a écrit : > * system: set of memory that is virtually accessible, but may not be > cache-coherent Mmm, I believe with kerrighed systems shared memory is cache-coherent (yes it's expensive). Samuel