Re: [hwloc-devel] towards PLPA-like API in 1.0
On Dec 10, 2009, at 5:22 PM, Samuel Thibault wrote: > > Are you saying that the API will be all OS/physical, with conversion > > functions from #3 to convert to/from logical? > > No, it should stay logical, conversion would be just to translate into > OS/physical. Ok -- good. Consistent (CLI and C) == good. We'll need translation functions for both directions, right? (to and from OS/physical) > > Additionally, what exactly is the logical ordering defined to be? > > See topology.c, it's always topologically ordered, and then ordered > by OS cpu numbers. So for instance if the topology is flat, the OS > CPU numbers would get used. Then the topology distorts the OS CPU > numbering. So logical ordering is as close as OS CPU numbering as > possible, constrained by topology. Awesome. As you noted, I only looked in the linux file; not the top-level topology.c. Cool. > > We need to guarantee that it is the same across every run, > > It is, unless the BIOS changes the CPU numbers. Agreed: nothing we can do about that. > > My point: if we're going to have a logical ordering, we should be able to > > provide at least some level of guarantee of stability about that logical > > ordering. > > For sure since that was precisely what I had in my when I put the > sorting code in the generic part. 2 steps ahead of me. Perfect. :-) -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] towards PLPA-like API in 1.0
Brice Goglin wrote: > Jeff Squyres wrote: > >> FWIW, having a "simple" API like that might be a Good Thing...? >> >> I.e., just be able to bind to a specific thread/core/socket with a >> minimum fuss/muss. Even if such an API would be mainly syntactic sugar >> for other hwloc functionality -- there definitely is something to be >> said for "make the simple things simple". It will definitely (IMNSHO) >> extend hwloc's reach into a larger class of applications. Meaning: >> there are a variety of hard-coded apps out there that we'll never see; >> apps that run on specific servers for specific purposes, where the >> developers hard code in there "bind to cores 1-4" or "bind to sockets >> 1,3" because they already know the setup and this app is not intended >> to be portable. >> > > I am looking at what we could add to the main API/helpers, here's what > could be useful: > * get_obj_under_by_type(topology, type, index, subtype, subindex) > returns for instance core 2 under socket 3. It's very easy > (get_obj_by_type+get_obj_inside_cpuset_by_type). > * Some people might want _under_under with 3 types/indexes. Not sure we > want it, or want to make it generic with arrays of types/indexes... > * Generic conversion routines between os_index and logical_index, like > get_obj_by_os_index(type, os_index) and get_os_index_by_type(type, index) > * Some kind of processor flag which tells us whether a physical proc > exists and is online > Any opinion about this? Should we drop the current plpa.h and just add the above new inlines to helper.h? (with some documentation about switching from PLPA into these new functions) Brice
Re: [hwloc-devel] towards PLPA-like API in 1.0
Jeff Squyres wrote: >> Well, the list of good ideas will be very short then :) Most remaining >> functions are about manipulating core and socket ids, we don't need that >> at all in hwloc anymore. > > FWIW, having a "simple" API like that might be a Good Thing...? > > I.e., just be able to bind to a specific thread/core/socket with a > minimum fuss/muss. Even if such an API would be mainly syntactic sugar > for other hwloc functionality -- there definitely is something to be > said for "make the simple things simple". It will definitely (IMNSHO) > extend hwloc's reach into a larger class of applications. Meaning: > there are a variety of hard-coded apps out there that we'll never see; > apps that run on specific servers for specific purposes, where the > developers hard code in there "bind to cores 1-4" or "bind to sockets > 1,3" because they already know the setup and this app is not intended > to be portable. Oh sorry, I wasn't clear. I am not against a socket+core API [1]. What I find useless with hwloc is using the physical/OS ids. hwloc/plpa.h uses obj->os_index everywhere, while os_index is meaningless excep at the PROC level (when binding). If you want the second core in the second socket, you don't care about their physical/OS indexes (which are often non-consecutive, repeating, ...), you just want to use logical indexes. OS/physical indexes change all the time anyway (when upgrading the BIOS, the kernel, ...). > It may be useful to have the API be extensible; I don't see how you could be extensible without being close the generic hwloc API. > one thing I have heard rumors about coming in the not-distant future > is the concept of "boards" (multiple motherboards in a single box). > I.e., socket IDs may be repeated; they are differentiated by board > number. Actually, we've seen many strange things like this in the non-x86 world. It doesn't look very different from Bull's QBBs or SGI Blades in large Itanium machines :) Brice [1] But we'll have to choose between socket+core and NUMA+core (often similar but not always), and maybe other useful basic setups.
Re: [hwloc-devel] towards PLPA-like API in 1.0
Jeff Squyres, le Thu 12 Nov 2009 10:05:32 -0800, a écrit : > apps that run on specific servers for specific purposes, where the > developers hard code in there "bind to cores 1-4" or "bind to sockets > 1,3" because they already know the setup and this app is not intended > to be portable. But should these IDs be hwloc IDs or OS IDs? > It may be useful to have the API be extensible; one thing I have heard > rumors about coming in the not-distant future is the concept of > "boards" (multiple motherboards in a single box). I.e., socket IDs > may be repeated; they are differentiated by board number. In the case of hwloc IDs it's not a problem. Samuel
Re: [hwloc-devel] towards PLPA-like API in 1.0
On Nov 12, 2009, at 9:56 AM, Brice Goglin wrote: Well, the list of good ideas will be very short then :) Most remaining functions are about manipulating core and socket ids, we don't need that at all in hwloc anymore. My feeling is that converting an application from PLPA's core_id/socket_id API into the hwloc API will be non-trivial. So at least the current hwloc_plpa_* functions will be document a bit how to switch to the hwloc API. FWIW, having a "simple" API like that might be a Good Thing...? I.e., just be able to bind to a specific thread/core/socket with a minimum fuss/muss. Even if such an API would be mainly syntactic sugar for other hwloc functionality -- there definitely is something to be said for "make the simple things simple". It will definitely (IMNSHO) extend hwloc's reach into a larger class of applications. Meaning: there are a variety of hard-coded apps out there that we'll never see; apps that run on specific servers for specific purposes, where the developers hard code in there "bind to cores 1-4" or "bind to sockets 1,3" because they already know the setup and this app is not intended to be portable. It may be useful to have the API be extensible; one thing I have heard rumors about coming in the not-distant future is the concept of "boards" (multiple motherboards in a single box). I.e., socket IDs may be repeated; they are differentiated by board number. -- Jeff Squyres jsquy...@cisco.com
Re: [hwloc-devel] towards PLPA-like API in 1.0
Jeff Squyres wrote: > Sorry for the delay in replying to this; been caught up in SC09 prep... Next time, I'll commit immediately instead of waiting for a week in case somebody ever reads my emails :) Brice
Re: [hwloc-devel] towards PLPA-like API in 1.0
Jeff Squyres wrote: > On Nov 12, 2009, at 9:25 AM, Samuel Thibault wrote: > >> > > There's certainly some desirable PLPA API features that could be >> > > imported to the HWLOC API -- but I would think that if people >> want to >> > > keep using the PLPA API, they can. It just won't [ever] be updated. >> > > The existing (and future) hwloc API is the migration path forward -- >> > > I'm not convinced that providing a new API that's halfway between >> PLPA >> > > and hwloc is worthwhile... >> > >> > Agreed, let's just remove this and tell people to use >> hwloc_[sg]et_*cpubind. >> >> What do you mean by "this"? The whole plpa.h or just >> hwloc_plpa_sched_getaffinity? >> > > > My $0.02 / 0.01EUR: let's not try to emulate the PLPA API at all > (i.e., no hwloc_plpa_* functions). Let's just take any good ideas > that were there and incorporate them into the future of the hwloc API > as appropriate. Well, the list of good ideas will be very short then :) Most remaining functions are about manipulating core and socket ids, we don't need that at all in hwloc anymore. My feeling is that converting an application from PLPA's core_id/socket_id API into the hwloc API will be non-trivial. So at least the current hwloc_plpa_* functions will be document a bit how to switch to the hwloc API. Brice
Re: [hwloc-devel] towards PLPA-like API in 1.0
Brice Goglin, le Thu 12 Nov 2009 18:13:34 +0100, a écrit : > Jeff Squyres wrote: > >> * PLPA-like API is prefixed with hwloc_plpa_ and all functions get a new > >> hwloc_topology_t parameter. The problematic ones are: > >> > >> + int hwloc_plpa_sched_getaffinity(pid_t pid, hwloc_cpuset_t cpuset); > >> > > > > Hmm. I'm a little confused. If we don't provide a drop-in PLPA > > replacement API implementation, what's the point of implementing a > > PLPA-like API? PLPA users will still need to modify their code -- > > shouldn't we be pointing them to the more-powerful hwloc API instead? > > > > There's certainly some desirable PLPA API features that could be > > imported to the HWLOC API -- but I would think that if people want to > > keep using the PLPA API, they can. It just won't [ever] be updated. > > The existing (and future) hwloc API is the migration path forward -- > > I'm not convinced that providing a new API that's halfway between PLPA > > and hwloc is worthwhile... > > Agreed, let's just remove this and tell people to use hwloc_[sg]et_*cpubind. What do you mean by "this"? The whole plpa.h or just hwloc_plpa_sched_getaffinity? Samuel
Re: [hwloc-devel] towards PLPA-like API in 1.0
Jeff Squyres wrote: >> * PLPA-like API is prefixed with hwloc_plpa_ and all functions get a new >> hwloc_topology_t parameter. The problematic ones are: >> >> + int hwloc_plpa_sched_getaffinity(pid_t pid, hwloc_cpuset_t cpuset); >> > > Hmm. I'm a little confused. If we don't provide a drop-in PLPA > replacement API implementation, what's the point of implementing a > PLPA-like API? PLPA users will still need to modify their code -- > shouldn't we be pointing them to the more-powerful hwloc API instead? > > There's certainly some desirable PLPA API features that could be > imported to the HWLOC API -- but I would think that if people want to > keep using the PLPA API, they can. It just won't [ever] be updated. > The existing (and future) hwloc API is the migration path forward -- > I'm not convinced that providing a new API that's halfway between PLPA > and hwloc is worthwhile... Agreed, let's just remove this and tell people to use hwloc_[sg]et_*cpubind. Brice
Re: [hwloc-devel] towards PLPA-like API in 1.0
Sorry for the delay in replying to this; been caught up in SC09 prep... On Nov 5, 2009, at 8:22 AM, Brice Goglin wrote: * PLPA-like API is prefixed with hwloc_plpa_ and all functions get a new hwloc_topology_t parameter. The problematic ones are: + int hwloc_plpa_sched_getaffinity(pid_t pid, hwloc_cpuset_t cpuset); Hmm. I'm a little confused. If we don't provide a drop-in PLPA replacement API implementation, what's the point of implementing a PLPA-like API? PLPA users will still need to modify their code -- shouldn't we be pointing them to the more-powerful hwloc API instead? There's certainly some desirable PLPA API features that could be imported to the HWLOC API -- but I would think that if people want to keep using the PLPA API, they can. It just won't [ever] be updated. The existing (and future) hwloc API is the migration path forward -- I'm not convinced that providing a new API that's halfway between PLPA and hwloc is worthwhile... (I'm really sorry that I didn't reply about this earlier! :-( ) It's just a hwloc_get_cpubind(), but we don't have it since it would not be supported on all OS. But I think we should add it anyway. + int hwloc_plpa_get_core_flags(hwloc_topology_t topology, int socket_id, int core_id, int *exists, int *online); Is says whether a core (given by core+socket os_index) exists and is online. First, we don't have topology information about offline processors. Secondly, on Nehalem you can disable a single thread within a hyperthreaded core, so an "offline core" doesn't mean much. I would just vote for returning whether the core exists and remove the online return value here (see below for more about offline CPUs). Good point. PLPA was definitely not thought through well with regards to hardware threads. This is another reason not to expose this function in hwloc at all. * Then we have all count-spec related API, which lets you look for information about all processors, or all online ones, or all offline ones. If people are really interested with offline CPUs, they can look at the get_offline_cpuset below. There is no topology information about offline CPUs on Linux anyway, so I am not sure it's worth trying to manage offline and online CPUs in a uniform way. I would rather remove the count-spec argument and just only work on available/online/enabled processors with: + int hwloc_plpa_get_processor_data(hwloc_topology_t topology, int *num_processors, int *max_processor_id); + int hwloc_plpa_get_processor_id(int processor_num, int *processor_id); I think Samuel pointed out that some OS's *do* return info about offline CPUs (Solaris?). That would make exposing offline/unavailable/ otherwise-not-usage CPUs useful -- you can tell that they're *there*, even if you can't *use* them. If nothing else, it's an excellent diagnostic tool. * Probing >From what I understand, plpa_have_topology_information() tells whether PLPA knows what's in the hardware, while plpa_api_probe() tells whether binding is supported. We could add: + hwloc_topology_support(hwloc_topology_t topology, unsigned *support) which fills "support" with a bitmask of things like OS is supported, binding a thread is possible, binding a processor is possible, getting the binding of a process is possible, ... Then we could reimplement + int hwloc_plpa_have_topology_information(hwloc_topology_t topology); + int hwloc_plpa_api_probe(hwloc_topology_t topology); I think it would be better to have a capabilities vector like you describe later -- might as well unify all this stuff. * Finally, I plan to reimplement the PLPA tools, either in tests/plpa/ or as a real (installed) tools for a transition period. + plpa-info already works in my tree. Are there people that really need it? "lstopo -v -" basically shows the same and even more (offline CPUs are not reported in the trunk but I modified my tree to print the number of offline CPUs and the corresponding cpuset). I'm ok with not re-implementing plpa-info. That tool still exists and people can use it if they have scripts that depend on its specific output. We should be pushing people to the hwloc executables for all future work -- plpa-info output should be for legacy stuff only (IMHO). + plpa-taskset needs a lot of work for convering its own cpuset stuff into ours. It has an advanced binding syntax that some people may be used to. hwloc-bind has an advanced but different syntax. Apart from that, the features are the same. I think it *might* be worthwhile to convert some of the command line syntax to be supported by hwloc-bind. I (really, really, really) didn't like some of the syntax that was supported, but I stole a bunch of ideas from taskset(1) -- I was trying to make plpa-taskset be a drop-in replacement for taskset. Hence, I had to support its syntax. I'm ok jettisoning much of that now -- hwloc-bind is just much mo' betta than
Re: [hwloc-devel] towards PLPA-like API in 1.0
Brice Goglin, le Mon 09 Nov 2009 15:18:11 +0100, a écrit : > I don't think we need SET_CPUBIND since (from what I understand) it > would be equivalent to SET_PROC_CPUBIND | SET_THREAD_CPUBIND. Being able to set oneself's cpuset is not the same as being able to set the cpuset of other processes or other threads. We should also probably expose whether strict binding is available and whether whole processes and single threads can be bound. In a word, I believe we should just expose which exact parts of the binding API will never return ENOSYS. That would make it a lot easier to comprehend and much more straightforward to use it even though the interface is very powerful and can thus lead to a lot of binding possibilities. If an application has requirements A, B and C and would be even happier if D and E were available, it can prepare arguments for A, B and C, and if D and E are announced as being available, add arguments for them, and then call the function. That saves trying with D and E, or only with D, or only with E, and eventually fallback to not using either.. > We'd have to keep in mind that 32bits in this flag bitmask is small > (we'll likely need many other flags in the future, for instance > IODEVICE_DISCOVERY, SET/GET_MEMBIND, SET/GET_RANGE_MEMBIND). That's probably a hint that we should find another way :) How about a function that returns a structure filled with one long per feature (detection, thread bind, memory bind, etc.), part of hwloc_topology_t? That way we can extend it at will. Samuel
Re: [hwloc-devel] towards PLPA-like API in 1.0
Brice Goglin, le Thu 05 Nov 2009 17:22:15 +0100, a écrit : > + int hwloc_plpa_sched_getaffinity(pid_t pid, hwloc_cpuset_t cpuset); > > It's just a hwloc_get_cpubind(), but we don't have it since it would not > be supported on all OS. But I think we should add it anyway. Being discussed in another thread. > * Then we have all count-spec related API, which lets you look for > information about all processors, or all online ones, or all offline ones. > > If people are really interested with offline CPUs, they can look at the > get_offline_cpuset below. There is no topology information about offline > CPUs on Linux anyway, And at least on some other OSes as well, but not on Solaris for instance. > + hwloc_cpuset_t hwloc_topology_get_offline_cpuset(hwloc_topology_t topology); > > Returns a CPU set of existing CPUs that are offline, disabled by > administrator, or unavailable to this process if we're restricting the > topology to the process origin binding for instance. I am not sure we > actually need to distinguish all these cases. Mmm, I think there's one more thing that is actually more precise in some way: "the CPUs that we don't provide topology objects for": some OSes don't expose unauthorized CPUs even if they are online. It would thus also include CPUs which have explicitly been ignored because HWLOC_TOPOLOGY_FLAG_WHOLE_SYSTEM is not set. I think it can be useful to provide the cpuset of allowed cpus. One could imagine a tool that negociates with the administrator tools which cpus to be added/removed from the allowed set, knowing the whole topology and which CPUs are allowed would be useful. Maybe powering up/down could be involved too, thus the offline mask too. The current binding is already available from hwloc_get_cpubind. Actually, that gives me the idea that it'd be probably nice to somehow show that in the graphical lstopo :) Samuel
Re: [hwloc-devel] towards PLPA-like API in 1.0
Brice Goglin wrote:
> * Probing
>
> >From what I understand, plpa_have_topology_information() tells whether
> PLPA knows what's in the hardware, while plpa_api_probe() tells whether
> binding is supported. We could add:
>
> + hwloc_topology_support(hwloc_topology_t topology, unsigned *support)
>
/** \brief Flags describing the actual OS support for this topology.
*
* Flags are retrieved with hwloc_topology_get_support().
*/
enum hwloc_topology_support_flags_e {
/* \brief Topology discovery is supported. */
HWLOC_SUPPORT_DISCOVERY = (1<<0),
/* \brief Binding a process is supported. */
HWLOC_SUPPORT_SET_PROC_CPUBIND = (1<<1),
/* \brief Binding a thread is supported. */
HWLOC_SUPPORT_SET_THREAD_CPUBIND = (1<<2),
/* \brief Getting the binding of a process is supported. */
HWLOC_SUPPORT_GET_PROC_CPUBIND = (1<<3),
/* \brief Getting the binding of a thread is supported. */
HWLOC_SUPPORT_GET_THREAD_CPUBIND = (1<<4),
};
/** \brief Retrieve the OR'ed flags of topology support. */
extern int hwloc_topology_get_support(hwloc_topology_t __hwloc_restrict
topology, unsigned long flags);
I don't think we need SET_CPUBIND since (from what I understand) it
would be equivalent to SET_PROC_CPUBIND | SET_THREAD_CPUBIND. We'd have
to keep in mind that 32bits in this flag bitmask is small (we'll likely
need many other flags in the future, for instance IODEVICE_DISCOVERY,
SET/GET_MEMBIND, SET/GET_RANGE_MEMBIND).
Brice
[hwloc-devel] towards PLPA-like API in 1.0
Hello, I've been looking at the PLPA API and here's what we could do to add a hwloc/plpa.h offering kind of the same features. * PLPA-like API is prefixed with hwloc_plpa_ and all functions get a new hwloc_topology_t parameter. The problematic ones are: + int hwloc_plpa_sched_getaffinity(pid_t pid, hwloc_cpuset_t cpuset); It's just a hwloc_get_cpubind(), but we don't have it since it would not be supported on all OS. But I think we should add it anyway. + int hwloc_plpa_get_core_flags(hwloc_topology_t topology, int socket_id, int core_id, int *exists, int *online); Is says whether a core (given by core+socket os_index) exists and is online. First, we don't have topology information about offline processors. Secondly, on Nehalem you can disable a single thread within a hyperthreaded core, so an "offline core" doesn't mean much. I would just vote for returning whether the core exists and remove the online return value here (see below for more about offline CPUs). + int hwloc_plpa_map_to_processor_id(hwloc_topology_t topology, int socket_id, int core_id, int *processor_id); Return the os_index of the processor given by socket/core os_index. If we have hyperthreading, all thread sibling are valid here. I would just return the os_index of first one since this function was supposed to be used for binding anyway. * Then we have all count-spec related API, which lets you look for information about all processors, or all online ones, or all offline ones. If people are really interested with offline CPUs, they can look at the get_offline_cpuset below. There is no topology information about offline CPUs on Linux anyway, so I am not sure it's worth trying to manage offline and online CPUs in a uniform way. I would rather remove the count-spec argument and just only work on available/online/enabled processors with: + int hwloc_plpa_get_processor_data(hwloc_topology_t topology, int *num_processors, int *max_processor_id); + int hwloc_plpa_get_processor_id(int processor_num, int *processor_id); * Probing >From what I understand, plpa_have_topology_information() tells whether PLPA knows what's in the hardware, while plpa_api_probe() tells whether binding is supported. We could add: + hwloc_topology_support(hwloc_topology_t topology, unsigned *support) which fills "support" with a bitmask of things like OS is supported, binding a thread is possible, binding a processor is possible, getting the binding of a process is possible, ... Then we could reimplement + int hwloc_plpa_have_topology_information(hwloc_topology_t topology); + int hwloc_plpa_api_probe(hwloc_topology_t topology); * New hwloc general-purpose helpers that will be used by PLPA wrappers but may still be useful for other reasons: + hwloc_obj_t hwloc_get_parent_obj_by_type(hwloc_topology_t topology, hwloc_obj_type_t type, hwloc_obj_t obj); + hwloc_obj_t hwloc_get_parent_obj_by_depth(hwloc_topology_t topology, unsigned depth, hwloc_obj_t obj); Walks the parent chain until it finds the right type (or depth). + hwloc_obj_t hwloc_get_proc_obj_by_os_index(hwloc_topology_t topology, unsigned proc_os_index); Given a processor os_index, returns the corresponding object. + hwloc_cpuset_t hwloc_topology_get_offline_cpuset(hwloc_topology_t topology); Returns a CPU set of existing CPUs that are offline, disabled by administrator, or unavailable to this process if we're restricting the topology to the process origin binding for instance. I am not sure we actually need to distinguish all these cases. * Finally, I plan to reimplement the PLPA tools, either in tests/plpa/ or as a real (installed) tools for a transition period. + plpa-info already works in my tree. Are there people that really need it? "lstopo -v -" basically shows the same and even more (offline CPUs are not reported in the trunk but I modified my tree to print the number of offline CPUs and the corresponding cpuset). + plpa-taskset needs a lot of work for convering its own cpuset stuff into ours. It has an advanced binding syntax that some people may be used to. hwloc-bind has an advanced but different syntax. Apart from that, the features are the same. By the way, I wonder if we want to add public functions converting between cpusets (0x0f00) and cpulist string (0-15,24-27) (plpa-taskset uses something like this). Brice
