I implemented most of this in the component branch. The changes in the core look heavy but it's not too bad in the end.
Now, each old backend registers a "component" that can be instantiated into a "backend" structure depending on the topology configuration. By default, you get your OS backend and the PCI backend. But you could also get a XML backend instead for instance. Each backend may contain a discover() callback (doing the main discovery, e.g. initial discovery through the OS) and a notify_new_object() callback to react when somebody else discovers something (currently only used by Linux to add OS devices when the PCI backend adds PCI devices). The list of components to create is defined by configure, depending on the OS, available libraries, and on plugin configuration. There are two types of plugins. "core" (for the above "components/backends") and "xml" (for libxml2 and nolibxml implementations behind the XML backend). At configure time, we can ask to build some components as plugins. Only "core-libpci" and "xml-libxml" for now (others do not depend on any external lib). Those components will be installed in $(libdir)/hwloc and loaded in topology_init(). Plugin are entirely disabled by default (even the core support is disabled) so it shouldn't break much. Testing with and without plugins didn't show any problem so far. Things to improve: 1) We load plugins and list existing components once per topology. We should do it only once per process. But that requires some locking in case multiple topologies are loaded simultaneously, which means we need thread-safety. Do we want pthread_mutex() for this? 2) Some internal functions are now exported to plugins. Do we want a special namespace? "hwloc__" instead of "hwloc_"? Is there anyway to make them visibile to plugins but not to applications? 3) I currently use the system libltdl. People usually ask libtool to copy it's libtldl implementation inside the project source at autogen to make sure it's compatible/working. So we won't have libltdl in SVN, we'll have it in tarballs, and we'll build it (with a sub-configure) if plugins are enabled. Is anybody against this? Brice Le 07/08/2012 11:02, Brice Goglin a écrit : > Antoine Rougier finished his internship recently so here's a summary of > what he did in the "backends" branch. For the record, the goal of his > work was to explore how to change our backends into proper plugins so > that we can easily avoid hard dependencies between the hwloc core and > external libraries such as CUDA, libpci, ... > > > He defined three types of backends: > * "Base" is a normal OS backend such as Linux/Solaris/... This guy > manages PU, Cores, Sockets, ... Only one of these can be used at the > same time. > * "Additional" is what is added to "Base" backends. This is libpci, and > things added inside PCI devices (CUDA, ...). Multiple of these may be > used at the same time. They are likely invoked sequentially after the > Base backend discovery. > * "Global" is a special case for XML, Synthetic and Custom, which > replaces both Base and Additional. It's not clear that this type will be > needed in the end, "Base" with a special flag to disable "Additional" > backends might be enough. > > During init(), all existing backends are placed on a list of "available" > backends. And a list of "enabled" backends is initialized to empty. > Between init() and load(), calling set_xml(), set_synthetic(), > set_fsroot(), ... will append some backends to the "enabled" list > (actually two lists, one for Base, one for Additional). > During load(), we check whether some Base backends are "enabled". > Otherwise we enable the default one of the current OS. Then we actually > discover things using the (unique) enabled Base backend followed by all > enabled Additional backends. > > > > Aside from the main "discover" callback, backends may also define some > callbacks to be invoked when new object are created. The main example is > Linux creating "OS devices" when a new "PCI device" is added by the PCI > backend. CUDA could use that too to fill GPU PCI devices. This is not > strictly needed since adding these devices could still be done later, > once the PCI backend is done. We'll see. > > I'll work on putting all this in a branch soon. > > > > All this is about making interaction between backends nicer. Once this > is done, we'll be able to make actual plugins out of all this. One > problem that will come is that some backends are almost directly used > from outside the core. For instance exporting a topology to XML is a > public API call going down to XML plugin. lstopo and hwloc-ps using > Linux-specific tid_get_cpubind() calls causes similar problems. > > Instead of allowing random API calls into plugin internals, we could > keep these backends internal, i.e. not making them plugins. At least for > OS backends, it makes sense. "synthetic" and "custom" also have no > reason to be pluginified either, they depend on nothing. > > XML would like to be a plugin so that we stop depending on libxml2, but > we have an intermediate level to ease this. The main XML functions do > not depend on libxml2, they can remain internal and call either libxml2 > or our minimalistic no-libxml2 support underneath. So we can keep the > common code and the minimalistic support internal, and only move the > libxml2-specific callbacks to a plugin. > > Summary of what plugins we could have in the end: > * for main backends: > + internal: synthetic, xml-core, custom, linux, solaris, ... > + plugins: pci, cuda, display, ... > * for "low-level" xml backends: > + internal: minimalistic xml support > + plugin: libxml2 > * and maybe lstopo backends one day > + internal: console, txt, fig, windows > + plugin: cairo > > Brice >
