Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 06/18/2013 01:15 PM, Daniel P. Berrange wrote: > By having all the private symbols change their version tag on every > release, we make it a total PITA for anyone to provide closed source, > out of tree, third party modules. Which is nice a feature :-) Using a symbol version file also means that symbols that aren't explicitly exported are not present in the dynamic symbol table. Which isn't *quite* as good as using compiler-level symbol visibility markup, but the linker can sometimes optimize code sequences in place or at minimum reduce a full symbol relocation to a relative relocation. > Symbol versioning like this isn't portable to Windows, but it works > on Solaris and Linux at least. I'm not sure about BSD support for > symbol versioning. Given that symbol versioning is mostly a forward compatible extension to ELF, assuming the linker doesn't date from the dark ages one should at least be able to take advantage of visibility improvements, even if actual symbol versions aren't enforced. r~
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 18 June 2013 21:19, Daniel P. Berrange wrote: > For example, use ELF symbol versioning > and make the version tags change on every single minor release Change on every git commit hash, please. Otherwise I bet there are going to be a lot of random weird failures in development due to things getting silently out of sync. -- PMM
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On Tue, Jun 18, 2013 at 07:42:11AM -0500, Anthony Liguori wrote: > Hi, > > On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev wrote: > > Hello. > > > > I looked at what's needed to support DSO (dynamic shared objects) > > in qemu, in order to be able to split functionality into loadable > > "plugins". It isn't exactly difficult, but a few steps are needed > > still. > > As it turns out, this is an area of interest of mine too. I put up a > quick page describing some ideas a few weeks ago. > > http://wiki.qemu.org/Features/Modules > > > The whole thing is about splitting the functionality into plugins > > from one, single, qemu build, for now, not about allowing 3rd-party > > plugins to be distributed outside of qemu. > > Ack. Really critical and important distinction. Agreed, if we introduce a module system to QEMU we should clearly and explicitly document that closed source out of tree modules are considered to be forbidden & a license violation. We should also make use of any technical measures available to make out of tree modules impratical to use. For example, use ELF symbol versioning and make the version tags change on every single minor release so you can't easily provide pre-compiled 3rd party modules. Obviously don't install header files anywhere in /usr/include, etc Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On Tue, Jun 18, 2013 at 02:17:08PM +0200, Laszlo Ersek wrote: > On 06/18/13 13:37, Michael Tokarev wrote: > > > Next, and this is the most complex part. The build system for > > modules, and configuring it. I heard there were plans to use > > something like kbuild system for that, has anything been done > > in this context? > > Sorry for responding separately... Some module/symbol versioning will be > necessary, as the binding will be delayed until runtime, and users > *will* mix and match qemu and modules, and get random crashes. AFAIK > versioning is a good way to avoid that (you would get an error in > dlopen(), and dlerror() would pinpoint problem). > > No idea how to implement this though, except I expect there's no chance > to do it halfway portably without libtool. In libvirt we use ELF symbol versioning for all symbols that are exported. Public symbols for applications to use get tagged with something like LIBVIRT_0.9.3 (according to the version the symbol arrived in), and then private symbols exported solely for use by loadable modules get tagged LIBVIRT_PRIVATE_1.0.3. The version number for the private tags is changed on every single release. This ensures that loadable modules can only be loaded by a libvirt that was built from the same version. # eu-readelf -a /usr/lib64/libvirt.so | grep LIBVIRT | awk '{print $8}' | less virPCIDeviceFileIterate@@LIBVIRT_PRIVATE_1.0.3 virDomainObjGetState@@LIBVIRT_PRIVATE_1.0.3 virXPathLongLong@@LIBVIRT_PRIVATE_1.0.3 virNetworkLookupByUUIDString@@LIBVIRT_0.2.0 sexpr_node_copy@@LIBVIRT_PRIVATE_1.0.3 virSecurityManagerPreFork@@LIBVIRT_PRIVATE_1.0.3 virDomainSaveStatus@@LIBVIRT_PRIVATE_1.0.3 virXPathULongLong@@LIBVIRT_PRIVATE_1.0.3 virDomainPinVcpu@@LIBVIRT_0.1.4 virCommandClearCaps@@LIBVIRT_PRIVATE_1.0.3 By having all the private symbols change their version tag on every release, we make it a total PITA for anyone to provide closed source, out of tree, third party modules. Which is nice a feature :-) Symbol versioning like this isn't portable to Windows, but it works on Solaris and Linux at least. I'm not sure about BSD support for symbol versioning. The other nice thing about symbol versioning (for RPM based distros) is that RPM will auto-add dependancies for symbol versions. So if you distribute loadable modules in a separate RPM from the main binary RPM, the dependancies will ensure you always have matched versions installed. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 06/18/2013 04:37 AM, Michael Tokarev wrote: > First of all, we need a global config switch/check for this kind of > service. On a POSIX system this should make the executable to be > linked with -rdynamic option, in order to make all symbols in the > executable to be available to plugins. The only way you're going to be able to make this work with windows dlls (and, frankly, a better way to make this work in elf without -export-dynamic), is to make the "main" part of qemu itself a shared library, so that the modules can link against it. This reduces the qemu executable to, more or less, _start +/- main. r~
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
Il 18/06/2013 15:12, Peter Maydell ha scritto: >>> >> o switching some modules to allow building them modular >> > >> > Pretty much any device is a candidate here. > Before you can do that you need to convert them so we can build > them once rather than once-per-target : target-specific versions > of modules are just going to be hopelessly confusing IMHO. Absolutely. The page that Anthony cited even says "Only devices that are compiled once can be modularized" (I don't remember if it was him or me who added it). Paolo
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
Il 18/06/2013 14:42, Anthony Liguori ha scritto: >> > Next, and this is the most complex part. The build system for >> > modules, and configuring it. I heard there were plans to use >> > something like kbuild system for that, has anything been done >> > in this context? > GSoC just kicked off yesterday. Paolo can perhaps shed some light on > what the plans are. The plans is just to add kconfig, not kbuild, similar to seabios/busybox/etc. It would provide configurability by target and board, and in addition make it obvious which bits of default-configs/ (such as PCI, USB, HPET, etc.) are actually configurable. I outlined a possible implementation at the end of http://wiki.qemu.org/Features/Modules, though something nicer may be possible (possibly taking inspiration from Kconfig). >> > With current config/build system, the following changes are >> > needed: >> > >> > o split individual libs from libs_softmmu into their own >> >variables. Ack. This could be doable already and independent from everything else. >> > o allow obj-m (and similar) in addition to obj-y, with build >> >flags and rules to produce an .so. > We also need a way to define the module targets. My Makefile-fu is > not all that great but I suspect we can do something like: > > libqemu-%.ko: $(eval $(obj-%-m)) > > I don't think that works but that's the rough idea. We would then > need to define each module target by hand but that's probably > reasonable in the short term until we have kconfig. You can limit that to multiple-source modules, which is the common case. Single-source modules are easily handled. Paolo
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 18 June 2013 13:42, Anthony Liguori wrote: > On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev wrote: >> o switching some modules to allow building them modular > > Pretty much any device is a candidate here. Before you can do that you need to convert them so we can build them once rather than once-per-target : target-specific versions of modules are just going to be hopelessly confusing IMHO. thanks -- PMM
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
Il 18/06/2013 14:05, Laszlo Ersek ha scritto: >> > +if (dlopen(path, RTLD_NOW) == NULL) > For inter-module sybmol resolution (motivated by any functional > dependencies, of course), RTLD_GLOBAL would be needed too (and then of > course order of loading would matter). I think we want to prohibit this specifically and use RTLD_LOCAL, at least in the beginning. If you have dependencies, just put them in a single module. As Michael mentioned, this is much simpler than the Linux kernel's module stuff. Paolo
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
Hi, On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev wrote: > Hello. > > I looked at what's needed to support DSO (dynamic shared objects) > in qemu, in order to be able to split functionality into loadable > "plugins". It isn't exactly difficult, but a few steps are needed > still. As it turns out, this is an area of interest of mine too. I put up a quick page describing some ideas a few weeks ago. http://wiki.qemu.org/Features/Modules > The whole thing is about splitting the functionality into plugins > from one, single, qemu build, for now, not about allowing 3rd-party > plugins to be distributed outside of qemu. Ack. Really critical and important distinction. > The plan is to keep the plugins in a single directory (or several > subdirectories, by type of plugins - for block, machine and other), > and load them during startup. Ack. > First of all, we need a global config switch/check for this kind of > service. On a POSIX system this should make the executable to be > linked with -rdynamic option, in order to make all symbols in the > executable to be available to plugins. > > This is an interesting case. We may end up in some symbols from > qemu libraries not linked in when building the executable, but > that symbol may be used in some plugin. Such situations, if will > happen, will need to be deal with on a case-by-case basis somehow. I wouldn't worry about this. There are a lot of things that can be made modules today that don't export any symbols at all. The other case to consider is things that expose QMP commands. We'll need to adjust how commands are registered such that the QMP commands are registered as part of modules. But that doesn't have to be in the initial implementation. > Next, the executable should be linked with -ldl (again, for a POSIX > system), to be able to dynamically load anything. This goes on the > same line as -rdynamic -- once we enable plugins, both linker options > are enabled. > > > Next, the module loading support. On POSIX it is and > dlopen(3), this one is trivial. https://developer.gnome.org/glib/stable/glib-Dynamic-Loading-of-Modules.html glib has a nice interface for modules that is cross platform. > I think the best is to hook up into util/module.c. Currently, > we explicitly call modue_call_init(type) for module types the > executable is interested with. So in that call we may also > load all plugins of the given type just before running all the > module init functions. > > For modules themselves, nothing changes -- __attribute__((constructor)) > does the same thing be it in a DSO or statically linked into > executable. We ought to start out by loading modules *before* doing any module init calls. That will keep things nice and simple. In the long term, we'll need to look at each subsystem and see what it takes to load modules after init. > Next, and this is the most complex part. The build system for > modules, and configuring it. I heard there were plans to use > something like kbuild system for that, has anything been done > in this context? GSoC just kicked off yesterday. Paolo can perhaps shed some light on what the plans are. > With current config/build system, the following changes are > needed: > > o split individual libs from libs_softmmu into their own >variables. > > o allow obj-m (and similar) in addition to obj-y, with build >flags and rules to produce an .so. We also need a way to define the module targets. My Makefile-fu is not all that great but I suspect we can do something like: libqemu-%.ko: $(eval $(obj-%-m)) I don't think that works but that's the rough idea. We would then need to define each module target by hand but that's probably reasonable in the short term until we have kconfig. > o ./configure --{enable,disable}-foo -- in addition to >that, I think --module-foo will be ok. If not, well, >any suggestion how to name it? You don't really need to touch configure to start with. We can just rely on people modifying the config*.mak file. We can wait for a proper menuconfig via kconfig. > > o modules installing > > o switching some modules to allow building them modular Pretty much any device is a candidate here. Regards, Anthony Liguori > o adding more types of modules, for example a new DISPLAY >type. > > > Below is a sample implementation of the loading part (without > configure checks, just POC). > > Thanks, > > /mjt > > --- a/util/module.c > +++ b/util/module.c > @@ -17,6 +17,62 @@ > #include "qemu/queue.h" > #include "qemu/module.h" > > +#define CONFIG_DSO_POSIX > + > +#ifdef CONFIG_DSO_POSIX > + > +#include > +#include > + > +static const char *plugins_dir = "plugins"; > + > +static const char *module_init_types[MODULE_INIT_MAX] = { > + "block", "machine", "qapi", "qom" > +}; > + > +static void load_modules(module_init_type type) > +{ > +DIR *dir; > +struct dirent *de; > +char path[PATH_MAX]; > + > +const char *typestr = module_init_type
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
18.06.2013 16:19, Michael Tokarev пишет: > 18.06.2013 16:17, Laszlo Ersek wrote: >> On 06/18/13 13:37, Michael Tokarev wrote: >> >>> Next, and this is the most complex part. The build system for >>> modules, and configuring it. I heard there were plans to use >>> something like kbuild system for that, has anything been done >>> in this context? >> >> Sorry for responding separately... Some module/symbol versioning will be >> necessary, as the binding will be delayed until runtime, and users >> *will* mix and match qemu and modules, and get random crashes. AFAIK >> versioning is a good way to avoid that (you would get an error in >> dlopen(), and dlerror() would pinpoint problem). >> >> No idea how to implement this though, except I expect there's no chance >> to do it halfway portably without libtool. > > Well, I think this is trivial really. > > Duding build, we re-#define register_module_init() to have a version > number (or even a build id), for example register_module_init_1_6_0(). > This is a function which each plugin will call inside its constructor. > Being unable to find this symbol in calling qemu it will fail to load. $ ./x86_64-softmmu/qemu-system-x86_64 -hda http://foo/bar warning: unable to load plugin plugins/block_curl.so: plugins/block_curl.so: undefined symbol: register_module_init_1_6_0 qemu-system-x86_64: -hda http://foo/bar: could not open disk image http://foo/bar: No such file or directory Looks clear enough. Also, we may add version number to the plugin name too. /mjt
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
18.06.2013 16:17, Laszlo Ersek wrote: > On 06/18/13 13:37, Michael Tokarev wrote: > >> Next, and this is the most complex part. The build system for >> modules, and configuring it. I heard there were plans to use >> something like kbuild system for that, has anything been done >> in this context? > > Sorry for responding separately... Some module/symbol versioning will be > necessary, as the binding will be delayed until runtime, and users > *will* mix and match qemu and modules, and get random crashes. AFAIK > versioning is a good way to avoid that (you would get an error in > dlopen(), and dlerror() would pinpoint problem). > > No idea how to implement this though, except I expect there's no chance > to do it halfway portably without libtool. Well, I think this is trivial really. Duding build, we re-#define register_module_init() to have a version number (or even a build id), for example register_module_init_1_6_0(). This is a function which each plugin will call inside its constructor. Being unable to find this symbol in calling qemu it will fail to load. Thanks, /mjt
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 06/18/13 13:37, Michael Tokarev wrote: > Next, and this is the most complex part. The build system for > modules, and configuring it. I heard there were plans to use > something like kbuild system for that, has anything been done > in this context? Sorry for responding separately... Some module/symbol versioning will be necessary, as the binding will be delayed until runtime, and users *will* mix and match qemu and modules, and get random crashes. AFAIK versioning is a good way to avoid that (you would get an error in dlopen(), and dlerror() would pinpoint problem). No idea how to implement this though, except I expect there's no chance to do it halfway portably without libtool. POSIX is very laconic regarding shared libraries, for example http://pubs.opengroup.org/onlinepubs/9699919799/utilities/c99.html in SUSv4 doesn't say anything about them. Laszlo
Re: [Qemu-devel] RFC: DSO (dynamic shared objects) support
On 06/18/13 13:37, Michael Tokarev wrote: > First of all, we need a global config switch/check for this kind of > service. On a POSIX system this should make the executable to be > linked with -rdynamic option, in order to make all symbols in the > executable to be available to plugins. > > This is an interesting case. We may end up in some symbols from > qemu libraries not linked in when building the executable, but > that symbol may be used in some plugin. Such situations, if will > happen, will need to be deal with on a case-by-case basis somehow. -rdynamic Pass the flag -export-dynamic to the ELF linker, on targets that support it. This instructs the linker to add all symbols, not only used ones, to the dynamic symbol table. This option is needed for some uses of "dlopen" or to allow obtaining backtraces from within a program. I think the problem you describe doesn't threaten therefore. OTOH link time and executable size could easily grow. > Next, the executable should be linked with -ldl (again, for a POSIX > system), to be able to dynamically load anything. This goes on the > same line as -rdynamic -- once we enable plugins, both linker options > are enabled. > > > Next, the module loading support. On POSIX it is and > dlopen(3), this one is trivial. dlopen(), yes, dlsym(), yes; type-casting the dlsym()-returned (void*) to the correct function pointer type, or the pointer-to-struct type, where the struct has a bunch of function pointers: not so trivial; some new type declarations would be necessary. > I think the best is to hook up into util/module.c. Currently, > we explicitly call modue_call_init(type) for module types the > executable is interested with. So in that call we may also > load all plugins of the given type just before running all the > module init functions. > > For modules themselves, nothing changes -- __attribute__((constructor)) > does the same thing be it in a DSO or statically linked into > executable. Ah. Basically, you wouldn't use dlsym() at all -- modules would register their stuff in their constructors, and the main executable would search by name? > +if (dlopen(path, RTLD_NOW) == NULL) For inter-module sybmol resolution (motivated by any functional dependencies, of course), RTLD_GLOBAL would be needed too (and then of course order of loading would matter). > +{ > +fprintf(stderr, "warning: unable to load plugin %s\n", path); dlerror() is standard, and it provides very good error messages on GNU/Linux. Laszlo