subject:"Re\: \[Qemu\-devel\] \[RFC\] QEMU Dbus support \- a proposal management API"


Fabrice Bellard wrote:

Hi,

At this point I am not interested in integrating it into QEMU as it is 
one more API level to maintain in addition to the command line 
monitor. However, I can change my mind if several projects insists to 
have a similar interface.




I think that many projects now want to control qemu programatically.  
The monitor is not a good interface since it is text-based, hard to 
parse, and liable to change without notice when new features are added.  
However, I agree that having many similar constructs is not a good 
thing, and that we should retain the monitor for non-programmatic control.


What do you say to implementing the qemu interface as a plugin API, and 
implementing the monitor on top of this API? e.g.:


qemu loads /usr/local/lib/qemu/libmonitor.so, which uses the API to 
export the good old qemu monitor interface.  If it finds 
/usr/local/lib/qemu/libdbus.so, it loads an additional dbus interface.  
If libvirt wants to drop a libvirtapi.so into that directory, it can 
control qemu through that.


--
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Avi Kivity wrote:

Fabrice Bellard wrote:

Hi,

At this point I am not interested in integrating it into QEMU as it 
is one more API level to maintain in addition to the command line 
monitor. However, I can change my mind if several projects insists to 
have a similar interface.




I think that many projects now want to control qemu programatically.  
The monitor is not a good interface since it is text-based, hard to 
parse, and liable to change without notice when new features are 
added.  However, I agree that having many similar constructs is not a 
good thing, and that we should retain the monitor for non-programmatic 
control.


What do you say to implementing the qemu interface as a plugin API, 
and implementing the monitor on top of this API? e.g.:


qemu loads /usr/local/lib/qemu/libmonitor.so, which uses the API to 
export the good old qemu monitor interface.  If it finds 
/usr/local/lib/qemu/libdbus.so, it loads an additional dbus 
interface.  If libvirt wants to drop a libvirtapi.so into that 
directory, it can control qemu through that.


Why not just improve the monitor interface?  Half the internet is based 
on human-readable text protocols.


I don't think there would be a lot of objections to adding a status 
field with each monitor command.  It could even be done in a way that 
was backwards compatible.  For instance:


(qemu) info kqemu
kqemu support is not compiled in
(qemu) verbosity status
(qemu) info kqemu
-38: kqemu support is not compiled in

The other two things to add to the monitor are support for multiple 
simultaneous connections and some sort of select command.


Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Dor Laor wrote:

Laurent Vivier wrote:

Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a écrit :
  

Hi,



Hi,

  
At this point I am not interested in integrating it into QEMU as it is 
one more API level to maintain in addition to the command line monitor. 
However, I can change my mind if several projects insists to have a 
similar interface.



perhaps the DBUS interface can replace the command line monitor ?
We have just to move the command line interface to a client speaking to
qemu through the DBUS interface.

  
This is a valid option but the problem is that local user will have to 
use another tool (client) to
send commands. Another option is to have a common backend with machine 
 user interfaces.
For example, if we use dbus as the backend, monitor commands will just 
be translated into dbus.

The opposite option is also valid.

Anyway, the motivation behind a new interface is that the monitor 
interface is not good enough for automation:
There are not return status for commands, no option for async 
notifications, no option for parallel actions in case

a command takes long time to complete (like snapshot).


All of these are valid, and addressable.  Return statuses can just be 
added to the beginning of the output of each command (similar to how 
POP3 works).  Async notification can be made to work by add support to 
the monitor for a select command.  Semantically, select would block 
the monitor and then output events.   For this to work really well, you 
would have to support multiple simultaneous monitor sessions.  The 
parallel options for long running commands is already address in KVM 
with the migration command.  We just have to rework the snapshotting to 
be properly asynchronous.




So we either a new interface is added or the existing one will be 
enhanced.
Since Qemu/KVM will be used in production its highly important to have 
a reliable channel to connects with mgmt daemons.
Dbus is a common practice for communication and used in Linux, 
libvirt, etc. The question is whether to add a dbus server to Qemu or

a client is sufficient.


The main objection I have to dbus is that it's very heavy weight.  It 
implies a rather fat infrastructure and it not very suitable for 
embedding.  QEMU has very few dependencies and that is a strength ATM.  
People interested in embedding QEMU still want a good management 
interface so enhancing the monitor seems more preferable to me than 
adding a dbus dependency.


Regards,

Anthony Liguori


Regards,
Dor.

I guess Yuval will be very happy to make this work :-D

Regards,
laurent

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Mon, Dec 10, 2007 at 10:28:01AM +0200, Yuval Kashtan wrote:
 Hello All,
 Attached is a proposed patch which adds DBus support to QEMU. DBus is a
 standard message bus for linux (
 http://www.freedesktop.org/wiki/Software/dbus )
 The idea behind this is to allow for external programs such as controlling
 GUI or HyperVisors to communicate and control QEMU, in a standard manner.
 Although it is possible to connect to the monitor, an API is more standard,
 theres no need to parse texts and error messages, no internationalization
 problems and in general is a much better idea.
 
 Implementation:
 This is done by creating a DBus server from QEMU which raise signals and act
 upon methods.
 Methods mainly implement same logic as the current command line options and
 monitor does.
 
 How to use:
 To enable DBus support you'll have to add --enable-dbus when you execure
 configure script.
 Then when you execute QEMU, you'll have to add -dbus-service service_name
 where service_name follows the DBus naming convention ( xxx.xxx.xxx)
 To support QEMU configuration through DBus,  you'll also have to add
 -dbus_configuration which will cause QEMU to wait until ConfigDone is called
 through the DBus interface. this allows external programs time to call DBus
 methods starting with Config_ which changes configuration (just as the CLI
 does) thus enabling invocation of QEMU without parameters (beside the DBus
 ones).
 With that, we have implemented a nice python script that reads VM's
 configuration from INI file.

Frankly this is complete  utter overkill. QEMU needs a configuration file
for machines, but writing a configuration file in an external app and then 
sending that over DBus to QEMU is just insane, when QEMU can simply implement
a config file directly. Spawning QEMU then just becomes 'qemu -config foo.cfg'
instead of a mass of DBus calls. Having QEMU do the config file directly
is also portable to all operating systems and simpler to maintain, and easier
to deploy since it does not mandate a depedancy on a message  bus.


 Currently support was added to all important CLI configuration and some
 online monitor functions (such as connecting and disconnecting usb devices).
 
 configuration methods begins with Config_ prefix and monitor ones with
 Monitor_.
 For a complete list you can just execute qemu and connect with dbus-viewer
 to see what is supported.

The style of the DBus API does not make any sense  does not follow the common
recommended approach for dbus services. 

The idea of well-known bus names is that they are well-known. Passing in the
DBus well-known name as a --service command line arg is non-sensical because
then it ceases to be well-known since it can change at the whim of the app /
admin running QEMU. 

Simply exporting a single object having populating that with a flat set of
methods mapping 1-1 onto the monitor commands is a bad idea. THis has been
done before and is very much unliked  discouraged (eg the Skype dbus control 
API which maps DBus methods straight onto the SKype monitor). The idea with
DBus is to provide a tree of objects mapping to objects managed within the
application, and then provide contextually relevant methods. 

So for a virtual machine management application you might have a main manager
object (eg '/qemu/Manager') which provides a way to list virtual machines. Each
virtual machine might then be its own object (eg '/qemu/Machines/name'). From
a machine one might have methods to list devices, and add  remove devices. Each
device though would be an object '/qemu/Machines/name/Devices/device'. The
device objects would implement different interfaces depending on the type of
device being managed. eg, you might have a qemu.Device.Disk interface, a 
qemu.Device.USB interface, etc, etc, and each interface would have methods
relevant to that type of device.

Which dovetails into the next point - having a DBus API for a single VM is not
really much use. To be generally useful it needs to have a single DBus service
to control all virtual machines (ie QEMU instances). This implies that any DBus
service doesn't belong in the QEMU process itself.

You're now back to square one - the DBus control service needs to spawn QEMU
instances and talk to the QEMU monitor.

IMHO, the entire idea of bolting DBus into QEMU to provide a control API is
just fundamentally flawed.



Looking at what you're trying to address I see two core problems

 - Interacting with QEMU at runtime to change various configs
 - Configuring QEMU at startup

For the first, we currently have the monitor. As you noted this is hard to
interact with programatically since it involves parsing  formating strings.
The way to deal with this problem is to provide a 'libqemumonitor.so' which
exposes a formal C api for the various runtime control needs. Internally
this library would talk to the monitor. This ensures that all the string
parsing  formatting is isolated in a single source, trivially usable by any 
application.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 01:05:38PM +0200, Avi Kivity wrote:
 Fabrice Bellard wrote:
 Hi,
 
 At this point I am not interested in integrating it into QEMU as it is 
 one more API level to maintain in addition to the command line 
 monitor. However, I can change my mind if several projects insists to 
 have a similar interface.
 
 
 I think that many projects now want to control qemu programatically.  
 The monitor is not a good interface since it is text-based, hard to 
 parse, and liable to change without notice when new features are added.  
 However, I agree that having many similar constructs is not a good 
 thing, and that we should retain the monitor for non-programmatic control.
 
 What do you say to implementing the qemu interface as a plugin API, and 
 implementing the monitor on top of this API? e.g.:
 
 qemu loads /usr/local/lib/qemu/libmonitor.so, which uses the API to 
 export the good old qemu monitor interface.  If it finds 
 /usr/local/lib/qemu/libdbus.so, it loads an additional dbus interface.  
 If libvirt wants to drop a libvirtapi.so into that directory, it can 
 control qemu through that.

To be honest this is overkill. IMHO, there should simply be a client side
'libqemumonitor.so' which provides a formal C API for applications to use.
This C api would then talk to the QEMU monitor, thus isolating all the
string parsing  formatting in one place. Behind the scenes I could imagine
dropping in an alternative QEMU monitor impl which used an XDR serialization
format for the args  reply to avoid the string parsing/formatting, but this
is a minor detail, compared to the big picture, which is that applications
would benefit from a simply C api implementing the monitor commands. I'd
be interested in using such an API in libvirt, since I could remove the 
string parsing/formatting code I currently have. Having libvirt drop a
custom .so into the QEMU process just feels horribly wrong.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 08:51:32AM -0600, Anthony Liguori wrote:
 Dor Laor wrote:
 Laurent Vivier wrote:
 Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a écrit :
   
 Hi,
 
 
 Hi,
 
   
 At this point I am not interested in integrating it into QEMU as it is 
 one more API level to maintain in addition to the command line monitor. 
 However, I can change my mind if several projects insists to have a 
 similar interface.
 
 
 perhaps the DBUS interface can replace the command line monitor ?
 We have just to move the command line interface to a client speaking to
 qemu through the DBUS interface.
 
   
 This is a valid option but the problem is that local user will have to 
 use another tool (client) to
 send commands. Another option is to have a common backend with machine 
  user interfaces.
 For example, if we use dbus as the backend, monitor commands will just 
 be translated into dbus.
 The opposite option is also valid.
 
 Anyway, the motivation behind a new interface is that the monitor 
 interface is not good enough for automation:
 There are not return status for commands, no option for async 
 notifications, no option for parallel actions in case
 a command takes long time to complete (like snapshot).
 
 All of these are valid, and addressable.  Return statuses can just be 
 added to the beginning of the output of each command (similar to how 
 POP3 works).  Async notification can be made to work by add support to 
 the monitor for a select command.  Semantically, select would block 
 the monitor and then output events.   For this to work really well, you 
 would have to support multiple simultaneous monitor sessions.  The 
 parallel options for long running commands is already address in KVM 
 with the migration command.  We just have to rework the snapshotting to 
 be properly asynchronous.
 
 
 So we either a new interface is added or the existing one will be 
 enhanced.
 Since Qemu/KVM will be used in production its highly important to have 
 a reliable channel to connects with mgmt daemons.
 Dbus is a common practice for communication and used in Linux, 
 libvirt, etc. The question is whether to add a dbus server to Qemu or
 a client is sufficient.
 
 The main objection I have to dbus is that it's very heavy weight.  It 
 implies a rather fat infrastructure and it not very suitable for 
 embedding.  QEMU has very few dependencies and that is a strength ATM.  
 People interested in embedding QEMU still want a good management 
 interface so enhancing the monitor seems more preferable to me than 
 adding a dbus dependency.

It is also not so easily portable to other OS like Windows  Mac OS, who
will still be lacking a decent control API. As I mentioned in my other
thread, embedding a DBus in each individual QEMU process is not the right
way to write DBus services either. If you want DBus (which is questionable
in itself), then you want to have a single service which manages all a QEMU
VMs. This mandates that the DBus service be outside the context of the QEMU
process itself.


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 08:51:32AM -0600, Anthony Liguori wrote:
 Dor Laor wrote:
 Laurent Vivier wrote:
 Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a écrit :
   
 Hi,
 
 
 Hi,
 
   
 At this point I am not interested in integrating it into QEMU as it is 
 one more API level to maintain in addition to the command line monitor. 
 However, I can change my mind if several projects insists to have a 
 similar interface.
 
 
 perhaps the DBUS interface can replace the command line monitor ?
 We have just to move the command line interface to a client speaking to
 qemu through the DBUS interface.
 
   
 This is a valid option but the problem is that local user will have to 
 use another tool (client) to
 send commands. Another option is to have a common backend with machine 
  user interfaces.
 For example, if we use dbus as the backend, monitor commands will just 
 be translated into dbus.
 The opposite option is also valid.
 
 Anyway, the motivation behind a new interface is that the monitor 
 interface is not good enough for automation:
 There are not return status for commands, no option for async 
 notifications, no option for parallel actions in case
 a command takes long time to complete (like snapshot).
 
 All of these are valid, and addressable.  Return statuses can just be 
 added to the beginning of the output of each command (similar to how 
 POP3 works).  Async notification can be made to work by add support to 
 the monitor for a select command.  Semantically, select would block 
 the monitor and then output events.   For this to work really well, you 
 would have to support multiple simultaneous monitor sessions.  The 
 parallel options for long running commands is already address in KVM 
 with the migration command.  We just have to rework the snapshotting to 
 be properly asynchronous.

Or have 2 monitor interaction modes. One mode uses the command line style
suitable for people / scripting languages. The other umode ses a binary XDR
protocol for serializing the args  returns values for formal control
APIs to use in a easy manner. It ought to be reasonably straightforward to
add a binary serialization format for all existing commands

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Daniel P. Berrange wrote:

On Tue, Dec 11, 2007 at 08:51:32AM -0600, Anthony Liguori wrote:
  

Dor Laor wrote:


Laurent Vivier wrote:
  

Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a écrit :
 


Hi,
   
  

Hi,

 

At this point I am not interested in integrating it into QEMU as it is 
one more API level to maintain in addition to the command line monitor. 
However, I can change my mind if several projects insists to have a 
similar interface.
   
  

perhaps the DBUS interface can replace the command line monitor ?
We have just to move the command line interface to a client speaking to
qemu through the DBUS interface.

 

This is a valid option but the problem is that local user will have to 
use another tool (client) to
send commands. Another option is to have a common backend with machine 
 user interfaces.
For example, if we use dbus as the backend, monitor commands will just 
be translated into dbus.

The opposite option is also valid.

Anyway, the motivation behind a new interface is that the monitor 
interface is not good enough for automation:
There are not return status for commands, no option for async 
notifications, no option for parallel actions in case

a command takes long time to complete (like snapshot).
  
All of these are valid, and addressable.  Return statuses can just be 
added to the beginning of the output of each command (similar to how 
POP3 works).  Async notification can be made to work by add support to 
the monitor for a select command.  Semantically, select would block 
the monitor and then output events.   For this to work really well, you 
would have to support multiple simultaneous monitor sessions.  The 
parallel options for long running commands is already address in KVM 
with the migration command.  We just have to rework the snapshotting to 
be properly asynchronous.



Or have 2 monitor interaction modes. One mode uses the command line style
suitable for people / scripting languages. The other umode ses a binary XDR
protocol for serializing the args  returns values for formal control
APIs to use in a easy manner. It ought to be reasonably straightforward to
add a binary serialization format for all existing commands
  


I don't think binary is inherently easier to parse than text provided 
that some thought is put into the format of the textual output.


I think we just want to levels of verbosity.

Regards,

Anthony Liguori


Dan.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Yuval Kashtan

As I can see,
There is HUGH interest in management API for QEMU.
seemly, DBus is NOT the right solution for direct integration into QEMU as
it is not cross platform enough, pose extra dependency and (probably) not
suitable for embedded systems.

Keeping only the old monitor interface with no formal interface will make
QEMU harder to integrate with as it requires extra work

C API is not a complete solution as it does not define well enough how will
you integrate with QEMU without changing or adding to the source..
But combined with AVI's idea of some plug-in mechanism, which will use this
C API, it sounds like a complete and valid solution.

The monitor should be kept for backward compatibility, human interface and
be implemented on top of that API as well, so there will also be a reference
implementation of how to use the API and create a plug-in.




As of Daniel remarks regarding DBus:
- I actually have a newer version where commands are divided into different
DBus interfaces. So (hopefully) that solves the methods naming problem.
- I had to pass the server name at the CLI, since I was executing multiple
instances of QEMU.. Having an external server, raises again the question of
how will that server communicate with each instance of QEMU?
- It was easier for my project to configure QEMU after it started and not
through a file, that why there is a configuration method for each option. of
course there could also be a bulk method which receive them all or even read
them from a file (and actually with some extra work, many of them could be
made dynamics)

On Dec 11, 2007 5:00 PM, Daniel P. Berrange [EMAIL PROTECTED] wrote:

 On Tue, Dec 11, 2007 at 08:51:32AM -0600, Anthony Liguori wrote:
  Dor Laor wrote:
  Laurent Vivier wrote:
  Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a écrit :
  
  Hi,
  
  
  Hi,
  
  
  At this point I am not interested in integrating it into QEMU as it
 is
  one more API level to maintain in addition to the command line
 monitor.
  However, I can change my mind if several projects insists to have a
  similar interface.
  
  
  perhaps the DBUS interface can replace the command line monitor ?
  We have just to move the command line interface to a client speaking
 to
  qemu through the DBUS interface.
  
  
  This is a valid option but the problem is that local user will have to
  use another tool (client) to
  send commands. Another option is to have a common backend with machine
   user interfaces.
  For example, if we use dbus as the backend, monitor commands will just
  be translated into dbus.
  The opposite option is also valid.
  
  Anyway, the motivation behind a new interface is that the monitor
  interface is not good enough for automation:
  There are not return status for commands, no option for async
  notifications, no option for parallel actions in case
  a command takes long time to complete (like snapshot).
 
  All of these are valid, and addressable.  Return statuses can just be
  added to the beginning of the output of each command (similar to how
  POP3 works).  Async notification can be made to work by add support to
  the monitor for a select command.  Semantically, select would block
  the monitor and then output events.   For this to work really well, you
  would have to support multiple simultaneous monitor sessions.  The
  parallel options for long running commands is already address in KVM
  with the migration command.  We just have to rework the snapshotting to
  be properly asynchronous.
 
  
  So we either a new interface is added or the existing one will be
  enhanced.
  Since Qemu/KVM will be used in production its highly important to have
  a reliable channel to connects with mgmt daemons.
  Dbus is a common practice for communication and used in Linux,
  libvirt, etc. The question is whether to add a dbus server to Qemu or
  a client is sufficient.
 
  The main objection I have to dbus is that it's very heavy weight.  It
  implies a rather fat infrastructure and it not very suitable for
  embedding.  QEMU has very few dependencies and that is a strength ATM.
  People interested in embedding QEMU still want a good management
  interface so enhancing the monitor seems more preferable to me than
  adding a dbus dependency.

 It is also not so easily portable to other OS like Windows  Mac OS, who
 will still be lacking a decent control API. As I mentioned in my other
 thread, embedding a DBus in each individual QEMU process is not the right
 way to write DBus services either. If you want DBus (which is questionable
 in itself), then you want to have a single service which manages all a
 QEMU
 VMs. This mandates that the DBus service be outside the context of the
 QEMU
 process itself.


 Dan.
 --
 |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496
 -=|
 |=-   Perl modules: 
 http://search.cpan.org/~danberr/http://search.cpan.org/%7Edanberr/  
-=|
 |=-   Projects:

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 04:17:51PM +0100, Jean-Christian de Rivaz wrote:
 Anthony Liguori a écrit :
   The main objection I have to dbus is that it's very heavy weight.  It
 implies a rather fat infrastructure and it not very suitable for 
 embedding.  QEMU has very few dependencies and that is a strength ATM.  
 People interested in embedding QEMU still want a good management 
 interface so enhancing the monitor seems more preferable to me than 
 adding a dbus dependency.
 
 I must object again that.
 
 I use D-Bus on a very small processor (Analog Devices Blackfin BF-537) 
 and it work very well. The library is around 200k on the target: this is 
 very supportable for embedded device, especially when you take in 
 account all the simplifications it allow in the code of each 
 applications. The D-Bus library have very few and simple dependency.
 
 D-Bus is a growing standard for the communication between applications. 
 I expect that sooner or later, QEMU will be a D-Bus service.

A DBus service for controlling QEMU, does not imply that QEMU needs to use
DBus. It is perfectly possible to write DBus services for managing QEMU
outside of the QEMU process. I've written 2 myself - the first exposed a
DBus service for managing a collection of VMs, and communicated with QEMU
via its monitor. The second, simply exposed the libvirt API over DBus
which allows managing of QEMU, Xen, KVM, etc via the same interface.

Embedding DBus within QEMU is flawed because it neccessarily restricts you
to only managing a single VM, and does not provide you any management 
capabilities when QEMU is not running.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Jean-Christian de Rivaz


Anthony Liguori a écrit :
  The main objection I have to dbus is that it's very heavy weight.  It
implies a rather fat infrastructure and it not very suitable for 
embedding.  QEMU has very few dependencies and that is a strength ATM.  
People interested in embedding QEMU still want a good management 
interface so enhancing the monitor seems more preferable to me than 
adding a dbus dependency.


I must object again that.

I use D-Bus on a very small processor (Analog Devices Blackfin BF-537) 
and it work very well. The library is around 200k on the target: this is 
very supportable for embedded device, especially when you take in 
account all the simplifications it allow in the code of each 
applications. The D-Bus library have very few and simple dependency.


D-Bus is a growing standard for the communication between applications. 
I expect that sooner or later, QEMU will be a D-Bus service.


Best regards,
--
Jean-Christian de Rivaz

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 05:21:08PM +0200, Yuval Kashtan wrote:
 As I can see,
 There is HUGH interest in management API for QEMU.
 seemly, DBus is NOT the right solution for direct integration into QEMU as
 it is not cross platform enough, pose extra dependency and (probably) not
 suitable for embedded systems.
 
 Keeping only the old monitor interface with no formal interface will make
 QEMU harder to integrate with as it requires extra work

The monitor does have an initial barrier to entry, but that can be addressed
by providing a C API which sends  receives monitor commands. This does not
require any intrusive modification of QEMU, at most incremental enhancements
to make the monitor more complete.

 C API is not a complete solution as it does not define well enough how will
 you integrate with QEMU without changing or adding to the source..
 But combined with AVI's idea of some plug-in mechanism, which will use this
 C API, it sounds like a complete and valid solution.

You are now exposing the internals of QEMU as a stable ABI which has to be 
maintained indefinittely to avoid breaking these out-of-tree plugins. THis
does not sound like a winning solution since it'll dramatically restrict 
the scope of future development of QEMU code.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Yuval Kashtan wrote:

As I can see,
There is HUGH interest in management API for QEMU.
seemly, DBus is NOT the right solution for direct integration into 
QEMU as it is not cross platform enough, pose extra dependency and 
(probably) not suitable for embedded systems.


Keeping only the old monitor interface with no formal interface will 
make QEMU harder to integrate with as it requires extra work


Why can't we make the monitor interface a formal interface?

Regards,

Anthony Liguori

C API is not a complete solution as it does not define well enough how 
will you integrate with QEMU without changing or adding to the source..
But combined with AVI's idea of some plug-in mechanism, which will use 
this C API, it sounds like a complete and valid solution.


The monitor should be kept for backward compatibility, human interface 
and be implemented on top of that API as well, so there will also be a 
reference implementation of how to use the API and create a plug-in.





As of Daniel remarks regarding DBus:
- I actually have a newer version where commands are divided into 
different DBus interfaces. So (hopefully) that solves the methods 
naming problem.
- I had to pass the server name at the CLI, since I was executing 
multiple instances of QEMU.. Having an external server, raises again 
the question of how will that server communicate with each instance of 
QEMU?
- It was easier for my project to configure QEMU after it started and 
not through a file, that why there is a configuration method for each 
option. of course there could also be a bulk method which receive them 
all or even read them from a file (and actually with some extra work, 
many of them could be made dynamics)


On Dec 11, 2007 5:00 PM, Daniel P. Berrange [EMAIL PROTECTED] 
mailto:[EMAIL PROTECTED] wrote:


On Tue, Dec 11, 2007 at 08:51:32AM -0600, Anthony Liguori wrote:
 Dor Laor wrote:
 Laurent Vivier wrote:
 Le mardi 11 décembre 2007 à 10:10 +0100, Fabrice Bellard a
écrit :
 
 Hi,
 
 
 Hi,
 
 
 At this point I am not interested in integrating it into QEMU
as it is
 one more API level to maintain in addition to the command
line monitor.
 However, I can change my mind if several projects insists to
have a
 similar interface.
 
 
 perhaps the DBUS interface can replace the command line monitor ?
 We have just to move the command line interface to a client
speaking to
 qemu through the DBUS interface.
 
 
 This is a valid option but the problem is that local user will
have to
 use another tool (client) to
 send commands. Another option is to have a common backend with
machine
  user interfaces.
 For example, if we use dbus as the backend, monitor commands
will just
 be translated into dbus.
 The opposite option is also valid.
 
 Anyway, the motivation behind a new interface is that the monitor
 interface is not good enough for automation:
 There are not return status for commands, no option for async
 notifications, no option for parallel actions in case
 a command takes long time to complete (like snapshot).

 All of these are valid, and addressable.  Return statuses can
just be
 added to the beginning of the output of each command (similar to how
 POP3 works).  Async notification can be made to work by add
support to
 the monitor for a select command.  Semantically, select would
block
 the monitor and then output events.   For this to work really
well, you
 would have to support multiple simultaneous monitor sessions.  The
 parallel options for long running commands is already address in KVM
 with the migration command.  We just have to rework the
snapshotting to
 be properly asynchronous.

 
 So we either a new interface is added or the existing one will be
 enhanced.
 Since Qemu/KVM will be used in production its highly important
to have
 a reliable channel to connects with mgmt daemons.
 Dbus is a common practice for communication and used in Linux,
 libvirt, etc. The question is whether to add a dbus server to
Qemu or
 a client is sufficient.

 The main objection I have to dbus is that it's very heavy
weight.  It
 implies a rather fat infrastructure and it not very suitable for
 embedding.  QEMU has very few dependencies and that is a
strength ATM.
 People interested in embedding QEMU still want a good management
 interface so enhancing the monitor seems more preferable to me than
 adding a dbus dependency.

It is also not so easily portable to other OS like Windows  Mac
OS, who
will still be lacking a decent control API. As I mentioned in my other
thread, embedding a DBus in each individual QEMU process is not
the right
way to write DBus services either. If you want DBus (which is
questionable

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Richard W.M. Jones

Anthony Liguori wrote:

Daniel P. Berrange wrote:

Or have 2 monitor interaction modes. One mode uses the command line style
suitable for people / scripting languages. The other umode ses a
binary XDR

protocol for serializing the args returns values for formal control
APIs to use in a easy manner. It ought to be reasonably
straightforward to

add a binary serialization format for all existing commands

I don't think binary is inherently easier to parse than text provided
that some thought is put into the format of the textual output.

XDR (aka RFC 1014 RFC 4506) does let you generate complex interfaces
with relative ease. For example, here's the description of the remote
protocol used by libvirt:

http://git.et.redhat.com/?p=libvirt.git;a=blob;f=qemud/remote_protocol.x;h=d409c74387c2642651896136aba9bc1e2b62b621;hb=HEAD

Parsing is done for you by stubs that are generated from the above file.

On the downside it turns out that it's not very well supported under
Windows. For libvirt I had to basically port an XDR implementation by
hand to MinGW and add extra functions from glibc to it.

I think we just want to levels of verbosity.

This would work too.

On the point of controlling multiple qemu instances on a machine from a
single place: Easiest way to do this would be to direct all the monitor
sockets into a single known directory. Something along the lines of:

qemu -monitor unix:/var/lib/qemu-monitors/`uuidgen`,nowait

A control process can then just keep an eye on entries under that
directory, and (unlike libvirtd) it's robust against the control process
restarting.

Rich.

--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903

smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 09:16:43AM +0200, Yuval Kashtan wrote:
 - This is very useful when you want to manage and control QEMU, for instance
 developing a GUI to attach and detach usb devices or controlling more than
 one instance of QEMU from a single management point, receiving parameters
 externally and more.

This capability is already possible using the monitor API. I've implemented
it in libvirt, and in virt-manager and can easily hotplug/remove USB devices.
So DBus is not required to do this and does not really make it any easier
to use. DBus is very verbose to use from languages like C, so switching from
the monitor to DBus API would not really help simplify my existing code. On
the other hand, having a library providing a simple client side C API to 
send  receive monitor commands, would be beneficial

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

Richard W.M. Jones wrote:

Anthony Liguori wrote:

Daniel P. Berrange wrote:
Or have 2 monitor interaction modes. One mode uses the command line
style
suitable for people / scripting languages. The other umode ses a
binary XDR

protocol for serializing the args returns values for formal control
APIs to use in a easy manner. It ought to be reasonably
straightforward to

add a binary serialization format for all existing commands

I don't think binary is inherently easier to parse than text provided
that some thought is put into the format of the textual output.

XDR (aka RFC 1014 RFC 4506) does let you generate complex interfaces
with relative ease. For example, here's the description of the remote
protocol used by libvirt:

http://git.et.redhat.com/?p=libvirt.git;a=blob;f=qemud/remote_protocol.x;h=d409c74387c2642651896136aba9bc1e2b62b621;hb=HEAD

Parsing is done for you by stubs that are generated from the above
file.

On the downside it turns out that it's not very well supported under
Windows. For libvirt I had to basically port an XDR implementation by
hand to MinGW and add extra functions from glibc to it.

I think we just want to levels of verbosity.

This would work too.

On the point of controlling multiple qemu instances on a machine from
a single place: Easiest way to do this would be to direct all the
monitor sockets into a single known directory. Something along the
lines of:

qemu -monitor unix:/var/lib/qemu-monitors/`uuidgen`,nowait

A control process can then just keep an eye on entries under that
directory, and (unlike libvirtd) it's robust against the control
process restarting.

Actually, this was the original intention of the -name parameter. What
a management tool would want to do is:

1) if -name is specified by user, generate one with uuidgen
2) pass -name name and -pidfile /path/to/well/known/location/name.pid

This will ensure uniqueness of name without requiring the creation tool
to maintain any state (so no daemon is required). Right now, you would
also have to store a monitor socket in that well known path. However,
I'm working on a VNC tunnels patch right now that would allow the
monitor to be tunneled through a VNC session. The idea here is that a
management tool could just store a hint about the VNC location and then
you can get at the rest of the character devices through the VNC session.

Otherwise, you end up with a bunch of temporary sockets for things like
the monitor, serial devices, parallel devices, etc.

Regards,

Anthony Liguori

Rich.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Daniel P. Berrange wrote:


I think that many projects now want to control qemu programatically.  
The monitor is not a good interface since it is text-based, hard to 
parse, and liable to change without notice when new features are added.  
However, I agree that having many similar constructs is not a good 
thing, and that we should retain the monitor for non-programmatic control.


What do you say to implementing the qemu interface as a plugin API, and 
implementing the monitor on top of this API? e.g.:


qemu loads /usr/local/lib/qemu/libmonitor.so, which uses the API to 
export the good old qemu monitor interface.  If it finds 
/usr/local/lib/qemu/libdbus.so, it loads an additional dbus interface.  
If libvirt wants to drop a libvirtapi.so into that directory, it can 
control qemu through that.



To be honest this is overkill. IMHO, there should simply be a client side
'libqemumonitor.so' which provides a formal C API for applications to use.
  


libqemumonitor.so is an excellent idea. perhaps the libvirt code can be 
used as a base?


We should also provide bindings to the saner languages that management 
apps are typically written in.


--
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

On Tue, Dec 11, 2007 at 09:48:22AM -0600, Anthony Liguori wrote:
 Richard W.M. Jones wrote:
 
 Actually, this was the original intention of the -name parameter.  What 
 a management tool would want to do is:
 
 1) if -name is specified by user, generate one with uuidgen
 2) pass -name name and -pidfile /path/to/well/known/location/name.pid
 
 This will ensure uniqueness of name without requiring the creation tool 
 to maintain any state (so no daemon is required).  Right now, you would 
 also have to store a monitor socket in that well known path.  However, 
 I'm working on a VNC tunnels patch right now that would allow the 
 monitor to be tunneled through a VNC session.  The idea here is that a 
 management tool could just store a hint about the VNC location and then 
 you can get at the rest of the character devices through the VNC session.
 
 Otherwise, you end up with a bunch of temporary sockets for things like 
 the monitor, serial devices, parallel devices, etc.

Tunnelling things like the serial devices / parallel devices over VNC
would be very useful, as using a pty/file isn't useful for remote
management. Tunnelling the monitor over VNC is questionable though because
it has very significant security implications - if you can connect to the
VNC server, you essentially own the entire user account that the VM is
running under since the monitor can be used to map arbitrary files into
the guest VM. Now this may be useful in some cases, but for libvirt we
really just want the monitor accessible locally via a well-known UNIX
socket. Also, VNC ports are typically auto-allocated at startup so not
neccessary a good predictable access method.

The reason libvirt use the PTY  direct PID ownership was because we stared
off with QEMU 0.8.0. We should really just mandate use of QEMU 0.9.0 with
libvirt now since its been around quite a while. Then we could just run
with  -daemonize, -pidfile and -monitor unix:/well/known/path.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-   Perl modules: http://search.cpan.org/~danberr/  -=|
|=-   Projects: http://freshmeat.net/~danielpb/   -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Dor Laor

Anthony Liguori wrote:

Richard W.M. Jones wrote:
Anthony Liguori wrote:
Daniel P. Berrange wrote:
Or have 2 monitor interaction modes. One mode uses the command line
style
suitable for people / scripting languages. The other umode ses a
binary XDR
protocol for serializing the args returns values for formal control
APIs to use in a easy manner. It ought to be reasonably
straightforward to
add a binary serialization format for all existing commands

I don't think binary is inherently easier to parse than text provided
that some thought is put into the format of the textual output.

XDR (aka RFC 1014 RFC 4506) does let you generate complex interfaces
with relative ease. For example, here's the description of the remote
protocol used by libvirt:

http://git.et.redhat.com/?p=libvirt.git;a=blob;f=qemud/remote_protocol.x;h=d409c74387c2642651896136aba9bc1e2b62b621;hb=HEAD

Parsing is done for you by stubs that are generated from the above
file.

On the downside it turns out that it's not very well supported under
Windows. For libvirt I had to basically port an XDR implementation by
hand to MinGW and add extra functions from glibc to it.

I think we just want to levels of verbosity.

This would work too.

On the point of controlling multiple qemu instances on a machine from
a single place: Easiest way to do this would be to direct all the
monitor sockets into a single known directory. Something along the
lines of:

qemu -monitor unix:/var/lib/qemu-monitors/`uuidgen`,nowait

A control process can then just keep an eye on entries under that
directory, and (unlike libvirtd) it's robust against the control
process restarting.

Actually, this was the original intention of the -name parameter. What
a management tool would want to do is:

1) if -name is specified by user, generate one with uuidgen
2) pass -name name and -pidfile /path/to/well/known/location/name.pid

While this is good for home usage, it does not match product needs because
you don't always want to allow the user of the VM to control it. For
example,
the user might hack memory values from the monitor and comprise his
permission

within the guest.

Otherwise, you end up with a bunch of temporary sockets for things like
the monitor, serial devices, parallel devices, etc.

Actually this is what we do in Qumranet to run multiple VMs.
We just need to improve/replace the monitor. Maybe enhancing the current
implementation
as you suggested might be good enough, although library interface might
be cleaner (no ugly return code

for the user).
Thanks,
Dor.

Regards,

Anthony Liguori

Rich.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Anthony Liguori wrote:

Yuval Kashtan wrote:

As I can see,
There is HUGH interest in management API for QEMU.
seemly, DBus is NOT the right solution for direct integration into 
QEMU as it is not cross platform enough, pose extra dependency and 
(probably) not suitable for embedded systems.


Keeping only the old monitor interface with no formal interface 
will make QEMU harder to integrate with as it requires extra work


Why can't we make the monitor interface a formal interface?



Because then fixing a type or extending the interface becomes a pain.

It's also much more difficult to specify a text-base interface 
completey, compared to a C api (where sometimes all you need is the 
header and a few comments).



--
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Richard W.M. Jones


Avi Kivity wrote:
libqemumonitor.so is an excellent idea. perhaps the libvirt code can be 
used as a base?


We should also provide bindings to the saner languages that management 
apps are typically written in.


Libvirt has most of the major languages covered now.  The only language 
I'm aware of that we are missing (and people request) is PHP.


Rich.

--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom.  Registered in
England and Wales under Company Registration No. 03798903


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API

2007-12-11 Thread Fabrice Bellard


Daniel P. Berrange wrote:

On Tue, Dec 11, 2007 at 09:16:43AM +0200, Yuval Kashtan wrote:

- This is very useful when you want to manage and control QEMU, for instance
developing a GUI to attach and detach usb devices or controlling more than
one instance of QEMU from a single management point, receiving parameters
externally and more.


This capability is already possible using the monitor API. I've implemented
it in libvirt, and in virt-manager and can easily hotplug/remove USB devices.
So DBus is not required to do this and does not really make it any easier
to use. DBus is very verbose to use from languages like C, so switching from
the monitor to DBus API would not really help simplify my existing code. On
the other hand, having a library providing a simple client side C API to 
send  receive monitor commands, would be beneficial


For the short term I think it is better as you said to rely on libvirt 
which already has some support for QEMU.


As it was suggested, improving the monitor seems the easiest, maybe with 
an additionnal mode to provide a standardized way to get a return value 
for each command. Providing an example client code to directly use the 
monitor seems a good idea too.


Regarding the configuration, I can assure there will be soon a QEMU 
configuration file. The question is whether it will be included before 
the next release is out !


Regards,

Fabrice.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Paul Brook wrote:

Why can't we make the monitor interface a formal interface?
  

Because then fixing a type or extending the interface becomes a pain.

It's also much more difficult to specify a text-base interface
completey, compared to a C api (where sometimes all you need is the
header and a few comments).



I disagree.

It's entirely possible to fully specify a text protocol, 


Of course it's possible; it has been done many times.  I've printed out 
several pounds of http myself.  It isn't easy, especially starting from 
something intended for human consumption.


and it's just as easy 
to extend it in a backwards compatible way. 


It's hard to correct a typo in a backwards compatible way.  I think it's 
best to separate the human readable protocol from the machine readable 
protocol.


A C API is about the most fixed 
interface you could possible use. 


But it's easy to add things without breaking it.

A C API also requires that both sides of 
the interface be part of the same process on the same machine. 


My plan was for the library to connect to the managing process using a 
protocol of its choice.  But I'm withdrawing it in favor of Dan 
Berrange's idea of having a client library.


If you prefer a dedicated machine readable protocol, then I think that's 
workable too.  But it has to be decoupled from the human readable 
monitor protocol.


--
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API


Daniel P. Berrange wrote:

On Tue, Dec 11, 2007 at 09:48:22AM -0600, Anthony Liguori wrote:
  

Richard W.M. Jones wrote:

Actually, this was the original intention of the -name parameter.  What 
a management tool would want to do is:


1) if -name is specified by user, generate one with uuidgen
2) pass -name name and -pidfile /path/to/well/known/location/name.pid

This will ensure uniqueness of name without requiring the creation tool 
to maintain any state (so no daemon is required).  Right now, you would 
also have to store a monitor socket in that well known path.  However, 
I'm working on a VNC tunnels patch right now that would allow the 
monitor to be tunneled through a VNC session.  The idea here is that a 
management tool could just store a hint about the VNC location and then 
you can get at the rest of the character devices through the VNC session.


Otherwise, you end up with a bunch of temporary sockets for things like 
the monitor, serial devices, parallel devices, etc.



Tunnelling things like the serial devices / parallel devices over VNC
would be very useful, as using a pty/file isn't useful for remote
management. Tunnelling the monitor over VNC is questionable though because
it has very significant security implications - if you can connect to the
VNC server, you essentially own the entire user account that the VM is
running under since the monitor can be used to map arbitrary files into
the guest VM. Now this may be useful in some cases, but for libvirt we
really just want the monitor accessible locally via a well-known UNIX
socket. Also, VNC ports are typically auto-allocated at startup so not
neccessary a good predictable access method.
  


The monitor is already tunneled through VNC ala a 'vc'.  The tunnels 
extension I'm developing would just expose 'vc's as plain-text instead 
of rendered.  If your management tools policy is to not make the monitor 
exposed over a 'vc', then it wouldn't be exposed via VNC tunneling either.



The reason libvirt use the PTY  direct PID ownership was because we stared
off with QEMU 0.8.0. We should really just mandate use of QEMU 0.9.0 with
libvirt now since its been around quite a while. Then we could just run
with  -daemonize, -pidfile and -monitor unix:/well/known/path.
  


Yeah, it would be very nice to eliminate the extra daemon.

Regards,

Anthony Liguori


Dan.

Re: [Qemu-devel] [RFC] QEMU Dbus support - a proposal management API