Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 04:53 AM, Anthony Liguori wrote: On 04/25/2010 06:51 AM, Avi Kivity wrote: It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon. I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present? Because libvirt has to perform this labelling because it loses the original user's security context. If you invoke qemu with the original user's credentials that launched the guest, then you don't need to do anything special with respect to security. IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place. What if one user has multiple guests? isolation is still needed. One user per guest does not satisfy some security requirements. The 'M' in selinux stands for mandatory, which means that the entities secured can't leak information even if they want to (scenario: G1 breaks into qemu, chmods files, G2 breaks into qemu, reads files). This is really the qemu model (as opposed to the xend model). (and the qemud model). And I've said in the past that I don't like the idea of a qemud :-) I must have missed it. Why not? Every other hypervisor has a central management entity. In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature. If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example. This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment). Because there are dozens of remote management APIs and then all have a concept of agents that run on the end nodes. When fitting virtualization management into an existing management infrastructure, you are going to always use a local API. When you manage esx, do you deploy an agent? I thought it was all done via their remote APIs. Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not? If that is the case then we should develop qemud (which libvirt and other apps can use). (even if it isn't the case I think qemud is a good idea) Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management). We do provide an API, qmp, and libvirt uses it? That's not what the libvirt community wants to do. We're very bias. We've made decisions about how features should be exposed and what features should be included. We want all of those features exposed exactly how we've implemented them because we think it's the right way. I'm not sure there's an obvious way forward unless we decide that there is going to be two ways to interact with qemu. One way is through the libvirt world-view and the other is through a more qemu centric view. The problem then becomes allowing those two models to co-exist happily together. I don't think there's a point in managing qemu through libvirt and directly in parallel. It means a user has to learn both APIs, and for every operation they need to check both to see what's the best way of exploiting the feature. There will invariably be some friction. Layers need to stack on top of each other, not live side by side or bypass each other. I agree with you theoretically but practically, I think it's immensely useful as a stop-gap. Sure. But please lets not start being clever with transactions and atomic operations and stuff, it has to come with a label that says, if you're using this, then something is wrong. The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that? For most hypervisors, that's exactly what libvirt does. For Xen, it also bypasses Xend and the hypervisor's API, but it shouldn't really. Historically, xend was so incredibly slow (especially for frequent statistics collection) that it was a necessity. Ah, reimplement rather than fix. Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist. sVirt is not just
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote: On 04/25/2010 06:51 AM, Avi Kivity wrote: Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist. sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it. The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface. If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling). That is not correct. You do *not* ever want the guests to have the same security context as the thing that created them, because that would allow the guest to access compromise resources belonging to the management app. Every guest must always have a unique context. The libvirt+sVirt dynamic labelling with unique contexts is applied even for the /session mode. The thing you save with the /session mode, is not having todo the user/group ownership management; sVirt labelling is always required. This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu. If you really want the qemu instance to inherit the context of the mgmt app, then you can just declare in the guest XML that it should use a static label, and pass in the apps' own label. This is *not* a more secure model though. Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 04:59 AM, Daniel P. Berrange wrote: On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote: On 04/25/2010 06:51 AM, Avi Kivity wrote: Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist. sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it. The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface. If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling). That is not correct. You do *not* ever want the guests to have the same security context as the thing that created them, because that would allow the guest to access compromise resources belonging to the management app. You assume that the management app is not smart enough to create a new context for the guest to run in. This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu. If you really want the qemu instance to inherit the context of the mgmt app, then you can just declare in the guest XML that it should use a static label, and pass in the apps' own label. This is *not* a more secure model though. There is more context than just selinux labelling. The problem with the daemon model is that to create a guest, you start with a lower set of privileges, escalate your privileges (by talking to libvirtd), then lower privileges to launch a guest. Running a guest is essentially running arbitrary code (since you can set the emulator path) so now you've provided an environment where a user can launch arbitrary code as a different user in a different security context. There is a new attack surface here. I think it's undeniable that there is certainly the possibility that something goes wrong and a user will find a way to escalate it's privileges. Compare that to a direct launch model. There is not new attack surface. The user's privileges never increase. In fact, what's most likely to happen is that a caller will drop some of it's privileges before launching a guest. Regards, Anthony Liguori Daniel
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/25/2010 09:50 AM, Avi Kivity wrote: On 04/23/2010 09:33 PM, Anthony Liguori wrote: This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log file, we can replay the JSON commands again and get the same results. Two monitor connections is just increasing complexity of code without any tangible benefit. I think you're assuming direct access to the second monitor? I'm not suggesting that. I'm suggesting that libvirt is still the one submitting commands to the second monitor and that it submits those commands in lock step. What about protocol extensions? For instance, pretend libvirt doesn't support async messages, what would it do when it receives one from the user's monitor? Protocol extensions could not be supported in this model. I agree, that's unfortunate. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 12:56 AM, Avi Kivity wrote: On 04/26/2010 04:53 AM, Anthony Liguori wrote: On 04/25/2010 06:51 AM, Avi Kivity wrote: It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon. I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present? Because libvirt has to perform this labelling because it loses the original user's security context. If you invoke qemu with the original user's credentials that launched the guest, then you don't need to do anything special with respect to security. IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place. What if one user has multiple guests? isolation is still needed. Don't confuse a management application's concept of users with using separate uid's to launch guests. One user per guest does not satisfy some security requirements. The 'M' in selinux stands for mandatory, which means that the entities secured can't leak information even if they want to (scenario: G1 breaks into qemu, chmods files, G2 breaks into qemu, reads files). If you're implementing a chinese firewall policy, then yes, you want to run each guest as a separate selinux context. Starting as separate users and setting DAC privileges appropriately will achieve this. But you're not always implementing that type of policy. If the guest inherits the uid, selinux context, and namespaces of whatever launches the guest, then you have the most flexibility from a security perspective. How do you launch a libvirt guest in a network namespace? How do you put it in a chroot? Today, you have to make changes to libvirt whereas in a direct launch model, you get all of the neat security features linux supports for free. And I've said in the past that I don't like the idea of a qemud :-) I must have missed it. Why not? Every other hypervisor has a central management entity. Because you end up launching all guests from a single security context. In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature. If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example. This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment). Because there are dozens of remote management APIs and then all have a concept of agents that run on the end nodes. When fitting virtualization management into an existing management infrastructure, you are going to always use a local API. When you manage esx, do you deploy an agent? I thought it was all done via their remote APIs. Historically, people have deployed agents into the console OS. In recent versions, ESX actually includes CIM agents by default. Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not? If that is the case then we should develop qemud (which libvirt and other apps can use). (even if it isn't the case I think qemud is a good idea) Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management). We do provide an API, qmp, and libvirt uses it? Yeah, but we need to support more features (like guest enumeration). The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that? For most hypervisors, that's exactly what libvirt does. For Xen, it also bypasses Xend and the hypervisor's API, but it shouldn't really. Historically, xend was so incredibly slow (especially for frequent statistics collection) that it was a necessity. Ah, reimplement rather than fix. There's a complicated history there. Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist. sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it. The reason it has to assign a context to a given
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Mon, Apr 26, 2010 at 08:13:03AM -0500, Anthony Liguori wrote: On 04/26/2010 04:59 AM, Daniel P. Berrange wrote: On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote: On 04/25/2010 06:51 AM, Avi Kivity wrote: Qemu is special due to the nonexistence of qemud. Why is sVirt implemented in libvirt? it's not the logical place for it; rather the logical place doesn't exist. sVirt is not just implemented in libvirt. libvirt implements a mechanism to set the context of a given domain and dynamically label it's resources to isolate it. The reason it has to assign a context to a given domain is that all domains are launched from the same security context (the libvirtd context) as the original user's context (the consumer of the libvirt API) has been lost via the domain socket interface. If you used the /session URL, then the domain would have the security context of whomever created the guest which means that dynamic labelling of the resources wouldn't be necessary (you would just do static labelling). That is not correct. You do *not* ever want the guests to have the same security context as the thing that created them, because that would allow the guest to access compromise resources belonging to the management app. You assume that the management app is not smart enough to create a new context for the guest to run in. This is certainly a more secure model and it's a feature of qemu that I really wish didn't get lost in libvirt. Again, /session can do this too but right now, /session really isn't usable in libvirt for qemu. If you really want the qemu instance to inherit the context of the mgmt app, then you can just declare in the guest XML that it should use a static label, and pass in the apps' own label. This is *not* a more secure model though. There is more context than just selinux labelling. The problem with the daemon model is that to create a guest, you start with a lower set of privileges, escalate your privileges (by talking to libvirtd), then lower privileges to launch a guest. Running a guest is essentially running arbitrary code (since you can set the emulator path) so now you've provided an environment where a user can launch arbitrary code as a different user in a different security context. No matter what emulator path is provided, that will never be run with elevated privileges unless the host sysadmin has explicitly configured that capability. The QEMU binaries always run in a dedicated account which is completely unprivileged. This setuid is done *before* QEMU is even exec()'d. This is one of the reasons why we don't use the QEMU -runas argument, which would allow custom emulator binaries to potentially gain evelated privileges. There is a new attack surface here. I think it's undeniable that there is certainly the possibility that something goes wrong and a user will find a way to escalate it's privileges. Of course all software has bugs. Compare that to a direct launch model. There is not new attack surface. The user's privileges never increase. In fact, what's most likely to happen is that a caller will drop some of it's privileges before launching a guest. What you describe is not inherant to the daemon model. This is why we have two separate models in libvirt. The system instance is pre-spawned with high privileges, to allow use of hosts resources which require high privileges to access. The session instance is auto-spawned when the app connects to libvirt, thus it inherits the privileges of the app that is using it. I don't deny that the system instance has a new attack surface, because it is running privileged. If the app needs to connect VMs to privileged resources, then the architecture has to have some privileged component to give access to those resoruces. You don't want the VM to be privileged, nor the whole management app to be privileged. The system daemon is thus the arbitrator for this privileged access. If you don't need todo anything that requires higher privileges though, just use the session instance which always matches the apps privileges. Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 04:14 PM, Anthony Liguori wrote: IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place. What if one user has multiple guests? isolation is still needed. Don't confuse a management application's concept of users with using separate uid's to launch guests. Then someone needs to manage those users. A user can't suid to any random user. You need someone privileged to allocate the new uid and su into it. One user per guest does not satisfy some security requirements. The 'M' in selinux stands for mandatory, which means that the entities secured can't leak information even if they want to (scenario: G1 breaks into qemu, chmods files, G2 breaks into qemu, reads files). If you're implementing a chinese firewall policy, then yes, you want to run each guest as a separate selinux context. Starting as separate users and setting DAC privileges appropriately will achieve this. But you're not always implementing that type of policy. If the guest inherits the uid, selinux context, and namespaces of whatever launches the guest, then you have the most flexibility from a security perspective. How do you launch a libvirt guest in a network namespace? How do you put it in a chroot? You pass the namespace fd and chroot fd using SCM_RIGHTS (except you probably can't do that). Today, you have to make changes to libvirt whereas in a direct launch model, you get all of the neat security features linux supports for free. But you lose tap networking, unless you have a privileged helper. And how is the privileged helper to authenticate the qemu calling it? And I've said in the past that I don't like the idea of a qemud :-) I must have missed it. Why not? Every other hypervisor has a central management entity. Because you end up launching all guests from a single security context. Run multiple qemuds? But what you say makes sense. It's similar to the fork() /* do interesting stuff */ exec() model, compared to the spawn(..., hardcoded list of interesting stuff). Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management). We do provide an API, qmp, and libvirt uses it? Yeah, but we need to support more features (like guest enumeration). What are our options? 1) qemud launches, enumerates 2) user launches, qemu registers in qemud 3) user launches, qemu registers in filesystem 4) you launched it, you enumerate it That's wrong for three reasons. First, selinux is not a uid replacement (if it was libvirt could just suid $random_user before launching qemu). Second, a single user's guests should be protected from each other. Third, in many deployments, the guest's owner isn't logged in to supply the credentials, it's system management that launches the guests. (1) uid's are just one part of an applications security context. There's an selinux context, all of the various namespaces, capabilities, etc. If you use a daemon to launch a guest, you lose all of that unless you have a very sophisticated api. True. In a perfect world, we'd use SCM_RIGHTS to channel all of these to libvirt or qemud. On the other hand, users don't want to do all these things by hand. They want management to do things for them. Self launch is very flexible, but it's not an API, and cannot be used remotely. We could use qemud plugins to allow the user to customize the launch process. (2) If you want to implement a policy that only a single guest can access a single image, you can create an SELinux policy and use static labelling to achieve that. That's just one type of policy though. It's also not going to work in an environment that doesn't preserve all security labels (like direct access to volumes; /dev is on tmpfs these days). (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 08:31 AM, Daniel P. Berrange wrote: What you describe is not inherant to the daemon model. This is why we have two separate models in libvirt. The system instance is pre-spawned with high privileges, to allow use of hosts resources which require high privileges to access. The session instance is auto-spawned when the app connects to libvirt, thus it inherits the privileges of the app that is using it. I don't deny that the system instance has a new attack surface, because it is running privileged. If the app needs to connect VMs to privileged resources, then the architecture has to have some privileged component to give access to those resoruces. You don't want the VM to be privileged, nor the whole management app to be privileged. The system daemon is thus the arbitrator for this privileged access. If you don't need todo anything that requires higher privileges though, just use the session instance which always matches the apps privileges. I regret saying more secure because I think it's a difficult concept to really quantify and that makes it hard to meaningfully discuss. The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. Regards, Anthony Liguori Daniel
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 08:41 AM, Avi Kivity wrote: Today, you have to make changes to libvirt whereas in a direct launch model, you get all of the neat security features linux supports for free. But you lose tap networking, unless you have a privileged helper. And how is the privileged helper to authenticate the qemu calling it? There are a variety of ways. My original proposal used a policy file. And I've said in the past that I don't like the idea of a qemud :-) I must have missed it. Why not? Every other hypervisor has a central management entity. Because you end up launching all guests from a single security context. Run multiple qemuds? But what you say makes sense. It's similar to the fork() /* do interesting stuff */ exec() model, compared to the spawn(..., hardcoded list of interesting stuff). Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management). We do provide an API, qmp, and libvirt uses it? Yeah, but we need to support more features (like guest enumeration). What are our options? 1) qemud launches, enumerates 2) user launches, qemu registers in qemud 3) user launches, qemu registers in filesystem 4) you launched it, you enumerate it Both 2 and 3 are appealing to me. (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. As I mentioned earlier, remoting is not a very important use-case to me. Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API. I suspect most uses of libvirt are actually local uses. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 04:46 PM, Anthony Liguori wrote: (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. As I mentioned earlier, remoting is not a very important use-case to me. Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API. Yes. I suspect most uses of libvirt are actually local uses. I expect the same, though I'm sure a design goal was to make use of libvirt be reasonable through the remote API. If we aren't able to fulfil it, much of the value of libvirt goes away. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote: On 04/26/2010 08:41 AM, Avi Kivity wrote: (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. As I mentioned earlier, remoting is not a very important use-case to me. NB, the remote protocol is no different from the local protocol the unprivileged app uses to talk to the privileged daemon. The only difference is unix sockets vs tcp sockets. We essentially get the remote access capability for free as part of the protocol used between the apps daemon. Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API. That is correct. That doesn't mean the remote interface isn't useful. I suspect most uses of libvirt are actually local uses. A key part of libvirt is interoperability between applications. So an app like vsdm can be using libvirt locally, while at the same time virt-top can be monitoring guests on a set of machines remotely, or virt-viewer can be run from your desktop talking to libvirt remotely to discover the VNC console details. For flexibility there are several options for remotely managing VMs. The native remote RPC system. The libvirt QPid/QMF model. The CIM binding. Application specific remote protocols. Since they all eventually talk to the same libvirtd daemon, their use is not mutually exclusive which is a good thing. Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 04:43 PM, Anthony Liguori wrote: The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. But what's the alternative? Teach the user how to do all these things? It's infinitely flexible, but it's not an API you can give to a management tool developer. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 09:01 AM, Avi Kivity wrote: On 04/26/2010 04:43 PM, Anthony Liguori wrote: The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. But what's the alternative? Teach the user how to do all these things? You can expose layers of API. The lowest layer makes no changes to the security context. A higher (optional) layer could do dynamic labelling. It's infinitely flexible, but it's not an API you can give to a management tool developer. I think the goal of a management API should be to make common things very simple to do but not preclude doing even the most advanced things. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 05:19 PM, Anthony Liguori wrote: On 04/26/2010 09:01 AM, Avi Kivity wrote: On 04/26/2010 04:43 PM, Anthony Liguori wrote: The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. But what's the alternative? Teach the user how to do all these things? You can expose layers of API. The lowest layer makes no changes to the security context. A higher (optional) layer could do dynamic labelling. Or a library that the user-written launcher calls. Or a plugin that qemud calls. It's infinitely flexible, but it's not an API you can give to a management tool developer. I think the goal of a management API should be to make common things very simple to do but not preclude doing even the most advanced things. Agreed. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 08:58 AM, Daniel P. Berrange wrote: On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote: On 04/26/2010 08:41 AM, Avi Kivity wrote: (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. As I mentioned earlier, remoting is not a very important use-case to me. NB, the remote protocol is no different from the local protocol the unprivileged app uses to talk to the privileged daemon. The only difference is unix sockets vs tcp sockets. We essentially get the remote access capability for free as part of the protocol used between the apps daemon. I think Avi's concern is that if you expose direct launch, then that's not really something you can usefully exploit in a remote protocol. For instance, if you're local, you can call setuid() before invoking the launch API whereas if you're remote, you will always launch guests as the context of the remoting daemon. This makes the remote API second-class in terms of functionality. Regards, Anthony Liguori Does RHEV-M actually use the remote libvirt interface? I assume it'll talk to vdsm via some protocol and vdsm will use the local libvirt API. That is correct. That doesn't mean the remote interface isn't useful. I suspect most uses of libvirt are actually local uses. A key part of libvirt is interoperability between applications. So an app like vsdm can be using libvirt locally, while at the same time virt-top can be monitoring guests on a set of machines remotely, or virt-viewer can be run from your desktop talking to libvirt remotely to discover the VNC console details. For flexibility there are several options for remotely managing VMs. The native remote RPC system. The libvirt QPid/QMF model. The CIM binding. Application specific remote protocols. Since they all eventually talk to the same libvirtd daemon, their use is not mutually exclusive which is a good thing. Daniel
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 09:25 AM, Avi Kivity wrote: On 04/26/2010 05:19 PM, Anthony Liguori wrote: On 04/26/2010 09:01 AM, Avi Kivity wrote: On 04/26/2010 04:43 PM, Anthony Liguori wrote: The reason I lean toward the direct launch model is that it gives the user a lot of flexibility in terms of using things like namespaces, DAC, cgroups, capabilities, etc. A lot of potential features are lost when you do indirect launch because you have to teach the daemon how to support each of these features. But what's the alternative? Teach the user how to do all these things? You can expose layers of API. The lowest layer makes no changes to the security context. A higher (optional) layer could do dynamic labelling. Or a library that the user-written launcher calls. Or a plugin that qemud calls. A plugin would lose the security context. It could attempt to recreate it that seems like a lot of unnecessary complexity. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Mon, Apr 26, 2010 at 09:26:55AM -0500, Anthony Liguori wrote: On 04/26/2010 08:58 AM, Daniel P. Berrange wrote: On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote: On 04/26/2010 08:41 AM, Avi Kivity wrote: (3) The system management application can certainly create whatever context it wants to launch a vm from. It's comes down to who's responsible for creating the context the guest runs under. I think doing that at the libvirt level takes away a ton of flexibility from the management application. If you want to push the flexibility slider all the way to the right you get bare qemu. It exposes 100% of qemu capabilities. And it's not so bad these days. But it's not something that can be remoted. As I mentioned earlier, remoting is not a very important use-case to me. NB, the remote protocol is no different from the local protocol the unprivileged app uses to talk to the privileged daemon. The only difference is unix sockets vs tcp sockets. We essentially get the remote access capability for free as part of the protocol used between the apps daemon. I think Avi's concern is that if you expose direct launch, then that's not really something you can usefully exploit in a remote protocol. For instance, if you're local, you can call setuid() before invoking the launch API whereas if you're remote, you will always launch guests as the context of the remoting daemon. This makes the remote API second-class in terms of functionality. It is not entirely outside the realms of possibility to consider making a way for libvirt to attach to a QEMU spawned by a 3rd party. eg, virsh qemu-attach /path/to/monitor PID Converting the QEMU config back into Libvirt XML is not trivial, but neither is it impossible. Particularly now we have the custom qemu:arg syntax for things libvirt can't represent, we can simply use that for bits we can't reverse map to XML. There's also the UUID/name uniqueness, but we could fail the attach operation if that wwas violated. This would let you use most of the other libvirt APIs remotely from any app, even if the VM was spawned by another one. Regards, Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 05:28 PM, Anthony Liguori wrote: Or a library that the user-written launcher calls. Or a plugin that qemud calls. A plugin would lose the security context. It could attempt to recreate it that seems like a lot of unnecessary complexity. A plugin would create the security context instead of the launcher. Currently security contexts are created by the login process. We could easily reuse that. Any other security context code would be custom written; so it can be written as a qemud plugin instead of a bit of code that goes before a qemu launch. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 09:38 AM, Avi Kivity wrote: On 04/26/2010 05:28 PM, Anthony Liguori wrote: Or a library that the user-written launcher calls. Or a plugin that qemud calls. A plugin would lose the security context. It could attempt to recreate it that seems like a lot of unnecessary complexity. A plugin would create the security context instead of the launcher. Currently security contexts are created by the login process. It's not always that centralized. An initial context is created by the login process, but then later something may come along and create a network namespace as part of containerization. We could easily reuse that. Any other security context code would be custom written; so it can be written as a qemud plugin instead of a bit of code that goes before a qemu launch. I think we're mostly in agreement with respect to the need to have more control over the security context the qemu runs in. Whether it's launched via a daemon or directly I think is an implementation detail that we can debate when we get closer to an actual implementation. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/26/2010 05:48 PM, Anthony Liguori wrote: We could easily reuse that. Any other security context code would be custom written; so it can be written as a qemud plugin instead of a bit of code that goes before a qemu launch. I think we're mostly in agreement with respect to the need to have more control over the security context the qemu runs in. Whether it's launched via a daemon or directly I think is an implementation detail that we can debate when we get closer to an actual implementation. Good, as I haven't decided yet which side I'm on. -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/25/2010 06:39 AM, Anthony Liguori wrote: On 04/24/2010 04:46 AM, Avi Kivity wrote: On 04/23/2010 09:29 PM, Anthony Liguori wrote: Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu. It's not orthogonal. If you allow qmp access behind libvirt's back, it's a problem that you will have. My point was, if libvirt is just exposing raw qemu features, then it should be possible for qemu to arbitrate concurrent access. If libvirt implements features on top of qemu, then no other third party will be able to co-exist with those features without interacting with qemu. It's an impossible problem for qemu to solve (arbitrating access to state stored in a third party management app). If libvirt implement features (like sVirt or network configuration) then it is indeed impossible for qemu to arbitrate. If we take all those features into qemu[d], then it becomes possible to arbitrate so long as the libvirt and the other management app don't step on each others toes. But that's impossible to guarantee if you upgrade your libvirt while keeping the other app unchanged. 1) Allow libvirt users to access features of qemu that are not exposed through libvirt That's an artificial problem. If libvirt exposes all features, you don't need to solve it. It won't. Otherwise, we wouldn't be having this discussion. Then libvirt will fade into uselessness. A successful app using libvirt will grow, and will have new requirements. As soon as libvirt doesn't meet those new requirements, the app will need to talk to qemu[d] directly. Once it does that, it may as well use qemu[d] for everything; if you can talk QMP and generate qemu command lines, there's not much that libvirt buys you. Even the cross-hypervisor support is not that hard to implement, especially if you only need to satisfy your own requirements. 2) Provide a means for non-libvirt users to interact with qemu We have qmp. It doesn't do multiple guest management. I think it's reasonable to have a qemud which does (and also does sVirt and the zillion other things libvirt does) provided we remove them from libvirt (long term). The only problem is that it's a lot of effort. It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon. I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present? This is really the qemu model (as opposed to the xend model). (and the qemud model). In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature. If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example. This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment). 3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library. What I'd like to avoid is user confusion. Should a user use libvirt or libqemu? If they make a decision to use libqemu and then down the road want to use libvirt, how hard is it to switch? Fragmentation hurts the ecosystem and discourages good applications from existing. I think it's our responsibility to ensure there's a good management API that exists for qemu that we can actively recommend to our users. libvirt is very good at typical virtualization uses of qemu but qemu is much more than just that and has lots of advanced features. Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not? If that is the case then we should develop qemud (which libvirt and other apps can use). (even if it isn't the case I think qemud is a good idea) For (1), we all agree that the best case scenario would be for libvirt to support every qemu
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 09:33 PM, Anthony Liguori wrote: This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log file, we can replay the JSON commands again and get the same results. Two monitor connections is just increasing complexity of code without any tangible benefit. I think you're assuming direct access to the second monitor? I'm not suggesting that. I'm suggesting that libvirt is still the one submitting commands to the second monitor and that it submits those commands in lock step. What about protocol extensions? For instance, pretend libvirt doesn't support async messages, what would it do when it receives one from the user's monitor? -- error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/25/2010 06:51 AM, Avi Kivity wrote: It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon. I don't follow. If you have multiple guests that you want off each other's turf you have to label their resources, either statically or dynamically. How is it related to a daemon being present? Because libvirt has to perform this labelling because it loses the original user's security context. If you invoke qemu with the original user's credentials that launched the guest, then you don't need to do anything special with respect to security. IOW, libvirt does not run guests as separate users which is why it needs to deal with security in the first place. This is really the qemu model (as opposed to the xend model). (and the qemud model). And I've said in the past that I don't like the idea of a qemud :-) In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature. If libvirt needs a local wrapper for interesting use cases, then it has failed. You can't have a local wrapper with the esx driver, for example. This is off-topic, but can you detail why you don't want remote dispatch (I assume we're talking about a multiple node deployment). Because there are dozens of remote management APIs and then all have a concept of agents that run on the end nodes. When fitting virtualization management into an existing management infrastructure, you are going to always use a local API. If you use a remote management API, you then have to worry about security, authorization, and authentication. Just supporting SASL really isn't enough because there are a lot fewer actual deployments of kerberos out there (most of them are actually Active Directory and SASL alone is not enough to interoperate with AD). 3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library. What I'd like to avoid is user confusion. Should a user use libvirt or libqemu? If they make a decision to use libqemu and then down the road want to use libvirt, how hard is it to switch? Fragmentation hurts the ecosystem and discourages good applications from existing. I think it's our responsibility to ensure there's a good management API that exists for qemu that we can actively recommend to our users. libvirt is very good at typical virtualization uses of qemu but qemu is much more than just that and has lots of advanced features. Every typical virtualization use will eventually grow some non-typical requirements. If libvirt explicitly refuses to support qemu features, I don't see how we can recommend it - even if it satisfies a user's requirements today, what about tomorrow? what about future qemu feature, will they be exposed or not? If that is the case then we should develop qemud (which libvirt and other apps can use). (even if it isn't the case I think qemud is a good idea) Yeah, that's where I'm at. I'd eventually like libvirt to use our provided API and I can see where it would add value to the stack (by doing things like storage and network management). That's not what the libvirt community wants to do. We're very bias. We've made decisions about how features should be exposed and what features should be included. We want all of those features exposed exactly how we've implemented them because we think it's the right way. I'm not sure there's an obvious way forward unless we decide that there is going to be two ways to interact with qemu. One way is through the libvirt world-view and the other is through a more qemu centric view. The problem then becomes allowing those two models to co-exist happily together. I don't think there's a point in managing qemu through libvirt and directly in parallel. It means a user has to learn both APIs, and for every operation they need to check both to see what's the best way of exploiting the feature. There will invariably be some friction. Layers need to stack on top of each other, not live side by side or bypass each other. I agree with you theoretically but practically, I think it's immensely useful as a stop-gap. The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that? For most hypervisors, that's exactly what libvirt
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 09:29 PM, Anthony Liguori wrote: Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu. It's not orthogonal. If you allow qmp access behind libvirt's back, it's a problem that you will have. Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt. Let's step back for a minute because I think we're missing the forest through the trees. We're trying to address a few distinct problems: 1) Allow libvirt users to access features of qemu that are not exposed through libvirt That's an artificial problem. If libvirt exposes all features, you don't need to solve it. 2) Provide a means for non-libvirt users to interact with qemu We have qmp. It doesn't do multiple guest management. I think it's reasonable to have a qemud which does (and also does sVirt and the zillion other things libvirt does) provided we remove them from libvirt (long term). The only problem is that it's a lot of effort. 3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library. For (1), we all agree that the best case scenario would be for libvirt to support every qemu feature. I think we can also all agree though that this is not really practical and certainly not practical for developers since there is a development cost associated with libvirt support (to model an API appropriately). All except me, perhaps. We already have two layers of feature modeling: first, we mostly emulate real life, not invent new features. PCI hotplug existed long before qemu had support for it. Second, we do give some thought into how we expose it through QMP. libvirt doesn't have to invent it again, it only has to expose it through its lovely xml and C APIs. The new API proposed addresses (1) by allowing a user to drill down to the QMP context. It's a good solution IMHO and I think we all agree that there's an inherent risk to this that users will have to evaluate on a case-by-case basis. It's a good stop-gap though. Agree. (2) is largely addressed by QMP and a config file. I'd like to see a nice C library, but I think a lot of other folks are happy with JSON support in higher level languages. I agree with them. C is a pretty bad choice for managing qemu (or even, C is a pretty bad choice). (3) is the place where there are still potential challenges. I think at the very least, our goal should be to enable conversion from (2) and (1) to be as easy as possible. That's why I have proposed implementing a C library for the JSON transport because we could plumb that through the new libvirt API. This would allow a user to very quickly port an application from QMP to libvirt. In order to do this, we need the libvirt API to expose a dedicated monitor because we'll need to be able to manipulate events and negotiate features. Most likely any application that talks QMP will hide the protocol behind a function call interface anyway. Beyond simple porting, there's a secondary question of having non-libvirt apps co-exist with libvirt apps. I think it's a good long term goal, but I don't think we should worry too much about it now. libvirt needs to either support all but the most esoteric use cases, or to get out of the way completely. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/24/2010 04:46 AM, Avi Kivity wrote: On 04/23/2010 09:29 PM, Anthony Liguori wrote: Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu. It's not orthogonal. If you allow qmp access behind libvirt's back, it's a problem that you will have. My point was, if libvirt is just exposing raw qemu features, then it should be possible for qemu to arbitrate concurrent access. If libvirt implements features on top of qemu, then no other third party will be able to co-exist with those features without interacting with qemu. It's an impossible problem for qemu to solve (arbitrating access to state stored in a third party management app). 1) Allow libvirt users to access features of qemu that are not exposed through libvirt That's an artificial problem. If libvirt exposes all features, you don't need to solve it. It won't. Otherwise, we wouldn't be having this discussion. 2) Provide a means for non-libvirt users to interact with qemu We have qmp. It doesn't do multiple guest management. I think it's reasonable to have a qemud which does (and also does sVirt and the zillion other things libvirt does) provided we remove them from libvirt (long term). The only problem is that it's a lot of effort. It depends on what things you think are important. A lot of libvirt's complexity is based on the fact that it uses a daemon and needs to deal with the security implications of that. You don't need explicit labelling if you don't use a daemon. This is really the qemu model (as opposed to the xend model). In theory, it does support this with the session urls but they are currently second-class citizens in libvirt. The remote dispatch also adds a fair bit of complexity and at least for the use-cases I'm interested in, it's not an important feature. 3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users This problem can be solved by the non-libvirt users adopting libvirt, or the libvirt users dropping libvirt. I don't understand why we need to add interoperability between users who choose an interoperability library and users who don't choose an interoperability library. What I'd like to avoid is user confusion. Should a user use libvirt or libqemu? If they make a decision to use libqemu and then down the road want to use libvirt, how hard is it to switch? Fragmentation hurts the ecosystem and discourages good applications from existing. I think it's our responsibility to ensure there's a good management API that exists for qemu that we can actively recommend to our users. libvirt is very good at typical virtualization uses of qemu but qemu is much more than just that and has lots of advanced features. For (1), we all agree that the best case scenario would be for libvirt to support every qemu feature. I think we can also all agree though that this is not really practical and certainly not practical for developers since there is a development cost associated with libvirt support (to model an API appropriately). All except me, perhaps. We already have two layers of feature modeling: first, we mostly emulate real life, not invent new features. PCI hotplug existed long before qemu had support for it. Second, we do give some thought into how we expose it through QMP. libvirt doesn't have to invent it again, it only has to expose it through its lovely xml and C APIs. That's not what the libvirt community wants to do. We're very bias. We've made decisions about how features should be exposed and what features should be included. We want all of those features exposed exactly how we've implemented them because we think it's the right way. I'm not sure there's an obvious way forward unless we decide that there is going to be two ways to interact with qemu. One way is through the libvirt world-view and the other is through a more qemu centric view. The problem then becomes allowing those two models to co-exist happily together. The alternative is to get libvirt to just act as a thin layer to expose qemu features directly. But honestly, what's the point of libvirt if they did that? For the most part, the pool management is not virtualization specific and you could have libvirt provide that functionality without it knowing a thing about qemu. The new API proposed addresses (1) by allowing a user to drill down to the QMP context. It's a good solution IMHO and I think we all agree that there's an inherent risk to this that users will have to evaluate on a case-by-case basis. It's a good stop-gap though. Agree. (2) is largely addressed by QMP and a config file. I'd like to see a nice C library, but I think a lot of other folks are happy with JSON support in higher
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote: On 04/09/2010 09:27 AM, Daniel P. Berrange wrote: On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote: domain type='kvm' namemyguest/name ... debug monitorpassthrough/ commandline extraqemu arguments/extra alter option=optname renamenewname/rename matchREGEXP/match modifyfoo=on/modify extra-bar/extra /alter /commandline /debug /domain The concept of command line monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example domain type='kvm' xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0; namemyguest/name ... qemu:commandline qemu:arg-device/arg qemu:arglsi/arg /qemu:commandline /domain I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file. Currently libvirt doesn't use any config file at all, so command line is the only possible option. We're going to evaluate switching to the config file at some point so can easily add further XML options to allow direct setting of config file entries at that point. There's no problem supporting both command line config file syntax in the XML. The command line does have the immediate advantage that 99% of all documentation about QEMU on the web today refers to command line args so that's what people are most familiar with. With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine. I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO. I think the opposite actually. If libvirt had two open monitor connections, one for normal use one for injection, then its open to racey usage where 2 monitor commands may be issued concurrently it is tricky to determine just which will be processed first, with all the scope for unexpected behaviour this entails. With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour. Regards, Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 05:28 AM, Daniel P. Berrange wrote: On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote: On 04/09/2010 09:27 AM, Daniel P. Berrange wrote: On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote: domain type='kvm' namemyguest/name ... debug monitorpassthrough/ commandline extraqemu arguments/extra alter option=optname renamenewname/rename matchREGEXP/match modifyfoo=on/modify extra-bar/extra /alter /commandline /debug /domain The concept of command line monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example domain type='kvm' xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0; namemyguest/name ... qemu:commandline qemu:arg-device/arg qemu:arglsi/arg /qemu:commandline /domain I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file. Currently libvirt doesn't use any config file at all, so command line is the only possible option. The config file is really just a structured command line argument so the two are really equivalent. From an XML perspective, the advantage of using the config file format is that it's very well structured whereas the command line arguments aren't. It's the difference between: qemu:commandline arg-drive/arg argfile=foo.img,if=virtio,cache=off/arg /qemu:commandline And: qemu:drive filefoo.img/file ifvirtio/if cacheoff/cache /qemu:drive The later being much more friendly for things like XPath and XQuery. We're going to evaluate switching to the config file at some point so can easily add further XML options to allow direct setting of config file entries at that point. I don't necessary think there's a super compelling reason for you to switch to pure config. Some things you'll have to do via -device and -set but it's still reasonable to keep using the command line. A key requirement of the config file effort was that everything that can be done via config file be doable via command line. With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine. I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO. I think the opposite actually. If libvirt had two open monitor connections, one for normal use one for injection, then its open to racey usage where 2 monitor commands may be issued concurrently it is tricky to determine just which will be processed first, with all the scope for unexpected behaviour this entails. Can you give an explicit example? The nature of this debug extension is such that I don't think 1 monitor really helps this problem... With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour. Except that libvirt doesn't know what the side effects of the debug commands are so it's intrinsically ambiguous :-) Regards, Anthony Liguori Regards, Daniel
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 07:48 AM, Avi Kivity wrote: On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device. Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device. There are a few ways to solve this, the simplest being that we give devices unique ids that are never reused and instead of pci_del taking a pci bus address, it takes a device id. That would address this race. You can get very far by just being clever about unique ids and notifications. There are some cases where a true RMW may be required but I can't really think of one off hand. The way LDAP addresses this is that it has a batched operation and a simple set of boolean comparison operations. This lets you execute a batched operation that will do a RMW. The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt. I don't think this is as difficult of a problem as you think it is. If you look at Active Directory and the whole set of management tools based on it, they certainly allow concurrent management applications. You can certainly get into trouble still but with just some careful considerations, you can make two management applications work together 90% of the time without much fuss on the applications part. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Fri, Apr 23, 2010 at 08:40:49AM -0500, Anthony Liguori wrote: On 04/23/2010 05:28 AM, Daniel P. Berrange wrote: On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote: On 04/09/2010 09:27 AM, Daniel P. Berrange wrote: On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote: domain type='kvm' namemyguest/name ... debug monitorpassthrough/ commandline extraqemu arguments/extra alter option=optname renamenewname/rename matchREGEXP/match modifyfoo=on/modify extra-bar/extra /alter /commandline /debug /domain The concept of command line monitor is something that is QEMU specific and thus is not suitable for the primary XML schema. IMHO, this needs to be done as a separate schema, linked in via an XML namespace. For example domain type='kvm' xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0; namemyguest/name ... qemu:commandline qemu:arg-device/arg qemu:arglsi/arg /qemu:commandline /domain I think it's problematic to focus too much on command line arguments. We are not introducing new command line arguments to qemu for the most part that aren't usable in the config file. Currently libvirt doesn't use any config file at all, so command line is the only possible option. The config file is really just a structured command line argument so the two are really equivalent. From an XML perspective, the advantage of using the config file format is that it's very well structured whereas the command line arguments aren't. It's the difference between: qemu:commandline arg-drive/arg argfile=foo.img,if=virtio,cache=off/arg /qemu:commandline And: qemu:drive filefoo.img/file ifvirtio/if cacheoff/cache /qemu:drive The later being much more friendly for things like XPath and XQuery. We're going to evaluate switching to the config file at some point so can easily add further XML options to allow direct setting of config file entries at that point. I don't necessary think there's a super compelling reason for you to switch to pure config. Some things you'll have to do via -device and -set but it's still reasonable to keep using the command line. A key requirement of the config file effort was that everything that can be done via config file be doable via command line. The main reason we're interested in the config is that it uses a better structured format, so we don't have such hairy escaping problems to deal with. Also, the command line is getting outrageously long these days with -device syntax. With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine. I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO. I think the opposite actually. If libvirt had two open monitor connections, one for normal use one for injection, then its open to racey usage where 2 monitor commands may be issued concurrently it is tricky to determine just which will be processed first, with all the scope for unexpected behaviour this entails. Can you give an explicit example? The nature of this debug extension is such that I don't think 1 monitor really helps this problem... Say libvirt is running a 'offline core dump' operation. This consists of us invoking stop migrate exec:cat foo.dump cont I don't want other debug commands accidentally being issued in between these steps. These 3 commands are in essence considered transactional block. Internally our locking model ensures that no other APIs can be invoked in this monitor connection during this sequence. Thus if someone uses the libvirt debug API to issue a monitor command, thus command will be blocked until the last 'cont' command is issued here. This provides us a known command ordering in the logs. With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour. Except that libvirt doesn't know what the side effects of the debug commands are so it's intrinsically ambiguous :-) This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 04:48 PM, Anthony Liguori wrote: On 04/23/2010 07:48 AM, Avi Kivity wrote: On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device. Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device. Obviously you should do the pci_del through libvirt. Once libvirt supports an API, use it. There are a few ways to solve this, the simplest being that we give devices unique ids that are never reused and instead of pci_del taking a pci bus address, it takes a device id. That would address this race. You can get very far by just being clever about unique ids and notifications. There are some cases where a true RMW may be required but I can't really think of one off hand. The way LDAP addresses this is that it has a batched operation and a simple set of boolean comparison operations. This lets you execute a batched operation that will do a RMW. I'm sure we can be very clever, but I'd rather direct this cleverness to qemu core issues, not to the QMP (which in turn requires that users be clever to use it correctly). QMP is a low bandwidth protocol, so races will never show up in testing. We're laying mines here for users to step on that we will never encounter ourselves. The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt. I don't think this is as difficult of a problem as you think it is. If you look at Active Directory and the whole set of management tools based on it, they certainly allow concurrent management applications. You can certainly get into trouble still but with just some careful considerations, you can make two management applications work together 90% of the time without much fuss on the applications part. Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt. -- Do not meddle in the internals of kernels, for they are subtle and quick to panic.
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Fri, Apr 23, 2010 at 08:48:51AM -0500, Anthony Liguori wrote: On 04/23/2010 07:48 AM, Avi Kivity wrote: On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device. Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device. This is what already happens with any QEMU = 0.12, where libvirt uses the new -device syntax with its 'id' parameter for all devices, and then uses 'device_id $ID' for unplug. The app still has to be careful it doesn't try to add a device using the same naming scheme as a device libvirt uses. This is easy enough though if they prepend some random string to all device IDs thy use, that won't clash with libvirt device ID names. Regards, Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
Anthony Liguori anth...@codemonkey.ws writes: On 04/23/2010 07:48 AM, Avi Kivity wrote: On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device. Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device. There are a few ways to solve this, the simplest being that we give devices unique ids that are never reused and instead of pci_del taking a pci bus address, it takes a device id. That would address this race. For what it's worth, that's how device_del works, except the ID is chosen by the user, who is free to reuse them. Really tangential to this thread, but it's worth repeating anyway: use device_add friends, not pci_add friends. [...]
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 09:24 AM, Avi Kivity wrote: On 04/23/2010 04:48 PM, Anthony Liguori wrote: On 04/23/2010 07:48 AM, Avi Kivity wrote: On 04/22/2010 09:49 PM, Anthony Liguori wrote: real API. Say, adding a device libvirt doesn't know about or stopping the VM while libvirt thinks it's still running or anything like that. Another problem is issuing Monitor commands that could confuse libvirt's We need to make libvirt and qemu smarter. We already face this problem today with multiple libvirt users. This is why sophisticated management mechanisms (like LDAP) have mechanisms to do transactions or at least a series of atomic operations. And people said qmp/json was overengineered... But seriously, transactions won't help anything. qemu maintains state, and when you have two updaters touching a shared variable not excepting each other to, things break, no matter how much locking there is. Let's consider some concrete examples. I'm using libvirt and QMP and in QMP, I want to hot unplug a device. Today, I do this by listing the pci devices, and issuing a pci_del that takes a PCI address. This is intrinsically racy though because in the worst case scenario, in between when I enumerate pci devices and do the pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add within libvirt of a completely different device. Obviously you should do the pci_del through libvirt. Once libvirt supports an API, use it. It was just an example... There are a few ways to solve this, the simplest being that we give devices unique ids that are never reused and instead of pci_del taking a pci bus address, it takes a device id. That would address this race. You can get very far by just being clever about unique ids and notifications. There are some cases where a true RMW may be required but I can't really think of one off hand. The way LDAP addresses this is that it has a batched operation and a simple set of boolean comparison operations. This lets you execute a batched operation that will do a RMW. I'm sure we can be very clever, but I'd rather direct this cleverness to qemu core issues, not to the QMP (which in turn requires that users be clever to use it correctly). QMP is a low bandwidth protocol, so races will never show up in testing. We're laying mines here for users to step on that we will never encounter ourselves. The only way that separate monitors could work is if they touch completely separate state, which is difficult to ensure if you upgrade your libvirt. I don't think this is as difficult of a problem as you think it is. If you look at Active Directory and the whole set of management tools based on it, they certainly allow concurrent management applications. You can certainly get into trouble still but with just some careful considerations, you can make two management applications work together 90% of the time without much fuss on the applications part. Maybe. We'll still have issues. For example, sVirt: if a QMP command names a labeled resource, the non-libvirt user will have no way of knowing how to label it. This is orthogonal to QMP and has to do strictly with how libvirt prepares a resource for qemu. Much better to exact a commitment from libvirt to track all QMP (and command line) capabilities. Instead of adding cleverness to QMP, add APIs to libvirt. Let's step back for a minute because I think we're missing the forest through the trees. We're trying to address a few distinct problems: 1) Allow libvirt users to access features of qemu that are not exposed through libvirt 2) Provide a means for non-libvirt users to interact with qemu 3) Provide a unified and interoperable view of the world for non-libvirt and libvirt users For (1), we all agree that the best case scenario would be for libvirt to support every qemu feature. I think we can also all agree though that this is not really practical and certainly not practical for developers since there is a development cost associated with libvirt support (to model an API appropriately). The new API proposed addresses (1) by allowing a user to drill down to the QMP context. It's a good solution IMHO and I think we all agree that there's an inherent risk to this that users will have to evaluate on a case-by-case basis. It's a good stop-gap though. (2) is largely addressed by QMP and a config file. I'd like to see a nice C library, but I think a lot of other folks are happy with JSON support in higher level languages. (3) is the place where there are still potential challenges. I think at the very least, our goal should be to enable conversion from (2) and (1) to be as easy as possible. That's why I have proposed implementing a C library for the JSON transport because we could plumb that through the new libvirt API. This would allow a user to very quickly port an application from QMP to libvirt. In order to do this, we need the
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/23/2010 09:21 AM, Daniel P. Berrange wrote: Say libvirt is running a 'offline core dump' operation. This consists of us invoking stop migrate exec:cat foo.dump cont I don't want other debug commands accidentally being issued in between these steps. These 3 commands are in essence considered transactional block. Internally our locking model ensures that no other APIs can be invoked in this monitor connection during this sequence. Thus if someone uses the libvirt debug API to issue a monitor command, thus command will be blocked until the last 'cont' command is issued here. This provides us a known command ordering in the logs. Ah, so what'd I'd suggest is that you still use two monitors but that you use the same locking for both. The reason a second monitor is useful is that you can allow different event masks to be setup on the second monitor which I think a QMP client will expect. I don't see any trouble with respect to command ordering here. libvirt is ultimately the one sending commands to monitor. You know which one was sent and which one completed in what order. With a single monitor connection, libvirt's current locking model for the monitor ensures that QMP monitor commands are reliably serialized onto the wire, giving unambiguous behaviour. Except that libvirt doesn't know what the side effects of the debug commands are so it's intrinsically ambiguous :-) This is a different ambiguity, about the semantic results of the commands, where as I'm refering to the execution order. If I look at a libvirt log file and see a set of JSON commands logged, I want to know that this ordering from the logs, was indeed the same as order in which qemu processed them. If you have two separate monitor connection you can't be sure of the order of execution. It is key for our bug troubleshooting that given a libvirt log file, we can replay the JSON commands again and get the same results. Two monitor connections is just increasing complexity of code without any tangible benefit. I think you're assuming direct access to the second monitor? I'm not suggesting that. I'm suggesting that libvirt is still the one submitting commands to the second monitor and that it submits those commands in lock step. Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
Hello,
In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors. The goals of
this API are:
1) To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2) To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.
Caveats:
1) Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
So in both cases the API and XML format can be reasonably guarenteed
between releases. What we can't guarentee is that usage of these
features will be reliable across releases of libvirt.
2) Again unlike other libvirt API's, access and configuration of
the debug section of a domain will be highly hypervisor dependent.
3) Application developers will be strongly discouraged from using
this API because of the above 2 issues. To help in this, the
API's will be in a separate library that developers will explicitly
have to link to, and it will have a different (but largely compatible)
wire protocol.
In terms of wire protocol, this would still have to run over the same
existing data channel, because we don't want apps to have to open up
multiple connections. Fortunately our protocol is designed to allow
this kind of extension
struct remote_message_header {
unsigned prog; /* REMOTE_PROGRAM */
unsigned vers; /* REMOTE_PROTOCOL_VERSION */
remote_procedure proc; /* REMOTE_PROC_x */
remote_message_type type;
unsigned serial;/* Serial number of message. */
remote_message_status status;
};
We currently have a single program 'REMOTE_PROGRAM' and its associated
'vers' and 'proc' numbers. What we would be doing is to define a new
program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus
start again from '1' without clashing with any of the existing APIs.
All the RPC marshalling/demarshalling code should work more or less
unchanged. We just hook in a different function dispatch table in the
daemon for that program.
4) We don't expect this API to solve all of the issues brought up
during the qemud discussion. Our initial goal is just to give
ready access of the qemu command-line and monitor to developers.
With that being said, our initial proposal follows. We expect this
to evolve over time as we get more feedback, but we think this
proposal addresses at least 2 of the major pain points qemu developers
have while trying to use libvirt.
The initial debug XML for qemu would be:
I wouldn't call them 'debug' options, since that's just one of possible
use cases for this. What we're really doing there is exposing hypervisor
specific features, so we should just be upfront about that in our
description / naming.
domain type='kvm'
namemyguest/name
...
debug
monitorpassthrough/
commandline
extraqemu arguments/extra
alter option=optname
renamenewname/rename
matchREGEXP/match
modifyfoo=on/modify
extra-bar/extra
/alter
/commandline
/debug
/domain
The concept of command line monitor is something that is QEMU specific
and thus is not suitable for the primary XML schema. IMHO, this needs to be
done as a separate schema, linked in via an XML namespace. For example
domain type='kvm' xmlns:qemu=http://libvirt.org/schemas/domain/qemu/1.0;
namemyguest/name
...
qemu:commandline
qemu:arg-device/arg
qemu:arglsi/arg
/qemu:commandline
/domain
I think it's problematic to focus too much on command line arguments.
We are not introducing new command line arguments to qemu for the most
part that aren't usable in the config file.
What I really think we would like, is a way to express the qemu
configuration file in terms of XML and then to use snippets of that
within an XML namespace. For instance:
[cpudef]
name = Opteron_G3
level = 5
vendor = AuthenticAMD
...
Could convert to:
cpudef
nameOperon_G3/name
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/12/2010 10:20 AM, Luiz Capitulino wrote:
On Fri, 9 Apr 2010 15:27:17 +0100
Daniel P. Berrangeberra...@redhat.com wrote:
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
Hello,
In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors. The goals of
this API are:
1) To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2) To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.
Caveats:
1) Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
Another problem is issuing Monitor commands that could confuse libvirt's
real API. Say, adding a device libvirt doesn't know about or stopping the VM
while libvirt thinks it's still running or anything like that.
We need to make libvirt and qemu smarter.
We already face this problem today with multiple libvirt users. This is
why sophisticated management mechanisms (like LDAP) have mechanisms to
do transactions or at least a series of atomic operations.
Regards,
Anthony Liguori
When I suggested the monitor passthrough idea, I thought it could be a
workaround for the feature time lag problem, but as stated by Avi, I'm
afraid that this can potentially cause more problems than it solves.
It's very desirable for debugging (item 2 above) and maybe to try new
things, but people couldn't use it for anything serious, which is the
major issue in this subject (item 1 above).
Now, Anthony's idea of having libqemu has the same issues, which seems
to mean that the only way to really fix the time lag is to fix the
time lag :)
Raw access to the qemu monitor will be disabled by default; the
monitorpassthrough/ tag enables the ability to send QMP (or
text, if you are using older qemu) messages straight through to the
monitor. To do this there will be an additional API entry point
named virDomainDebugCommand() which takes an arbitrary string
and passes it to the monitor, and returns an arbitrary string as
a result. Thus you could pass in either info cpus if using the
text monitor or '{ execute: query-cpus }' if using QMP.
Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file
#includelibvirt/libvirt-qemu.h
Containing APIs like
int virDomainQEMUInvokeMonitor(virDomainPtr dom,
const char *command,
char **reply);
typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
const char *data,
void *opaque)
int virConnectQEMUDomainEventRegister(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
virDomainQEMUMonitorCallback cb,
void *opaque);
What about all the wonders of json? This will force clients to
have their own json- text converter, maybe we could provide two
APIs, just like qemu driver works internally?
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/22/2010 01:45 PM, Anthony Liguori wrote: Which in turn, could be embedded as: domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0' namemyguest/name ... qemu:cpudef nameOperon_G3/name level5/level vendorAuthenticAMD/vendor /qemu:cpudef /domain With respect to injecting QMP commands directly, I think the proposed debug API is probably reasonable. We could build a libqemu that used that API as a transport which means that one could use libqemu and libvirt simultaneously which is certainly a key requirement of mine. I think it's important that it's a dedicated monitor session though. It shouldn't just be injecting commands within an existing QMP session IMHO. Another important thing to mention is that this type of conversion has two purposes. A user can inject qemu config snippets into the libvirt description, but libvirt can also inject qemu config snippets that it cannot represent via it's own XML for the user. For instance, let's say that a user adds a disk via QMP with an option that libvirt doesn't recognize. When a virsh dumpxml is executed, libvirt could display the XML using the qemu:drive tag instead of the disk tag that it normally would use. Regards, Anthony Liguori Regards, Anthony Liguori
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote: Richard W.M. Jones wrote: On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote: Daniel P. Berrange wrote: I think this alteration of existing args is fr too complex fragile, and way overkill. Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted? I might be missing something but I thought you could already do this in current libvirt. ie. Change the emulator element [1] to point to your shell script: emulator/usr/local/bin/my-qemu-wrapper.sh/emulator The wrapper should get called with the command line arguments and you can alter them however you like. Isn't that right, Dan? That forces the shell script to be relatively complicated and recognise all qemu arguments (including version specific ones) to skip the ones with parameters - and to know where to look for the real target executable is. It's not a very friendly way to add or change parts of specific subsystems, for example modifying a blockdev parameter, but it can be done that way if necessary. It's not that hard to write this for trivial extra options: emulator/bin/sh -c 'qemu $0 $@ -extra-flag'/emulator (if that works). But for things like modifications to particular devices, wouldn't it be better to associate add cache=special option to a section about one of the blockdevs which libvirt knows about, rather than the awful prospect of parsing libvirt's output to guess which of three -device arguments corresponds with the second SCSI drive mentioned in the libvirt config - which might not necessarily be the second on the command line, even. Every device has a unique alias in the XML, which maps onto the 'id=' parameter for QEMU so you can match up them. We need to consider the tradeoff between what's likely to be neccessary vs cost of supporting the extensions. Allowing addition of arbitrary extra command lines is a trivial extension with low support cost covers the most pressing feature timelag problems. Munging existing dev options is a less common scenario with significantly higher support burden which I don't really consider worth explicitly supporting beyond the existing ability to add a wrapper script. Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On 04/12/2010 08:41 AM, Daniel P. Berrange wrote: I don't think there's much to be gained from having an XML element to turn on/off use of these APIs. If an app doesn't want to use them, it can simply not link to libvirt-qemu.so The reason I wanted to do this was mostly for debug/support reasons. That is, with this element in place we can easily tell from the dumpxml output whether a person was using the unreliable API's, and thus we can tell them to try and reproduce without that in place. That doesn't tell you whether they have actually used any API or not. It is also inconvenient if you start a guest without it, and only later realize you want to use the extra APIs. If we want to track the actual usage, then the first time a direct monitor command is issued, we should simply log a warning message. The problem with logging a message is that it is easy to lose it. What I'm trying to avoid here is debugging somebody's setup for hours only to find out that they did a pci_del behind libvirt's back. Maybe we can just make the monitorpassthrough/ a read-only flag; it is ignored in the parsing, but it is set by GetXMLDesc when it detects that the virDomainQemuInvokeMonitor has been called. -- Chris Lalancette
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Mon, Apr 12, 2010 at 09:56:50AM -0400, Chris Lalancette wrote: On 04/12/2010 08:41 AM, Daniel P. Berrange wrote: I don't think there's much to be gained from having an XML element to turn on/off use of these APIs. If an app doesn't want to use them, it can simply not link to libvirt-qemu.so The reason I wanted to do this was mostly for debug/support reasons. That is, with this element in place we can easily tell from the dumpxml output whether a person was using the unreliable API's, and thus we can tell them to try and reproduce without that in place. That doesn't tell you whether they have actually used any API or not. It is also inconvenient if you start a guest without it, and only later realize you want to use the extra APIs. If we want to track the actual usage, then the first time a direct monitor command is issued, we should simply log a warning message. The problem with logging a message is that it is easy to lose it. What I'm trying to avoid here is debugging somebody's setup for hours only to find out that they did a pci_del behind libvirt's back. Maybe we can just make the monitorpassthrough/ a read-only flag; it is ignored in the parsing, but it is set by GetXMLDesc when it detects that the virDomainQemuInvokeMonitor has been called. If the logging message is at level 'WARN' it will be included in syslog for libvirtd by default, which is basic support data always collected. The guest XML is a description of the guest configuration, which is not a place for ad-hoc flags related to API usage. Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Fri, 9 Apr 2010 15:27:17 +0100
Daniel P. Berrange berra...@redhat.com wrote:
On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
Hello,
In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors. The goals of
this API are:
1) To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2) To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.
Caveats:
1) Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU the libvirt driver.
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash.
Another problem is issuing Monitor commands that could confuse libvirt's
real API. Say, adding a device libvirt doesn't know about or stopping the VM
while libvirt thinks it's still running or anything like that.
When I suggested the monitor passthrough idea, I thought it could be a
workaround for the feature time lag problem, but as stated by Avi, I'm
afraid that this can potentially cause more problems than it solves.
It's very desirable for debugging (item 2 above) and maybe to try new
things, but people couldn't use it for anything serious, which is the
major issue in this subject (item 1 above).
Now, Anthony's idea of having libqemu has the same issues, which seems
to mean that the only way to really fix the time lag is to fix the
time lag :)
Raw access to the qemu monitor will be disabled by default; the
monitorpassthrough/ tag enables the ability to send QMP (or
text, if you are using older qemu) messages straight through to the
monitor. To do this there will be an additional API entry point
named virDomainDebugCommand() which takes an arbitrary string
and passes it to the monitor, and returns an arbitrary string as
a result. Thus you could pass in either info cpus if using the
text monitor or '{ execute: query-cpus }' if using QMP.
Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file
#include libvirt/libvirt-qemu.h
Containing APIs like
int virDomainQEMUInvokeMonitor(virDomainPtr dom,
const char *command,
char **reply);
typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
const char *data,
void *opaque)
int virConnectQEMUDomainEventRegister(virConnectPtr conn,
virDomainPtr dom,
const char *eventname,
virDomainQEMUMonitorCallback cb,
void *opaque);
What about all the wonders of json? This will force clients to
have their own json - text converter, maybe we could provide two
APIs, just like qemu driver works internally?
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote: Daniel P. Berrange wrote: I think this alteration of existing args is fr too complex fragile, and way overkill. Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted? I might be missing something but I thought you could already do this in current libvirt. ie. Change the emulator element [1] to point to your shell script: emulator/usr/local/bin/my-qemu-wrapper.sh/emulator The wrapper should get called with the command line arguments and you can alter them however you like. Isn't that right, Dan? Rich. [1] http://libvirt.org/formatdomain.html#elementsDevices -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
Richard W.M. Jones wrote: On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote: Daniel P. Berrange wrote: I think this alteration of existing args is fr too complex fragile, and way overkill. Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted? I might be missing something but I thought you could already do this in current libvirt. ie. Change the emulator element [1] to point to your shell script: emulator/usr/local/bin/my-qemu-wrapper.sh/emulator The wrapper should get called with the command line arguments and you can alter them however you like. Isn't that right, Dan? That forces the shell script to be relatively complicated and recognise all qemu arguments (including version specific ones) to skip the ones with parameters - and to know where to look for the real target executable is. It's not a very friendly way to add or change parts of specific subsystems, for example modifying a blockdev parameter, but it can be done that way if necessary. It's not that hard to write this for trivial extra options: emulator/bin/sh -c 'qemu $0 $@ -extra-flag'/emulator (if that works). But for things like modifications to particular devices, wouldn't it be better to associate add cache=special option to a section about one of the blockdevs which libvirt knows about, rather than the awful prospect of parsing libvirt's output to guess which of three -device arguments corresponds with the second SCSI drive mentioned in the libvirt config - which might not necessarily be the second on the command line, even. Parsing libvirt output and having to guess which option corresponds to what from the libvirt config sounds very fragile and also a rather large amount of effort for something which should be easy. And then there's modifying how network tap devices are set up and other networking, outside qemu, but requested by libvirt. Doing that in emulator sounds like a bad idea, but script hooks are the only sane way to make networking fit every environment and requirement. Ideally without forcing the hook writer to reimplement the normal network setup from scratch just to tweak it a bit. I apologise if libvirt already provides such hooks - I haven't looked at that part of it. -- Jamie
Re: [Qemu-devel] Re: [libvirt] Libvirt debug API
Daniel P. Berrange wrote: I think this alteration of existing args is fr too complex fragile, and way overkill. Would it not be simpler, for the target audience, for the config to contain a one-line shell script to transform particular matched arguments in any way that's wanted? If the arg that libvirt generates isn't what someone needs, then remove the bit of the guest config responsible for that and add a complete extra arg, rather than munging the existing one. Problem with that: You might want libvirt to think it has a device that it's controlling, still show in the GUI, still tie up with the rest of the stack, still do the management setup, but you just want to change some qemu-specific parameters. For example, cache=somethingelse for block devices, iptables=tobedefined for net devices, usbimplementation=hack for USB devices. -- Jamie
