Re: [Libvir] save/restore support for KVM
On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: On Thu, Aug 09, 2007 at 05:26:43PM -0400, Jim Paris wrote: Hi, I've implemented save/restore support for KVM using the live migration feature. First, a few patches to KVM to fix bugs: Very cool :-) Then, another patch to KVM to support inbound migration from a filename. It already supports migration from stdin, but adding this seemed easier from a libvirt perspective. Subject: [PATCH] qemu: accept filename for incoming migration http://article.gmane.org/gmane.comp.emulators.kvm.devel/5590 Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. Finally, the libvirt side. Some notes: - Arbitrary decisions about VM status: I pause the VM before starting migration, and destroy it once it's finished. Neither is required by KVM but I'd be concerned about the disk image state otherwise. Also, after resuming, the VM is still paused. I don't know how Xen behaves in this regard but the behavior here is easy enough to change. Xen doesn't mind whether the VM is running or paused when saving it. Pausing it seems reasonable - its going to be stopped shortly anyway, so letting it continue running just increases the saved image size by having to process dirtied memory. As for restore, I'd be inclined to leave it running after restore to match our Xen driver. I must admit I feel more comfortable with pausing, reduces the complexity of the operation and could avoid nasty bugs near impossible to track. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Fix paravirt FB with xm driver
On Fri, Aug 10, 2007 at 05:16:28PM +0100, Richard W.M. Jones wrote: Daniel P. Berrange wrote: The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. I too am having trouble following the explanation / code. Can you explain a bit more ...? Two core issues For paravirt the framebuffer can be configured in either old style: vnc=1 vncunused=1 Or new style vfb = [type=vnc,vncunused=1 The problem was that with Xen 3.0.3 we were only handling the latter and not the former. We needed to process both styles for compatability. The second part of it is that while we handled graphics type='vnc' port='-1' / Converting to 'vncunused=1'. We did not handle graphics type='vnc' port='5905' / Which needs to convert to vncunused=0, and vncdisplay=5 Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
On Fri, Aug 10, 2007 at 12:36:05PM -0400, Jim Paris wrote: Daniel Veillard wrote: On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. I'm not sure the kvm qemu binary even reports the kvm version anywhere. I'll ask on kvm-devel to see if qemu/VERSION could get updated with each KVM release. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. I think appending unrelated data to the migration image is a bit of a hack anyway. A better plan would be a file containing header XML config migration data On save, libvirt writes header and XML config, then closes it and uses dd of=path oflag=append conv=notrunc or just cat path as the migration command. On restore, libvirt reads the header and XML config, and then feeds the remaining migration data to KVM using -incoming stdio. I had wanted to avoid the trouble of feeding data via stdin, but maybe a well placed dup2(fd,STDIN_FILENO) would do the trick automatically. That sounds like a reasonable plan This file format would also make it easier for e.g. virt-manager to determine that a file is a valid libvirt restore image. A little magic in the first couple of bytes would be handy - thats what we check currently for Xen. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Richard W.M. Jones wrote: Jim Paris wrote: +if (strchr(path, '\'') || strchr(path, '\\') ) { +qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED, + invalid filename); +return -1; +} [...] +/* Migrate to file. */ +if (asprintf (command, migrate \exec:dd of='%s' 2/dev/null\\n, + path) == -1) { +qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED, + out of memory); +return -1; +} The patch is fine, except I'm wondering whether the quoting above is safe. We check if the path contains ' or \ and refuse to proceed. I _think_ you don't need to check for \ however I think you're right. An even better fix would be to explicitly escape bad characters in the path before passing them along. Giving an error on the filename Jim's VM as it would do right now isn't ideal. -jim -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Daniel Veillard wrote: On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. I'm not sure the kvm qemu binary even reports the kvm version anywhere. I'll ask on kvm-devel to see if qemu/VERSION could get updated with each KVM release. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. I think appending unrelated data to the migration image is a bit of a hack anyway. A better plan would be a file containing header XML config migration data On save, libvirt writes header and XML config, then closes it and uses dd of=path oflag=append conv=notrunc or just cat path as the migration command. On restore, libvirt reads the header and XML config, and then feeds the remaining migration data to KVM using -incoming stdio. I had wanted to avoid the trouble of feeding data via stdin, but maybe a well placed dup2(fd,STDIN_FILENO) would do the trick automatically. This file format would also make it easier for e.g. virt-manager to determine that a file is a valid libvirt restore image. -jim -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[Libvir] PATCH: Fix paravirt FB with xm driver
The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| diff -rup libvirt-0.2.3.orig/src/xm_internal.c libvirt-0.2.3.new/src/xm_internal.c --- libvirt-0.2.3.orig/src/xm_internal.c2007-07-24 14:32:41.0 -0400 +++ libvirt-0.2.3.new/src/xm_internal.c 2007-08-09 12:46:58.0 -0400 @@ -916,7 +916,8 @@ char *xenXMDomainFormatXML(virConnectPtr } if (xenXMConfigGetInt(conf, sdl, val) == 0 val) sdl = 1; -} else { /* New PV guests use this format */ +} +if (!hvm !sdl !vnc) { /* New PV guests use this format */ list = virConfGetValue(conf, vfb); if (list list-type == VIR_CONF_LIST list-list list-list-type == VIR_CONF_STRING @@ -943,6 +944,8 @@ char *xenXMDomainFormatXML(virConnectPtr sdl = 1; } else if (!strncmp(key, type=vnc, 8)) { vnc = 1; +} else if (!strncmp(key, vncunused=, 10)) { +vncunused = strtol(key+10, NULL, 10); } else if (!strncmp(key, vnclisten=, 10)) { vnclisten = key + 10; } else if (!strncmp(key, vncpasswd=, 10)) { @@ -950,11 +953,7 @@ char *xenXMDomainFormatXML(virConnectPtr } else if (!strncmp(key, keymap=, 7)) { keymap = key + 7; } else if (!strncmp(key, vncdisplay=, 11)) { -int port = strtol(key+11, NULL, 10); -if (port == -1) -vncunused = 1; -else -port = port - 5900; +vncdisplay = strtol(key+11, NULL, 10); } while (nextkey (nextkey[0] == ',' || @@ -1899,11 +1917,18 @@ virConfPtr xenXMParseXMLToConfig(virConn cannot set the keymap parameter) 0) goto error; -/* XXX vncdisplay */ -/* - if (xenXMConfigSetIntFromXPath(conn, conf, ctxt, vncdisplay, string(int(/domain/devices/[EMAIL PROTECTED]'vnc']/@vncport) - 5900)), 0, 0) 0) - goto error; -*/ +obj = xmlXPathEval(BAD_CAST string(/domain/devices/[EMAIL PROTECTED]'vnc']/@port), ctxt); +if ((obj != NULL) (obj-type == XPATH_STRING) +(obj-stringval != NULL)) { +int port = strtol((const char *)obj-stringval, NULL, 10); +if (port != -1) { +char portstr[50]; +snprintf(portstr, sizeof(portstr), %d, port-5900); +if (xenXMConfigSetString(conf, vncdisplay, portstr) 0) +goto error; +} +} +xmlXPathFreeObject(obj); } else { virConfValuePtr vfb; obj = xmlXPathEval(BAD_CAST /domain/devices/graphics, ctxt); -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[Libvir] [PATCH] Block device and network stats
This is the implementation (currently Xen, local only). It includes a couple of virsh commands if you want to play with looking at the stats. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 Index: include/libvirt/libvirt.h.in === RCS file: /data/cvs/libvirt/include/libvirt/libvirt.h.in,v retrieving revision 1.31 diff -u -p -r1.31 libvirt.h.in --- include/libvirt/libvirt.h.in 18 Jul 2007 10:11:10 - 1.31 +++ include/libvirt/libvirt.h.in 10 Aug 2007 14:30:21 - @@ -14,6 +14,9 @@ #ifndef __VIR_VIRLIB_H__ #define __VIR_VIRLIB_H__ +#include sys/types.h +#include stdint.h + #ifdef __cplusplus extern C { #endif @@ -197,6 +200,37 @@ int virDomainSetSchedulerParameters (vir virSchedParameterPtr params, int nparams); +/* Block device stats for virDomainBlockStats. + * + * Hypervisors may return a field set to (int64_t)-1 which indicates + * that the hypervisor does not support that statistic. + */ +struct _virDomainBlockStats { + int64_t rd_req; + int64_t rd_bytes; + int64_t wr_req; + int64_t wr_bytes; + int64_t errs; // In Xen this returns the mysterious 'oo_req'. +}; +typedef struct _virDomainBlockStats *virDomainBlockStatsPtr; + +/* Network interface stats for virDomainInterfaceStats. + * + * Hypervisors may return a field set to (int64_t)-1 which indicates + * that the hypervisor does not support that statistic. + */ +struct _virDomainInterfaceStats { + int64_t rx_bytes; + int64_t rx_packets; + int64_t rx_errs; + int64_t rx_drop; + int64_t tx_bytes; + int64_t tx_packets; + int64_t tx_errs; + int64_t tx_drop; +}; +typedef struct _virDomainInterfaceStats *virDomainInterfaceStatsPtr; + /** * VIR_NODEINFO_MAXCPUS: * @nodeinfo: virNodeInfo instance @@ -369,6 +403,16 @@ int virDomainGetMaxVcpus (virDomainPtr char * virDomainGetXMLDesc (virDomainPtr domain, int flags); +int virDomainBlockStats (virDomainPtr dom, + const char *path, + virDomainBlockStatsPtr stats, + size_t size); +int virDomainInterfaceStats (virDomainPtr dom, + const char *path, + virDomainInterfaceStatsPtr stats, + size_t size); + + /* * defined but not running domains */ Index: src/driver.h === RCS file: /data/cvs/libvirt/src/driver.h,v retrieving revision 1.32 diff -u -p -r1.32 driver.h --- src/driver.h 27 Jul 2007 23:23:00 - 1.32 +++ src/driver.h 10 Aug 2007 14:30:21 - @@ -181,6 +181,17 @@ typedef int virSchedParameterPtr params, int nparams); +typedef int +(*virDrvDomainBlockStats) +(virDomainPtr domain, + const char *path, + struct _virDomainBlockStats *stats); +typedef int +(*virDrvDomainInterfaceStats) +(virDomainPtr domain, + const char *path, + struct _virDomainInterfaceStats *stats); + typedef struct _virDriver virDriver; typedef virDriver *virDriverPtr; @@ -245,6 +256,8 @@ struct _virDriver { virDrvDomainGetSchedulerType domainGetSchedulerType; virDrvDomainGetSchedulerParameters domainGetSchedulerParameters; virDrvDomainSetSchedulerParameters domainSetSchedulerParameters; +virDrvDomainBlockStats domainBlockStats; +virDrvDomainInterfaceStats domainInterfaceStats; }; typedef int Index: src/libvirt.c === RCS file: /data/cvs/libvirt/src/libvirt.c,v retrieving revision 1.93 diff -u -p -r1.93 libvirt.c --- src/libvirt.c 9 Aug 2007 20:19:12 - 1.93 +++ src/libvirt.c 10 Aug 2007 14:30:23 - @@ -1830,6 +1830,116 @@ virDomainSetSchedulerParameters(virDomai } +/** + * virDomainBlockStats + * @dom: pointer to the domain object + * @path: path to the block device + * @stats: block device stats (returned) + * @size: size of stats structure + * + * This function returns block device (disk) stats for block + * devices attached to the domain. + * + * The path parameter is the name of the block device. Get this + * by calling virDomainGetXMLDesc and finding the target dev='...' + * attribute within //domain/devices/disk. (For example, xvda). + * + * Domains may have more than one block device. To get stats for + * each you should make multiple calls to this function. + * + * Individual fields within the stats structure may be returned + * as -1, which indicates that the hypervisor does not support + * that particular statistic. + * + * Returns: 0 in case of success or -1 in case of failure. + */ +int +virDomainBlockStats (virDomainPtr dom, const char *path, + virDomainBlockStatsPtr stats,
Re: [Libvir] PATCH: Fix paravirt FB with xm driver
On Fri, Aug 10, 2007 at 05:10:46PM +0100, Daniel P. Berrange wrote: On Fri, Aug 10, 2007 at 12:08:26PM -0400, Daniel Veillard wrote: On Fri, Aug 10, 2007 at 04:39:03PM +0100, Daniel P. Berrange wrote: The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. I'm not sure I undertand the logic for setting the port, why going whould xenstore there ? I'm not sure what you mean - xenstore isn't involved in this. This is for taking the XML and writing out the /etc/xen config - the port stuff is basically making sure that port=5903 is translated into the config option 'vncdisplay=3'. Previously only port=-1 was working. Whoops reread, yeah, I don't understand why I saw a xenstore operation there ! weird ... Okay +1 Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Xen-users] Re: [Libvir] Xen physical host statistics
Mindaugas Kiznis wrote: Thank you for your quick answer. In my opinion, I/O statistics are critical as CPU, memory usage in data center environment to optimally plan, arrange, spread VMs in a resource or say Xenhost pool. Since there are many Xen hosts in my case and this number is growing, it is a need to monitor each host's resource usage in a centralized console, graphing engine, etc. Hope to see these additions in the next releases of libvirt. Absolutely. As Daniel V said however, defining a suitable, portable structure which can handle future hypervisors and future devices is non-trivial. I'm working on it, and when I post patches you can help by reviewing them against the needs of your program. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Daniel P. Berrange wrote: - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. How does this (or even _does_ this) work in the Xen case? Does Xen save this info like UUID, name, etc. in the migration image? Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Fix paravirt FB with xm driver
Daniel P. Berrange wrote: The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. I too am having trouble following the explanation / code. Can you explain a bit more ...? Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[Libvir] Xen physical host statistics
Hello, - I am trying to make an application, which would connect to a remote xen host and fetch some information about total (physical) CPU usage, total memory usage, network and disk I/O. I have managed to do a remote statistics grabber for CPU and memory usage using libvirt. I couldn't find any source of information about libvirt ability to return I/O numbers. Also haven't found any other solution yet. Any suggestions? Thank you in advance! Best regards, Mindaugas Kiznis -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Jim Paris wrote: +if (strchr(path, '\'') || strchr(path, '\\') ) { +qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED, + invalid filename); +return -1; +} [...] +/* Migrate to file. */ +if (asprintf (command, migrate \exec:dd of='%s' 2/dev/null\\n, + path) == -1) { +qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED, + out of memory); +return -1; +} The patch is fine, except I'm wondering whether the quoting above is safe. We check if the path contains ' or \ and refuse to proceed. I _think_ you don't need to check for \ however, according to this section from the bash manual page and my testing: Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash. Perhaps it is better to be safe than sorry though. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] Xen physical host statistics
Mindaugas Kiznis wrote: - I am trying to make an application, which would connect to a remote xen host and fetch some information about total (physical) CPU usage, total memory usage, network and disk I/O. I have managed to do a remote statistics grabber for CPU and memory usage using libvirt. I couldn't find any source of information about libvirt ability to return I/O numbers. Also haven't found any other solution yet. Any suggestions? It's very interesting that you should ask this, because I am investigating adding network I/O and block I/O stats to libvirt. We would get the information from /sys/devices/xen-backend/... in the same way that libxenstat does it at the moment, and pass that up through libvirt APIs. As with all libvirt-related stuff we need to think about how it will apply to the other hypervisors that we support and could support in the future. You are quite right that there is no facility for this in libvirt right now (just CPU-/memory-based stats at the moment). See also: http://et.redhat.com/~rjones/virt-top/ Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Improve performance on Xen
On Thu, Aug 09, 2007 at 11:13:00PM +0100, Daniel P. Berrange wrote: The Xen implementations of virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece have to lookup the other piece. Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD. Great As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall. Sounds good to though definitely not a bottleneck. That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though. Well if you have 100 guests, that may be slower, but in the average situation of only a couple of guests, it could be a real speedup. The problem is that a lot of domain may accumulate in xenstore /local/domain even if they are not running, both implementation are likely to have completely different behaviour based on the context. But from a cache locality perspective hitting xenstore may scale way better under loaded machines, so it may prove faster even on machines with hundreds of domains. Doing a fair performance comparison may prove really hard. Any for the 3 improvements I have done, the 'virsh dominfo' call has improved when using either ID or UUID: [...] A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently. Too good to ignore, that looks even too good to be true :-) Index: src/xen_internal.c Patch looks good to me +1, fanxastic ! Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Fix paravirt FB with xm driver
On Fri, Aug 10, 2007 at 04:39:03PM +0100, Daniel P. Berrange wrote: The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. I'm not sure I undertand the logic for setting the port, why going whould xenstore there ? Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] [PATCH] Block device and network stats
Daniel P. Berrange wrote: The virbrN / xenbrN devices are bridges so see all traffic from any guest attached. We need to look at ths individual vifN.N or vnetN devices which are per-guest. This info isn't included in the XML though. If we can figure out a efficient way to export vif name in the XML we could use that, otherwie the API should probably take the MAC address on the device and use that to resolve the underlying VIF. The former is nicer though if we can do it. The best I could come up with is this little bundle of joy ... diff -u -p -r1.132 xend_internal.c --- src/xend_internal.c 9 Aug 2007 20:19:12 - 1.132 +++ src/xend_internal.c 10 Aug 2007 15:07:20 - @@ -1354,6 +1354,7 @@ xend_parse_sexp_desc(virConnectPtr conn, int max_mem, cur_mem; unsigned char uuid[VIR_UUID_BUFLEN]; char uuidstr[VIR_UUID_STRING_BUFLEN]; +int vif_index = 0; if (root == NULL) { /* ERROR */ @@ -1626,6 +1627,9 @@ xend_parse_sexp_desc(virConnectPtr conn, if (tmp) virBufferVSprintf(buf, target dev='%s'/\n, tmp); +else +virBufferVSprintf(buf, target dev='vif%d.%d'/\n, + domid, vif_index); tmp = sexpr_node(node, device/vif/mac); if (tmp) virBufferVSprintf(buf, mac address='%s'/\n, @@ -1639,6 +1643,7 @@ xend_parse_sexp_desc(virConnectPtr conn, tmp2); virBufferAdd(buf, /interface\n, 17); +vif_index++; } else if (sexpr_lookup(node, device/vfb)) { /* New style graphics config for PV guests in = 3.0.4, * or for HVM guests in = 3.0.5 */ If that isn't acceptable then I suggest we just modify the Xen implementation of domainInterfaceStats so that it accepts a path of the form vifdomid.portnum for now, and return to the problem of getting the real vif later. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] Proposal: Block device and network stats
Daniel P. Berrange wrote: It is probably worthwhile though to ensure that we design the APIs so that the structs are always allocated by the internal driver and not the caller. This allows us to add more fields at a later date if needed. Instead of the above, the caller has to pass in the size of the struct. Pros to passing in the struct size: * New caller / old libvirt can be detected, rather than causing a segfault. * Caller is less likely to forget to free the struct (because it is mostly likely on their stack, or they explicitly malloc'd it). But yes, unsafe C linkage lack of garbage collection sucks ... Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] Proposal: Block device and network stats
On Fri, Aug 10, 2007 at 01:59:34PM +0100, Richard W.M. Jones wrote: It was suggested to me that we should provide a way to return bytes read and written on block devices (not just requests). Xen doesn't support that however, so I have also changed the fields in this structure so that they can be returned set to -1 to indicate no data / not supported. The updated stats structures are shown below. Rich. /* Block device stats for virDomainBlockStats. * * Hypervisors may return a field set to (int64_t)-1 which indicates * that the hypervisor does not support that statistic. */ struct _virDomainBlockStats { int64_t rd_req; int64_t rd_bytes; int64_t wr_req; int64_t wr_byes; int64_t errs; // In Xen this returns the mysterious 'oo_req'. }; typedef struct _virDomainBlockStats *virDomainBlockStatsPtr; /* Network interface stats for virDomainInterfaceStats. * * Hypervisors may return a field set to (int64_t)-1 which indicates * that the hypervisor does not support that statistic. */ struct _virDomainInterfaceStats { int64_t rx_bytes; int64_t rx_packets; int64_t rx_errs; int64_t rx_drop; int64_t tx_bytes; int64_t tx_packets; int64_t tx_errs; int64_t tx_drop; }; typedef struct _virDomainInterfaceStats *virDomainInterfaceStatsPtr; Seems like a reasonable set of fields. It is probably worthwhile though to ensure that we design the APIs so that the structs are always allocated by the internal driver and not the caller. This allows us to add more fields at a later date if needed. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] Proposal: Block device and network stats
On Fri, Aug 10, 2007 at 01:59:34PM +0100, Richard W.M. Jones wrote: It was suggested to me that we should provide a way to return bytes read and written on block devices (not just requests). Xen doesn't support that however, so I have also changed the fields in this structure so that they can be returned set to -1 to indicate no data / not supported. yeah, it's better to have both when you can because being able to detect change in average block size transfer is important too from a monitoring perspective. The updated stats structures are shown below. Rich. /* Block device stats for virDomainBlockStats. * * Hypervisors may return a field set to (int64_t)-1 which indicates * that the hypervisor does not support that statistic. */ struct _virDomainBlockStats { int64_t rd_req; int64_t rd_bytes; int64_t wr_req; int64_t wr_byes; wr_bytes :-) int64_t errs; // In Xen this returns the mysterious 'oo_req'. }; typedef struct _virDomainBlockStats *virDomainBlockStatsPtr; then yes that's okay. Those interfaces are what I suggested as the low level ones, I guess they are needed even if tehy don't really scale as the number of domain and more importantly the number of node increases. But it allows to build higher level monitoring implementations. My current POV based on previous monitoring work is that if you are monitoring up to a few dozen machines then aggregating the data to the monitoring application is fine, where you can then apply the user based policy to raise the events (and subsequent rules or UI alerts), but if you want to scale you have to export the monitoring down to each node, and push the policies there, just gathering the events/alerts at the monitoring application or console level. In any case having the low level API is needed so something like those entry points are needed. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] [PATCH] Block device and network stats
On Fri, Aug 10, 2007 at 03:35:08PM +0100, Richard W.M. Jones wrote: This is the implementation (currently Xen, local only). It includes a couple of virsh commands if you want to play with looking at the stats. The network interface stuff isn't quiet correct +/* Paths have the form virbr\d+ or xenbr\d+. On Linux we open + * /proc/net/dev and look for the device called vifdomid.n. + */ +int +xenHypervisorDomainInterfaceStats (virDomainPtr dom, + const char *path, + struct _virDomainInterfaceStats *stats) The virbrN / xenbrN devices are bridges so see all traffic from any guest attached. We need to look at ths individual vifN.N or vnetN devices which are per-guest. This info isn't included in the XML though. If we can figure out a efficient way to export vif name in the XML we could use that, otherwie the API should probably take the MAC address on the device and use that to resolve the underlying VIF. The former is nicer though if we can do it. That all said, it could be nice to also have a getStats method for the virNetworkPtr objects to get cummulative bridge data - the backend impl would be near identical. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Improve performance on Xen
On Fri, Aug 10, 2007 at 10:57:50AM +0100, Richard W.M. Jones wrote: Daniel P. Berrange wrote: The Xen implementations of virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece have to lookup the other piece. A pretty good speedup really considering how frequently these calls are made if you're monitoring domains frequently. +1 Rich. Ok, just commited this now. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Improve performance on Xen
On Fri, Aug 10, 2007 at 06:39:56AM -0400, Daniel Veillard wrote: On Thu, Aug 09, 2007 at 11:13:00PM +0100, Daniel P. Berrange wrote: The Xen implementations of virDomainLookupByID virDomainLookupByUUID virDomainLookupByName virDomainGetOSType all have sub-optimal performance since they speak to XenD. The lookupXXX functions all basically require 3 pieces of info in the end (name,id,uuid). They each get given one piece have to lookup the other piece. Every running domain has to have an entry in XenStore on /local/domain/N where 'N' is the id. Under this location there is always a 'name' field. So if we have the id we can efficiently get the name. I just realized that the getdomaininfo hypercall struct contains the uuid and id. So for the ByID and ByUUID calls we can get all the info we need with a hypercall and a read of xenstore. This is very efficient compared to hitting XenD. Great As of Xen 3.1.0 hypervisor the flags in the getdomaininfo hypercall struct also mark whether the guest is HVM vs paraivrt, so we can also implement the GetOSType api with a single hypercall. Sounds good to though definitely not a bottleneck. That just leaves the ByName impl. The only way I can think of doing this is to scan /local/domain/N in xenstore until we find it. I'm going to try this, but I'm not convinced it'll be any significantly than talking to XenD. I may be surprised though. Well if you have 100 guests, that may be slower, but in the average situation of only a couple of guests, it could be a real speedup. The problem is that a lot of domain may accumulate in xenstore /local/domain even if they are not running, both implementation are likely to have completely different behaviour based on the context. But from a cache locality perspective hitting xenstore may scale way better under loaded machines, so it may prove faster even on machines with hundreds of domains. Doing a fair performance comparison may prove really hard. Actually the /local/domain/[ID] subtree is guarenteed to only contain running VMs. The /vm/[UUID] subtree is where cruft accumulates over time, so its safe to rely on info in the former, but not the latter Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Fix virtual neworks with xm driver
On Fri, Aug 10, 2007 at 05:06:30PM +0100, Daniel P. Berrange wrote: On Fri, Aug 10, 2007 at 12:03:10PM -0400, Daniel Veillard wrote: On Fri, Aug 10, 2007 at 04:36:25PM +0100, Daniel P. Berrange wrote: The virtual networks stuff was never added to the xm driver for Xen 3.0.3 or earlier. This means that if adding a virtual network the bridge device won't get defined in the config. This patch addreses this, and also fixes a tiny mem leak in the equivalent code in the xend driver. Looks fine, I'm just surprised by @@ -1694,9 +1707,12 @@ static char *xenXMParseXMLVif(xmlNodePtr if (typ == 0) { strcat(buf, ,bridge=); strcat(buf, (const char*)source); -} else { -strcat(buf, ,mac=); +} else if (typ == 1) { +strcat(buf, ,dev=); strcat(buf, (const char*)source); +} else { +strcat(buf, ,bridge=); +strcat(buf, bridge); } } So we never emit the mac definition anymore ? The MAC is already being added further up - you just can't see it in the diff context. This hunk was just bogus. Okidoc, +1 :-) Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[Libvir] Proposal: Block device and network stats
With the very timely question that has been raised about block and network device stats, I'm posting my proposed interface. I almost have this working for the Xen case on my test machine. The idea of this interface is to be optimised for the following case: * stats need to be fetched frequently (eg. once per second) * the layout of devices doesn't change often (ie. adding or removing devices from domains is very infrequent) * most domains have only a single block and single network device With the above assumptions, the idea is that you would use the API like this: (1) When the program starts up or (infrequently) is notified of a change or when a new domain appears, the program calls virDomainGetXMLDesc and parses out the //domain/devices/interface and //domain/devices/disk fields to get the list of network interfaces and block devices. So for each domain, you'll have two lists like this: dom[1][blockdevs] = [xvda] dom[1][interfaces] = [virbr0] (2) Frequently (eg. once per second) the program calls (in the above case): virDomainBlockStats (dom1, xvda, bstats, sizeof bstats); virDomainInterfaceStats (dom1, virbr0, istats, sizeof istats); (3) Since stats are cumulative the program must do its own subtraction and calculation in order to display throughput per second. The implementation goes directly from the name xvda to the backend device (/sys/devices/xen-backend/- [type]-[domid]-[major:minor]/statistics) for block devices, and slightly more complicatedly to a particular line in /proc/net/dev for network interfaces. (Note in the Xen case, the name virbr0 is little more than a placeholder to mean the zeroth interface for domain d). In particular the current implementation doesn't cache anything. This should all work fine in the Linux / Xen case. libxenstat gives us sample code that we can copy for the Solaris / Xen case, but it would need testing from someone with access to such a machine. I don't think qemu supports stats at all. Initially we won't support stats for tap devices because there needs to be an upstream patch to support this (http://lists.xensource.com/archives/html/xen-changelog/2007-02/msg00278.html). The extra size parameter will allow us to extend the stats structures in future, maintaining binary backwards compatibility with existing clients. Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 +/* Block device stats for virDomainBlockStats. + */ +struct _virDomainBlockStats { + uint64_t rd_req; + uint64_t wr_req; + uint64_t errs; // In Xen this returns the mysterious 'oo_req'. +}; +typedef struct _virDomainBlockStats *virDomainBlockStatsPtr; + +/* Network interface stats for virDomainInterfaceStats. + */ +struct _virDomainInterfaceStats { + uint64_t rx_bytes; + uint64_t rx_packets; + uint64_t rx_errs; + uint64_t rx_drop; + uint64_t tx_bytes; + uint64_t tx_packets; + uint64_t tx_errs; + uint64_t tx_drop; +}; +typedef struct _virDomainInterfaceStats *virDomainInterfaceStatsPtr; + [...] +/** + * virDomainBlockStats + * @dom: pointer to the domain object + * @path: path to the block device + * @stats: block device stats (returned) + * @size: size of stats structure + * + * This function returns block device (disk) stats for block + * devices attached to the domain. + * + * The path parameter is the name of the block device. Get this + * by calling virDomainGetXMLDesc and finding the target dev='...' + * attribute within //domain/devices/disk. (For example, xvda). + * + * Domains may have more than one block device. To get stats for + * each you should make multiple calls to this function. + * + * Returns: 0 in case of success or -1 in case of failure. + */ +int +virDomainBlockStats (virDomainPtr dom, const char *path, + virDomainBlockStatsPtr stats, size_t size) +{ [...] +} +/** + * virDomainInterfaceStats + * @dom: pointer to the domain object + * @path: path to the interface + * @stats: network interface stats (returned) + * @size: size of stats structure + * + * This function returns network interface stats for interfaces + * attached to the domain. + * + * The path parameter is the name of the network interface. Get + * this by calling virDomainGetXMLDesc and finding the + * source ... attribute within //domain/devices/interface. + * (For example, xenbr0 or virbr0). + * + * Domains may have more than network interface. To get stats for + * each you should make multiple calls to this function. + * + * Returns: 0 in case of success or -1 in case of failure. + */ +int +virDomainInterfaceStats (virDomainPtr dom, const char *path, + virDomainInterfaceStatsPtr stats, size_t size) +{ [...] +} smime.p7s Description: S/MIME Cryptographic Signature --
Re: [Libvir] Proposal: Block device and network stats
It was suggested to me that we should provide a way to return bytes read and written on block devices (not just requests). Xen doesn't support that however, so I have also changed the fields in this structure so that they can be returned set to -1 to indicate no data / not supported. The updated stats structures are shown below. Rich. /* Block device stats for virDomainBlockStats. * * Hypervisors may return a field set to (int64_t)-1 which indicates * that the hypervisor does not support that statistic. */ struct _virDomainBlockStats { int64_t rd_req; int64_t rd_bytes; int64_t wr_req; int64_t wr_byes; int64_t errs; // In Xen this returns the mysterious 'oo_req'. }; typedef struct _virDomainBlockStats *virDomainBlockStatsPtr; /* Network interface stats for virDomainInterfaceStats. * * Hypervisors may return a field set to (int64_t)-1 which indicates * that the hypervisor does not support that statistic. */ struct _virDomainInterfaceStats { int64_t rx_bytes; int64_t rx_packets; int64_t rx_errs; int64_t rx_drop; int64_t tx_bytes; int64_t tx_packets; int64_t tx_errs; int64_t tx_drop; }; typedef struct _virDomainInterfaceStats *virDomainInterfaceStatsPtr; -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] PATCH: Fix paravirt FB with xm driver
On Fri, Aug 10, 2007 at 12:08:26PM -0400, Daniel Veillard wrote: On Fri, Aug 10, 2007 at 04:39:03PM +0100, Daniel P. Berrange wrote: The code for dealing with the paravirt framebuffer was missing code to deal with an explicit vncdisplay=N setting - only working for vncunused=1. It was also not flexible enough in processing the 2 different styles of FB config. I'm not sure I undertand the logic for setting the port, why going whould xenstore there ? I'm not sure what you mean - xenstore isn't involved in this. This is for taking the XML and writing out the /etc/xen config - the port stuff is basically making sure that port=5903 is translated into the config option 'vncdisplay=3'. Previously only port=-1 was working. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list