Re: [Libvir] save/restore support for KVM
On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: On Thu, Aug 09, 2007 at 05:26:43PM -0400, Jim Paris wrote: Hi, I've implemented save/restore support for KVM using the live migration feature. First, a few patches to KVM to fix bugs: Very cool :-) Then, another patch to KVM to support inbound migration from a filename. It already supports migration from stdin, but adding this seemed easier from a libvirt perspective. Subject: [PATCH] qemu: accept filename for incoming migration http://article.gmane.org/gmane.comp.emulators.kvm.devel/5590 Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. Finally, the libvirt side. Some notes: - Arbitrary decisions about VM status: I pause the VM before starting migration, and destroy it once it's finished. Neither is required by KVM but I'd be concerned about the disk image state otherwise. Also, after resuming, the VM is still paused. I don't know how Xen behaves in this regard but the behavior here is easy enough to change. Xen doesn't mind whether the VM is running or paused when saving it. Pausing it seems reasonable - its going to be stopped shortly anyway, so letting it continue running just increases the saved image size by having to process dirtied memory. As for restore, I'd be inclined to leave it running after restore to match our Xen driver. I must admit I feel more comfortable with pausing, reduces the complexity of the operation and could avoid nasty bugs near impossible to track. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ [EMAIL PROTECTED] | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
On Fri, Aug 10, 2007 at 12:36:05PM -0400, Jim Paris wrote: Daniel Veillard wrote: On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. I'm not sure the kvm qemu binary even reports the kvm version anywhere. I'll ask on kvm-devel to see if qemu/VERSION could get updated with each KVM release. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. I think appending unrelated data to the migration image is a bit of a hack anyway. A better plan would be a file containing header XML config migration data On save, libvirt writes header and XML config, then closes it and uses dd of=path oflag=append conv=notrunc or just cat path as the migration command. On restore, libvirt reads the header and XML config, and then feeds the remaining migration data to KVM using -incoming stdio. I had wanted to avoid the trouble of feeding data via stdin, but maybe a well placed dup2(fd,STDIN_FILENO) would do the trick automatically. That sounds like a reasonable plan This file format would also make it easier for e.g. virt-manager to determine that a file is a valid libvirt restore image. A little magic in the first couple of bytes would be handy - thats what we check currently for Xen. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Richard W.M. Jones wrote:
Jim Paris wrote:
+if (strchr(path, '\'') || strchr(path, '\\') ) {
+qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED,
+ invalid filename);
+return -1;
+}
[...]
+/* Migrate to file. */
+if (asprintf (command, migrate \exec:dd of='%s' 2/dev/null\\n,
+ path) == -1) {
+qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED,
+ out of memory);
+return -1;
+}
The patch is fine, except I'm wondering whether the quoting above is
safe. We check if the path contains ' or \ and refuse to proceed. I
_think_ you don't need to check for \ however
I think you're right. An even better fix would be to explicitly
escape bad characters in the path before passing them along. Giving
an error on the filename Jim's VM as it would do right now isn't
ideal.
-jim
--
Libvir-list mailing list
Libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Daniel Veillard wrote: On Thu, Aug 09, 2007 at 10:55:10PM +0100, Daniel P. Berrange wrote: Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Would checking for the kvm version from the console sufficient ? Since KVM makes even more releases than libvirt in average I guess that would be fine. I'm not sure the kvm qemu binary even reports the kvm version anywhere. I'll ask on kvm-devel to see if qemu/VERSION could get updated with each KVM release. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. +1 . The only problem is that the XML has no predefined size, so it may be hard to stack more stuff behind it. I would ask first on the KVM list to check if it's okay to add a variable lenght data structure at the end, they might want to extend it in the future and that would be hard to handle. I think appending unrelated data to the migration image is a bit of a hack anyway. A better plan would be a file containing header XML config migration data On save, libvirt writes header and XML config, then closes it and uses dd of=path oflag=append conv=notrunc or just cat path as the migration command. On restore, libvirt reads the header and XML config, and then feeds the remaining migration data to KVM using -incoming stdio. I had wanted to avoid the trouble of feeding data via stdin, but maybe a well placed dup2(fd,STDIN_FILENO) would do the trick automatically. This file format would also make it easier for e.g. virt-manager to determine that a file is a valid libvirt restore image. -jim -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Daniel P. Berrange wrote: - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. How does this (or even _does_ this) work in the Xen case? Does Xen save this info like UUID, name, etc. in the migration image? Rich. -- Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/ Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in England and Wales under Company Registration No. 03798903 smime.p7s Description: S/MIME Cryptographic Signature -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
Jim Paris wrote:
+if (strchr(path, '\'') || strchr(path, '\\') ) {
+qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED,
+ invalid filename);
+return -1;
+}
[...]
+/* Migrate to file. */
+if (asprintf (command, migrate \exec:dd of='%s' 2/dev/null\\n,
+ path) == -1) {
+qemudReportError(dom-conn, dom, NULL, VIR_ERR_OPERATION_FAILED,
+ out of memory);
+return -1;
+}
The patch is fine, except I'm wondering whether the quoting above is
safe. We check if the path contains ' or \ and refuse to proceed. I
_think_ you don't need to check for \ however, according to this section
from the bash manual page and my testing:
Enclosing characters in single quotes preserves the
literal value of each character within the quotes.
A single quote may not occur between single quotes,
even when preceded by a backslash.
Perhaps it is better to be safe than sorry though.
Rich.
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
smime.p7s
Description: S/MIME Cryptographic Signature
--
Libvir-list mailing list
Libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] save/restore support for KVM
On Thu, Aug 09, 2007 at 05:26:43PM -0400, Jim Paris wrote: Hi, I've implemented save/restore support for KVM using the live migration feature. First, a few patches to KVM to fix bugs: Subject: [PATCH 1/3] qemu: fix freed pointer dereference http://article.gmane.org/gmane.comp.emulators.kvm.devel/5572 Subject: [PATCH 2/3] qemu: don't start a new migration if one is already in progress http://article.gmane.org/gmane.comp.emulators.kvm.devel/5575 Subject: [PATCH 3/3] qemu: reset buffer pointers after CR/LF http://article.gmane.org/gmane.comp.emulators.kvm.devel/5573 (If compatibility with old KVM is wanted, it might be possible to work around the kvm bugs in other ways, but I'm not sure). These are all genuine bugs so its not unreasonable to require that anyone using save/restore ensure they're patched. Then, another patch to KVM to support inbound migration from a filename. It already supports migration from stdin, but adding this seemed easier from a libvirt perspective. Subject: [PATCH] qemu: accept filename for incoming migration http://article.gmane.org/gmane.comp.emulators.kvm.devel/5590 Just been committed to KVM repos I see. Should be an easy patch to backport too. As long as we can detect failure if this is missing report it back then I'm fine depending on this. Finally, the libvirt side. Some notes: - Arbitrary decisions about VM status: I pause the VM before starting migration, and destroy it once it's finished. Neither is required by KVM but I'd be concerned about the disk image state otherwise. Also, after resuming, the VM is still paused. I don't know how Xen behaves in this regard but the behavior here is easy enough to change. Xen doesn't mind whether the VM is running or paused when saving it. Pausing it seems reasonable - its going to be stopped shortly anyway, so letting it continue running just increases the saved image size by having to process dirtied memory. As for restore, I'd be inclined to leave it running after restore to match our Xen driver. - I append the domain's UUID at the end of the migration image. This doesn't affect KVM at all (it ignores the extra data). Does that seem reasonable? It's unclear how the saved image is supposed to get associated with a particular VM configuration without doing something like this. Actually I'd store the entire XML config appended to the end of the image. Its quite possible the saved image may be restored on a different machine so libvirt will need the XML config there its not much work to automatically append it all use it when restoring later. - I added the migrateFrom field to the qemu_vm struct. Dunno if that's the best place. Could also add put it in qemu_vm_def, or add parameters to qemudStartVMDaemon, etc.. Seems reasonable place for now - its not really permanent data so the main 'qemu_vm' struct is more appropriate than 'qemu_vm_def'. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
