Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On 14/07/14 16:25, Johannes Erdfelt wrote: > On Mon, Jul 14, 2014, Daniel P. Berrange wrote: >> I think that I'd probably say there is an expectation that the rescue >> image will be different from the primary image the OS was booted from. > > So every image would now need a corresponding rescue image? My original comment was on the current state of affairs. If re-using the existing image as a rescue image works for you, there's no reason to change that. However, I observed in my testing that it's not a terribly good idea for the reasons I outlined. If you want to move away from that, though, I believe you could create a generic rescue image which would be good for most/all Linux instances at the very least. In fact, there are plenty of examples of generic rescue images out there already. Matt -- Matthew Booth Red Hat Engineering, Virtualisation Team Phone: +442070094448 (UK) GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On Mon, Jul 14, 2014, Daniel P. Berrange wrote: > I think that I'd probably say there is an expectation that the rescue > image will be different from the primary image the OS was booted from. So every image would now need a corresponding rescue image? JE ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On Mon, Jul 14, 2014 at 10:48:17AM +0100, Matthew Booth wrote: > > That's interesting. I didn't realise that other drivers had the same > limitations. Does anybody understand the original thinking which lead to > this design? The single VM approach seems intuitively correct to me, so > presumably at some point there was a good reason not to choose it. I imagine it was just done for ease of implementation initially, and/or the rescue mode hasn't been updated as new features were added, so it fell behind. > I'm not sufficiently familiar with the libvirt driver to know what state > persists in the hypervisor, but in the vmware driver an attached volume > remains attached until detached. By re-using the original VM for rescue, > we wouldn't have to concern ourselves with volumes at all, because they > are all already attached. > > As for following the linked BP, I *think* we would achieve that 'for > free' with a single VM approach. Is there any semantic subtlety here > relating to actually attaching the volumes? i.e. Does it matter that we > wouldn't attach them, because they are already attached? > > I like the USB idea, and I'm fairly sure it should be achievable in the > VMware driver. Is it worth codifying it in a BP? Perhaps the same BP. It is in a gray area, you could probably argue both ways as to whether it could be slipped in as part of the existing BP :-) It has user visible change though, so might need a bit of bikeshedding in a BP to agree on best way to deal with the idea. eg perhaps opt-in to use of USB vs existing disk approach via an image property. > Incidentally, I hit an obvious problem when testing this with the Cirros > image, which mounts filesystems by label. If you have an image which > mounts by label, or has LVM volumes, and you use the same image as a > rescue disk, you are adding a second disk containing the same filesystem > labels and/or LVM volumes. Under these circumstances, the behaviour of > mount during the boot sequence is not well defined afaik. Consequently, > re-using the original image as a rescue image doesn't sound to me like a > good idea in general. I think that I'd probably say there is an expectation that the rescue image will be different from the primary image the OS was booted from. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On 11/07/14 12:36, Daniel P. Berrange wrote: > On Fri, Jul 11, 2014 at 12:30:19PM +0100, John Garbutt wrote: >> On 10 July 2014 16:52, Matthew Booth wrote: >>> Currently we create a rescue instance by creating a new VM with the >>> original instance's image, then adding the original instance's first >>> disk to it, and booting. This means we have 2 VMs, which we need to be >>> careful of when cleaning up. Also when suspending, and probably other >>> edge cases. We also don't support: >>> >>> * Rescue images other than the instance's creation image >>> * Rescue of an instance which wasn't created from an image >>> * Access to cinder volumes from a rescue instance >>> >>> I've created a dirty hack which, instead of creating a new VM, attaches >>> the given rescue image to the VM and boots from it: >>> >>> https://review.openstack.org/#/c/106078/ >> >> I do worry about different drivers having such radically different >> implementation approaches. >> >> Currently rescue only attaches the root disk to the rescue image. >> Having a separate VM does side step having to work out where to >> reattach all the disks when you boot up the original VM, as you >> haven't modified that. But there are plans to change that here: >> http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/rescue-attach-all-disks.rst >> >> You can now specify and image when you go into rescue mode: >> http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/allow-image-to-be-specified-during-rescue.rst >> >> I guess the rescue image could technically change how the VM boots, or >> what hardware it has attached, so you might end up making so many >> "tweaks" to the original VM that you might want to just create a new >> VM, then through way those changes when you restore the original VM. >> >> It feels a lot like we need to better understand the use cases for >> this feature, and work out what we need in the long term. >> >>> Does this seem a reasonable way to go? >> >> Maybe, but I am not totally keen on making it such a different >> implementation to all the other drivers. Mostly for the sake of people >> who might run two hypervisors in their cloud, or people who support >> customers running various hypervisors. > > My view is that rescue mode should have as few differences from > normal mode as possible. Ideally the exact same VM configuration > would be used, with the exception that you add in one extra disk > and set the BIOS to boot of that new disk. The spec you mention > above gets us closer to that in libvirt, but it still has the > problem that it re-shuffles the disk order. To fix this I think > we need to change the rescue image disk so that isntead of being > a virtio-blk or IDE disk, it is a hotplugged USB disk and make > the BIOS boot from this USB disk. That way none of the existing > disk attachments will change in any way. This would also feel > more like the way a physical machine would be rescued where you > would typically insert a bootable CDROM or a rescue USB stick > > So in that sense I think that Matt suggests for VMWare is good > because it gets the vmware driver moving in the right direction. > I'd encourage them to also follow that libvirt blueprint and > ensure all disks are attached. That's interesting. I didn't realise that other drivers had the same limitations. Does anybody understand the original thinking which lead to this design? The single VM approach seems intuitively correct to me, so presumably at some point there was a good reason not to choose it. I'm not sufficiently familiar with the libvirt driver to know what state persists in the hypervisor, but in the vmware driver an attached volume remains attached until detached. By re-using the original VM for rescue, we wouldn't have to concern ourselves with volumes at all, because they are all already attached. As for following the linked BP, I *think* we would achieve that 'for free' with a single VM approach. Is there any semantic subtlety here relating to actually attaching the volumes? i.e. Does it matter that we wouldn't attach them, because they are already attached? I like the USB idea, and I'm fairly sure it should be achievable in the VMware driver. Is it worth codifying it in a BP? Perhaps the same BP. Incidentally, I hit an obvious problem when testing this with the Cirros image, which mounts filesystems by label. If you have an image which mounts by label, or has LVM volumes, and you use the same image as a rescue disk, you are adding a second disk containing the same filesystem labels and/or LVM volumes. Under these circumstances, the behaviour of mount during the boot sequence is not well defined afaik. Consequently, re-using the original image as a rescue image doesn't sound to me like a good idea in general. Matt -- Matthew Booth Red Hat Engineering, Virtualisation Team Phone: +442070094448 (UK) GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 __
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On Fri, Jul 11, 2014 at 12:30:19PM +0100, John Garbutt wrote: > On 10 July 2014 16:52, Matthew Booth wrote: > > Currently we create a rescue instance by creating a new VM with the > > original instance's image, then adding the original instance's first > > disk to it, and booting. This means we have 2 VMs, which we need to be > > careful of when cleaning up. Also when suspending, and probably other > > edge cases. We also don't support: > > > > * Rescue images other than the instance's creation image > > * Rescue of an instance which wasn't created from an image > > * Access to cinder volumes from a rescue instance > > > > I've created a dirty hack which, instead of creating a new VM, attaches > > the given rescue image to the VM and boots from it: > > > > https://review.openstack.org/#/c/106078/ > > I do worry about different drivers having such radically different > implementation approaches. > > Currently rescue only attaches the root disk to the rescue image. > Having a separate VM does side step having to work out where to > reattach all the disks when you boot up the original VM, as you > haven't modified that. But there are plans to change that here: > http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/rescue-attach-all-disks.rst > > You can now specify and image when you go into rescue mode: > http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/allow-image-to-be-specified-during-rescue.rst > > I guess the rescue image could technically change how the VM boots, or > what hardware it has attached, so you might end up making so many > "tweaks" to the original VM that you might want to just create a new > VM, then through way those changes when you restore the original VM. > > It feels a lot like we need to better understand the use cases for > this feature, and work out what we need in the long term. > > > Does this seem a reasonable way to go? > > Maybe, but I am not totally keen on making it such a different > implementation to all the other drivers. Mostly for the sake of people > who might run two hypervisors in their cloud, or people who support > customers running various hypervisors. My view is that rescue mode should have as few differences from normal mode as possible. Ideally the exact same VM configuration would be used, with the exception that you add in one extra disk and set the BIOS to boot of that new disk. The spec you mention above gets us closer to that in libvirt, but it still has the problem that it re-shuffles the disk order. To fix this I think we need to change the rescue image disk so that isntead of being a virtio-blk or IDE disk, it is a hotplugged USB disk and make the BIOS boot from this USB disk. That way none of the existing disk attachments will change in any way. This would also feel more like the way a physical machine would be rescued where you would typically insert a bootable CDROM or a rescue USB stick So in that sense I think that Matt suggests for VMWare is good because it gets the vmware driver moving in the right direction. I'd encourage them to also follow that libvirt blueprint and ensure all disks are attached. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
On 10 July 2014 16:52, Matthew Booth wrote: > Currently we create a rescue instance by creating a new VM with the > original instance's image, then adding the original instance's first > disk to it, and booting. This means we have 2 VMs, which we need to be > careful of when cleaning up. Also when suspending, and probably other > edge cases. We also don't support: > > * Rescue images other than the instance's creation image > * Rescue of an instance which wasn't created from an image > * Access to cinder volumes from a rescue instance > > I've created a dirty hack which, instead of creating a new VM, attaches > the given rescue image to the VM and boots from it: > > https://review.openstack.org/#/c/106078/ I do worry about different drivers having such radically different implementation approaches. Currently rescue only attaches the root disk to the rescue image. Having a separate VM does side step having to work out where to reattach all the disks when you boot up the original VM, as you haven't modified that. But there are plans to change that here: http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/rescue-attach-all-disks.rst You can now specify and image when you go into rescue mode: http://git.openstack.org/cgit/openstack/nova-specs/tree/specs/juno/allow-image-to-be-specified-during-rescue.rst I guess the rescue image could technically change how the VM boots, or what hardware it has attached, so you might end up making so many "tweaks" to the original VM that you might want to just create a new VM, then through way those changes when you restore the original VM. It feels a lot like we need to better understand the use cases for this feature, and work out what we need in the long term. > Does this seem a reasonable way to go? Maybe, but I am not totally keen on making it such a different implementation to all the other drivers. Mostly for the sake of people who might run two hypervisors in their cloud, or people who support customers running various hypervisors. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
Definitely +1. This change will allow us to get rid of the of the ugly special cases where we use the instance name instead of the uuid and will make the code cleaner. Let's go for it! Thanks, Rado - Original Message - > Currently we create a rescue instance by creating a new VM with the > original instance's image, then adding the original instance's first > disk to it, and booting. This means we have 2 VMs, which we need to be > careful of when cleaning up. Also when suspending, and probably other > edge cases. We also don't support: > > * Rescue images other than the instance's creation image > * Rescue of an instance which wasn't created from an image > * Access to cinder volumes from a rescue instance > > I've created a dirty hack which, instead of creating a new VM, attaches > the given rescue image to the VM and boots from it: > > https://review.openstack.org/#/c/106078/ > > It works for me. It supports all of the above, doesn't require special > handling on destroy, and works with suspend[1]. It also doesn't trigger > the spurious warning message about unknown VMs on the cluster which, > while unimportant in itself, is an example of an edge case caused by > having 2 VMs. > > Does this seem a reasonable way to go? It would be dependent on a > refactoring of the image cache code so we could cache the rescue image. > > Matt > > [1] If suspend of a rescued image wasn't broken at the api level, > anyway. I have a patch for that: https://review.openstack.org/#/c/106082/ > -- > Matthew Booth > Red Hat Engineering, Virtualisation Team > > Phone: +442070094448 (UK) > GPG ID: D33C3490 > GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
Nice idea. I say Œship it¹ The code was legacy and we just made sure it worked. Nice proposal On 7/10/14, 6:52 PM, "Matthew Booth" wrote: >Currently we create a rescue instance by creating a new VM with the >original instance's image, then adding the original instance's first >disk to it, and booting. This means we have 2 VMs, which we need to be >careful of when cleaning up. Also when suspending, and probably other >edge cases. We also don't support: > >* Rescue images other than the instance's creation image >* Rescue of an instance which wasn't created from an image >* Access to cinder volumes from a rescue instance > >I've created a dirty hack which, instead of creating a new VM, attaches >the given rescue image to the VM and boots from it: > >https://review.openstack.org/#/c/106078/ > >It works for me. It supports all of the above, doesn't require special >handling on destroy, and works with suspend[1]. It also doesn't trigger >the spurious warning message about unknown VMs on the cluster which, >while unimportant in itself, is an example of an edge case caused by >having 2 VMs. > >Does this seem a reasonable way to go? It would be dependent on a >refactoring of the image cache code so we could cache the rescue image. > >Matt > >[1] If suspend of a rescued image wasn't broken at the api level, >anyway. I have a patch for that: https://review.openstack.org/#/c/106082/ >-- >Matthew Booth >Red Hat Engineering, Virtualisation Team > >Phone: +442070094448 (UK) >GPG ID: D33C3490 >GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova][vmware] Convert to rescue by adding the rescue image and booting from it
Currently we create a rescue instance by creating a new VM with the original instance's image, then adding the original instance's first disk to it, and booting. This means we have 2 VMs, which we need to be careful of when cleaning up. Also when suspending, and probably other edge cases. We also don't support: * Rescue images other than the instance's creation image * Rescue of an instance which wasn't created from an image * Access to cinder volumes from a rescue instance I've created a dirty hack which, instead of creating a new VM, attaches the given rescue image to the VM and boots from it: https://review.openstack.org/#/c/106078/ It works for me. It supports all of the above, doesn't require special handling on destroy, and works with suspend[1]. It also doesn't trigger the spurious warning message about unknown VMs on the cluster which, while unimportant in itself, is an example of an edge case caused by having 2 VMs. Does this seem a reasonable way to go? It would be dependent on a refactoring of the image cache code so we could cache the rescue image. Matt [1] If suspend of a rescued image wasn't broken at the api level, anyway. I have a patch for that: https://review.openstack.org/#/c/106082/ -- Matthew Booth Red Hat Engineering, Virtualisation Team Phone: +442070094448 (UK) GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
