Re: [openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend

[Jon Bernard]

* melanie witt  wrote:
> On Fri, 19 Oct 2018 23:21:01 +0800 (GMT+08:00), Boxiang Zhu wrote:
> > 
> > The version of my cinder and nova is Rocky. The scope of the cinder spec[1]
> > is only for available volume migration between two pools from the same
> > ceph cluster.
> > If the volume is in-use status[2], it will call the generic migration
> > function. So that as you
> > describe it, on the nova side, it raises NotImplementedError(_("Swap
> > only supports host devices").
> > The get_config of net volume[3] has not source_path.
> 
> Ah, OK, so you're trying to migrate a volume across two separate ceph
> clusters, and that is not supported.
> 
> > So does anyone try to succeed to migrate volume(in-use) with ceph
> > backend or is anyone doing something of it?
> 
> Hopefully someone can share their experience with trying to migrate volumes
> across separate ceph clusters. I unfortunately don't know anything about it.

If this is the case, then Cinder cannot request a storage-specific
migration which is typically more efficient.  The migration will require
a complete copy of each allocated block.  Whether the volume is attached
or not will determine who (cinder or nova) will perform the operation.

-- 
Jon

> 
> Best,
> -melanie
> 
> > [1] https://review.openstack.org/#/c/296150
> > [2] https://review.openstack.org/#/c/256091/23/cinder/volume/drivers/rbd.py
> > [3] 
> > https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/volume/net.py#L101
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] [nova] Problem of Volume(in-use) Live Migration with ceph backend

[Jon Bernard]

* melanie witt  wrote:
> On Mon, 22 Oct 2018 11:45:55 +0800 (GMT+08:00), Boxiang Zhu wrote:
> > I created a new vm and a new volume with type 'ceph'[So that the volume
> > will be created on one of two hosts. I assume that the volume created on
> > host dev@rbd-1#ceph this time]. Next step is to attach the volume to the
> > vm. At last I want to migrate the volume from host dev@rbd-1#ceph to
> > host dev@rbd-2#ceph, but it failed with the exception
> > 'NotImplementedError(_("Swap only supports host devices")'.
> > 
> > So that, my real problem is that is there any work to migrate
> > volume(*in-use*)(*ceph rbd*) from one host(pool) to another host(pool)
> > in the same ceph cluster?
> > The difference between the spec[2] with my scope is only one is
> > *available*(the spec) and another is *in-use*(my scope).
> > 
> > 
> > [1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/
> > [2] https://review.openstack.org/#/c/296150
> 
> Ah, I think I understand now, thank you for providing all of those details.
> And I think you explained it in your first email, that cinder supports
> migration of ceph volumes if they are 'available' but not if they are
> 'in-use'. Apologies that I didn't get your meaning the first time.
> 
> I see now the code you were referring to is this [3]:
> 
> if volume.status not in ('available', 'retyping', 'maintenance'):
> LOG.debug('Only available volumes can be migrated using backend '
>   'assisted migration. Falling back to generic migration.')
> return refuse_to_migrate
> 
> So because your volume is not 'available', 'retyping', or 'maintenance',
> it's falling back to generic migration, which will end up with an error in
> nova because the source_path is not set in the volume config.
> 
> Can anyone from the cinder team chime in about whether the ceph volume
> migration could be expanded to allow migration of 'in-use' volumes? Is there
> a reason not to allow migration of 'in-use' volumes?

Generally speaking, Nova must facilitate the migration of a live (or
in-use) volume.  A volume attached to a running instance requires code
in the I/O path to correctly route traffic to the correct location - so
Cinder must refuse (or defer) a migrate operation if the volume is
attached.  Until somewhat recently Qemu and Libvirt did not support the
migration to non-block (RBD) targets which is the reason for lack of
support.  I believe we now have all of the pieces to perform this
operation successfully, but I suspect it will require a setup with
correct versions of all the related software.  I will try to verify this
during the current release cycle and report back.

-- 
Jon

> 
> [3] 
> https://github.com/openstack/cinder/blob/c42fdc470223d27850627fd4fc9d8cb15f2941f8/cinder/volume/drivers/rbd.py#L1618-L1621
> 
> Cheers,
> -melanie
> 
> 
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Required Ceph rbd image features

[Jon Bernard]

* Chris MacNaughton  wrote:
> Hello,
> 
> I'm working on identifying the required RBD image features to be compatible
> with both the nova-kvm and nova-lxd drivers. The requirement derives from
> nova-lxd using the kernel driver for Ceph, while nova-kvm handles rbd
> through userspace.

I believe kernel rbd supports image format 2 since linux ~3.11.  There
was a time when striping feature was not supported in kernel rbd, I'm
not sure if that's still the case today but should be easy to test for.
I suspect you'd want to enable layering at minimum.

-- 
Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder] critical fix for ceph job

[Jon Bernard]

* Matt Riedemann  wrote:
> On 4/7/2017 9:43 AM, Jordan Pittier wrote:
> > 
> > 
> > On Fri, Apr 7, 2017 at 4:15 PM, Ghanshyam Mann  > > wrote:
> > 
> > Thanks. I am not sure these kind of driver specific behavior on APIs
> > side. This bring up question that should not cinder APIs be consistent
> > from usage point. In this case[1], create backup API can accept
> > 'container' param and do/don't create pool as per configured driver?
> > Then have better documentation for that what all driver honor that or
> > not.
> > 
> > Any suggestion ?
> > 
> > ..1 https://review.openstack.org/#/c/454321/3
> > 
> > 
> > 
> > Yeah, I've left a comment on that review. And another comment on
> > https://review.openstack.org/#/c/454722 :
> > 
> > "I'd rather we revert the change completely than to see this merged.
> > 
> > If the Ceph backup driver doesn't support the container argument it
> > should either grow support for it, or ignore that argument, or we change
> > Cinder's API completely so that the container argument is not part of
> > the public API anymore.
> > 
> > Do we expect each and every user to know what each and every drivers
> > support ? I don"t think so, so Tempest shouldn"t either."
> > 
> > 
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> I left a comment in there too. This wasn't the right way to get around this.
> I've tried the same thing before, back when the encrypted volume tests were
> failing for ceph because it simply wasn't supported in nova.
> 
> Jon, as we discussed at the PTG, you need to get a whitelist or blacklist
> file into nova like we have for the cells v1 job and we can use that on the
> ceph job config to control the tests that are run, so we don't need to make
> changes like this in Tempest. Let's work on that and then we can revert this
> workaround to Tempest.

Ok, I understand the logic and I'm happy work towards this.  For
reference, this commit https://review.openstack.org/#/c/345411/ added
support for container names to the ceph backup driver and I think a
discussion within Cinder is needed.  I will first create an analogous
patch for nova's whitelist, and then revert this one.  And if we decide
to change cinder's behaviour then all of it can go away.

-- 
Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][cinder] critical fix for ceph job

[Jon Bernard]

I just posted a critical fix the ceph job [1].  I'm anxious to get this
landed so I can start working on the other intermittent failures in the
gate.  Any feedback is much appreciated.

[1] https://review.openstack.org/#/c/454321/

-- 
Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Does the OpenStack community(or Cinder team) allow one driver to call another driver's public method?

[Jon Bernard]

* liuxinguo  wrote:
> Hi Cinder team,
> 
> We are going to implement storage-assisted volume migrate in our
> driver between different backend storage array or even different array
> of different vendors.  This is really high-efficiency than the
> host-copy migration between different array of different vendors.

Could you elaborate more on this?  During a volume migration operation
we give the driver an opportunity to more-intelligently relocate the
volume's data.  This is done through the migrate_volume() method defined
in the driver itself.  If this method exists, it will be called before
falling back to a byte-for-byte copy approach - and if it succeeds the
volume is considered migrated and the operation returns.

Is this what you were looking for, or did you have something different
in mind?

-- 
Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] OPENSTACK with CEPH storage backend cant down size an instance

[Jon Bernard]

* James Galvin jgal...@servecentric.com wrote:
 Hi All
 
 I am having some trouble with down sizing an instance,
 
 I can resize the instance from say small flavour to medium flavour but
 when trying to resize the instance back from medium to small
 
 I get the following :
 
 Error: Failed to perform requested operation on instance jg-10, the
 instance has an error status: Please try again later [Error: Flavor's
 disk is too small for requested image.].
 
 I am using ceph as the storage backend clustered over 3 nodes with 3
 pools volumes vms images

Someone will correct me if I'm wrong, but I believe we do not support
reducing the size of a volume, as this would require an FS resize.
Shrinking a filesystem will often leave you with a suboptimal on-disk
layout.  I believe this is a general openstack rule, and not just
specific to RBD volumes.

-- 
Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* John Garbutt j...@johngarbutt.com wrote:
 On 24 January 2014 17:05, Jon Bernard jbern...@tuxion.com wrote:
  * Vishvananda Ishaya vishvana...@gmail.com wrote:
 
  On Jan 16, 2014, at 1:28 PM, Jon Bernard jbern...@tuxion.com wrote:
 
   * Vishvananda Ishaya vishvana...@gmail.com wrote:
  
   On Jan 14, 2014, at 2:10 PM, Jon Bernard jbern...@tuxion.com wrote:
  
  
   snip
   As you've defined the feature so far, it seems like most of it could
   be implemented client side:
  
   * pause the instance
   * snapshot the instance
   * snapshot any attached volumes
  
   For the first milestone to offer crash-consistent snapshots you are
   correct.  We'll need some additional support from libvirt, but the
   patchset should be straightforward.  The biggest question I have
   surrounding initial work is whether to use an existing API call or
   create a new one.
  
  
   I think you might have missed the client side part of this point. I 
   agree
   that the snapshot multiple volumes and package it up is valuable, but I 
   was
   trying to make the point that you could do all of this stuff client side
   if you just add support for snapshotting ephemeral drives. An all-in-one
   snapshot command could be valuable, but you are talking about 
   orchestrating
   a lot of commands between nova, glance, and cinder and it could get kind
   of messy to try to run the whole thing from nova.
  
   If you expose each primitive required, then yes, the client could
   implement the logic to call each primitive in the correct order, handle
   error conditions, and exit while leaving everything in the correct
   state.  But that would mean you would have to implement it twice - once
   in python-novaclient and once in Horizon.  I would speculate that doing
   this on the client would be even messier.
  
   If you are concerned about the complexity of the required interactions,
   we could narrow the focus in this way:
  
Let's say that taking a full snapshot/backup (all volumes) operates
only on persistent storage volumes.  Users who booted from an
ephemeral glance image shouldn't expect this feature because, by
definition, the boot volume is not expected to live a long life.
  
   This should limit the communication to Nova and Cinder, while leaving
   Glance out (initially).  If the user booted an instance from a cinder
   volume, then we have all the volumes necessary to create an OVA and
   import to Glance as a final step.  If the boot volume is an image then
   I'm not sure, we could go in a few directions:
  
1. No OVA is imported due to lack of boot volume
2. A copy of the original image is included as a boot volume to create
   an OVA.
3. Something else I am failing to see.
 
  
   If [2] seems plausible, then it probably makes sense to just ask glance
   for an image snapshot from nova while the guest is in a paused state.
  
   Thoughts?
 
  This already exists. If you run a snapshot command on a volume backed 
  instance
  it snapshots all attached volumes. Additionally it does throw a bootable 
  image
  into glance referring to all of the snapshots.  You could modify create 
  image
  to do this for regular instances as well, specifying block device mapping 
  but
  keeping the vda as an image. It could even do the same thing with the 
  ephemeral
  disk without a ton of work. Keeping this all as one command makes a lot of 
  sense
  except that it is unexpected.
 
  There is a benefit to only snapshotting the root drive sometimes because it
  keeps the image small. Here's what I see as the ideal end state:
 
  Two commands(names are a strawman):
create-full-image -- image all drives
create-root-image -- image just the root drive
 
  These should work the same regardless of whether the root drive is volume 
  backed
  instead of the craziness we have to day of volume-backed snapshotting all 
  drives
  and instance backed just the root.  I'm not sure how we manage 
  expectations based
  on the current implementation but perhaps the best idea is just adding 
  this in
  v3 with new names?
 
  FYI the whole OVA thing seems moot since we already have a way of 
  representing
  multiple drives in glance via block_device_mapping properites.
 
  I've had some time to look closer at nova and rethink things a bit and
  I see what you're saying.  You are correct, taking snapshots of attached
  volumes is currently supported - although not in the way that I would
  like to see.  And this is where I think we can improve.
 
  Let me first summarize my understanding of what we currently have.
  There are three way of creating a snapshot-like thing in Nova:
 
1. create_image - takes a snapshot of the root volume and may take
   snapshots of the attached volumes depending on the volume type of
   the root volume.  I/O is not quiesced.
 
2. create_backup - takes a snapshot of the root volume with options
   to specify how often to repeat and how many previous

Re: [openstack-dev] [nova] bp proposal: libvirt-resize-disk-down

[Jon Bernard]

* sahid sahid.ferdja...@cloudwatt.com wrote:
   Greetings,
 
 A blueprint is being discussed about the disk resize down feature of libvirt 
 driver.
   https://blueprints.launchpad.net/nova/+spec/libvirt-resize-disk-down
 
 The current implementation does not handle disk resize down and just skips the
 step during a resize down of the instance. I'm really convinced we can 
 implement 
 this feature by using the good job of disk resize down of the driver xenapi.

In case it hasn't been considered yet, shrinking a filesystem can result
in terrible fragmentation.  The block allocator in resize2fs does not do
a great job of handling this case.  The result will be a very
non-optimal file layout and measurably worse performance, especially for
drives with a relatively high average seek time.

-- 
Jon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* Vishvananda Ishaya vishvana...@gmail.com wrote:
 
 On Jan 16, 2014, at 1:28 PM, Jon Bernard jbern...@tuxion.com wrote:
 
  * Vishvananda Ishaya vishvana...@gmail.com wrote:
  
  On Jan 14, 2014, at 2:10 PM, Jon Bernard jbern...@tuxion.com wrote:
  
  
  snip
  As you’ve defined the feature so far, it seems like most of it could
  be implemented client side:
  
  * pause the instance
  * snapshot the instance
  * snapshot any attached volumes
  
  For the first milestone to offer crash-consistent snapshots you are
  correct.  We'll need some additional support from libvirt, but the
  patchset should be straightforward.  The biggest question I have
  surrounding initial work is whether to use an existing API call or
  create a new one.
  
  
  I think you might have missed the “client side” part of this point. I agree
  that the snapshot multiple volumes and package it up is valuable, but I was
  trying to make the point that you could do all of this stuff client side
  if you just add support for snapshotting ephemeral drives. An all-in-one
  snapshot command could be valuable, but you are talking about orchestrating
  a lot of commands between nova, glance, and cinder and it could get kind
  of messy to try to run the whole thing from nova.
  
  If you expose each primitive required, then yes, the client could
  implement the logic to call each primitive in the correct order, handle
  error conditions, and exit while leaving everything in the correct
  state.  But that would mean you would have to implement it twice - once
  in python-novaclient and once in Horizon.  I would speculate that doing
  this on the client would be even messier.
  
  If you are concerned about the complexity of the required interactions,
  we could narrow the focus in this way:
  
   Let's say that taking a full snapshot/backup (all volumes) operates
   only on persistent storage volumes.  Users who booted from an
   ephemeral glance image shouldn't expect this feature because, by
   definition, the boot volume is not expected to live a long life.
  
  This should limit the communication to Nova and Cinder, while leaving
  Glance out (initially).  If the user booted an instance from a cinder
  volume, then we have all the volumes necessary to create an OVA and
  import to Glance as a final step.  If the boot volume is an image then
  I'm not sure, we could go in a few directions:
  
   1. No OVA is imported due to lack of boot volume
   2. A copy of the original image is included as a boot volume to create
  an OVA.
   3. Something else I am failing to see.
 
  
  If [2] seems plausible, then it probably makes sense to just ask glance
  for an image snapshot from nova while the guest is in a paused state.
  
  Thoughts?
 
 This already exists. If you run a snapshot command on a volume backed instance
 it snapshots all attached volumes. Additionally it does throw a bootable image
 into glance referring to all of the snapshots.  You could modify create image
 to do this for regular instances as well, specifying block device mapping but
 keeping the vda as an image. It could even do the same thing with the 
 ephemeral
 disk without a ton of work. Keeping this all as one command makes a lot of 
 sense
 except that it is unexpected.
 
 There is a benefit to only snapshotting the root drive sometimes because it
 keeps the image small. Here’s what I see as the ideal end state:
 
 Two commands(names are a strawman):
   create-full-image — image all drives
   create-root-image — image just the root drive
 
 These should work the same regardless of whether the root drive is volume 
 backed
 instead of the craziness we have to day of volume-backed snapshotting all 
 drives
 and instance backed just the root.  I’m not sure how we manage expectations 
 based
 on the current implementation but perhaps the best idea is just adding this in
 v3 with new names?
 
 FYI the whole OVA thing seems moot since we already have a way of representing
 multiple drives in glance via block_device_mapping properites.

I've had some time to look closer at nova and rethink things a bit and
I see what you're saying.  You are correct, taking snapshots of attached
volumes is currently supported - although not in the way that I would
like to see.  And this is where I think we can improve.

Let me first summarize my understanding of what we currently have.
There are three way of creating a snapshot-like thing in Nova:

  1. create_image - takes a snapshot of the root volume and may take
 snapshots of the attached volumes depending on the volume type of
 the root volume.  I/O is not quiesced.

  2. create_backup - takes a snapshot of the root volume with options
 to specify how often to repeat and how many previous snapshots to
 keep around. I/O is not quiesced.

  3. os-assisted-snapshot - takes a snapshot of a single cinder volume.
 The volume is first quiesced before the snapshot is initiated.

My general thesis is that I/O should

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* Vishvananda Ishaya vishvana...@gmail.com wrote:
 
 On Jan 14, 2014, at 2:10 PM, Jon Bernard jbern...@tuxion.com wrote:
 
  
  snip
  As you’ve defined the feature so far, it seems like most of it could
  be implemented client side:
  
  * pause the instance
  * snapshot the instance
  * snapshot any attached volumes
  
  For the first milestone to offer crash-consistent snapshots you are
  correct.  We'll need some additional support from libvirt, but the
  patchset should be straightforward.  The biggest question I have
  surrounding initial work is whether to use an existing API call or
  create a new one.
  
 
 I think you might have missed the “client side” part of this point. I agree
 that the snapshot multiple volumes and package it up is valuable, but I was
 trying to make the point that you could do all of this stuff client side
 if you just add support for snapshotting ephemeral drives. An all-in-one
 snapshot command could be valuable, but you are talking about orchestrating
 a lot of commands between nova, glance, and cinder and it could get kind
 of messy to try to run the whole thing from nova.

If you expose each primitive required, then yes, the client could
implement the logic to call each primitive in the correct order, handle
error conditions, and exit while leaving everything in the correct
state.  But that would mean you would have to implement it twice - once
in python-novaclient and once in Horizon.  I would speculate that doing
this on the client would be even messier.

If you are concerned about the complexity of the required interactions,
we could narrow the focus in this way:

  Let's say that taking a full snapshot/backup (all volumes) operates
  only on persistent storage volumes.  Users who booted from an
  ephemeral glance image shouldn't expect this feature because, by
  definition, the boot volume is not expected to live a long life.

This should limit the communication to Nova and Cinder, while leaving
Glance out (initially).  If the user booted an instance from a cinder
volume, then we have all the volumes necessary to create an OVA and
import to Glance as a final step.  If the boot volume is an image then
I'm not sure, we could go in a few directions:

  1. No OVA is imported due to lack of boot volume
  2. A copy of the original image is included as a boot volume to create
 an OVA.
  3. Something else I am failing to see.

If [2] seems plausible, then it probably makes sense to just ask glance
for an image snapshot from nova while the guest is in a paused state.

Thoughts?

-- 
Jon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* Vishvananda Ishaya vishvana...@gmail.com wrote:
 
 On Jan 6, 2014, at 3:50 PM, Jon Bernard jbern...@tuxion.com wrote:
 
  Hello all,
  
  I would like to propose instance-level snapshots as a feature for
  inclusion in Nova.  An initial draft of the more official proposal is
  here [1], blueprint is here [2].
  
  In a nutshell, this feature will take the existing create-image
  functionality a few steps further by providing the ability to take
  a snapshot of a running instance that includes all of its attached
  volumes.  A coordinated snapshot of multiple volumes for backup
  purposes.  The snapshot operation should occur while the instance is in
  a paused and quiesced state so that each volume snapshot is both
  consistent within itself and with respect to its sibling snapshots.
  
  I still have some open questions on a few topics:
  
  * API changes, two different approaches come to mind:
  
   1. Nova already has a command `createImage` for creating an image of an
  existing instance.  This command could be extended to take an
  additional parameter `all-volumes` that signals the underlying code
  to capture all attached volumes in addition to the root volume.  The
  semantic here is important, `createImage` is used to create
  a template image stored in Glance for later reuse.  If the primary
  intent of this new feature is for backup only, then it may not be
  wise to overlap the two operations in this way.  On the other hand,
  this approach would introduce the least amount of change to the
  existing API, requiring only modification of an existing command
  instead of the addition of an entirely new one.
  
   2. If the feature's primary use is for backup purposes, then a new API
  call may be a better approach, and leave `createImage` untouched.
  This new call could be called `createBackup` and take as a parameter
  the name of the instance.  Although it introduces a new member to the
  API reference, it would allow this feature to evolve without
  introducing regressions in any existing calls.  These two calls could
  share code at some point in the future.
 
 You’ve mentioned “If the feature’s use case is backup” a couple of times
 without specifying the answer. I think this is important to the above
 question. Also relevant is how the snapshot is stored and potentially
 restored.

The question I'm attempting to raise relates to the intended use case.
I can see the feature being valuable in two aspects:

  1. A single entry point for creating a consistent snapshot of all of
 volumes attached to a compute node.  This would make it easy to create
 a full backup of a node without having to individually snapshot each
 volume.  An added benefit here would be that each volume would be
 consistent both within itself and with respect to the other volumes
 present on the node during the time of the snapshot.  The main use
 case would be for backup purposes.  With this in mind, the
 createImage API call might be inappropriate.

  2. The same value as stated in 1, except instead of leaving the
 snapshot data stored in Cinder, the volumes are joined into
 a single OVA file and then imported into Glance.  The use case
 here would be to create a bootable image of a machine with multiple
 volumes.

I can see both use cases being valuable and I'm not positive that we
have to choose between the two. Thoughts and opinions here would be much
appreciated.

 As you’ve defined the feature so far, it seems like most of it could
 be implemented client side:
 
 * pause the instance
 * snapshot the instance
 * snapshot any attached volumes

For the first milestone to offer crash-consistent snapshots you are
correct.  We'll need some additional support from libvirt, but the
patchset should be straightforward.  The biggest question I have
surrounding initial work is whether to use an existing API call or
create a new one.

 The only thing missing in this scenario is snapshotting any ephemeral
 drives. There are workarounds for this such as:
  * use flavor with no ephemeral storage
  * boot from volume

True, but I think this is a fairly limiting set of constraints.
I suspect users would like to have an instance-level backup option that
works for them regardless of the number of attached volumes or whether
the boot volume is managed by Cinder.

 It is also worth mentioning that snapshotting a boot from volume instance
 will actually do most of this for you (everything but pausing the instance)
 and additionally give you an image which when booted will lead to a clone
 of all of the snapshotted volumes.
 
 So unless there is some additional feature regarding storing or restoring
 the backup, I only see one potential area for improvement inside of nova:
 Modifying the snapshot command to allow for snapshotting of ephemeral
 drives.

I don't want the createImage call to be overloaded such that it does far
more

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* Mark Washenberger mark.washenber...@markwash.net wrote:
 On Mon, Jan 6, 2014 at 3:50 PM, Jon Bernard jbern...@tuxion.com wrote:
 
  Hello all,
 
  I would like to propose instance-level snapshots as a feature for
  inclusion in Nova.  An initial draft of the more official proposal is
  here [1], blueprint is here [2].
 
  In a nutshell, this feature will take the existing create-image
  functionality a few steps further by providing the ability to take
  a snapshot of a running instance that includes all of its attached
  volumes.  A coordinated snapshot of multiple volumes for backup
  purposes.  The snapshot operation should occur while the instance is in
  a paused and quiesced state so that each volume snapshot is both
  consistent within itself and with respect to its sibling snapshots.
 
  I still have some open questions on a few topics:
 
  * API changes, two different approaches come to mind:
 
1. Nova already has a command `createImage` for creating an image of an
   existing instance.  This command could be extended to take an
   additional parameter `all-volumes` that signals the underlying code
   to capture all attached volumes in addition to the root volume.  The
   semantic here is important, `createImage` is used to create
   a template image stored in Glance for later reuse.  If the primary
   intent of this new feature is for backup only, then it may not be
   wise to overlap the two operations in this way.  On the other hand,
   this approach would introduce the least amount of change to the
   existing API, requiring only modification of an existing command
   instead of the addition of an entirely new one.
 
2. If the feature's primary use is for backup purposes, then a new API
   call may be a better approach, and leave `createImage` untouched.
   This new call could be called `createBackup` and take as a parameter
   the name of the instance.  Although it introduces a new member to the
   API reference, it would allow this feature to evolve without
   introducing regressions in any existing calls.  These two calls could
   share code at some point in the future.
 
  * Existing libvirt support:
 
  To initially support consistent-across-multiple-volumes snapshots,
  we must be able to ask libvirt for a snapshot of an already paused
  guest.  I don't believe such a call is currently supported, so
  changes to libvirt may be a prerequisite for this feature.
 
  Any contribution, comments, and pieces of advice are much appreciated.
 
  [1]: https://wiki.openstack.org/wiki/Nova/InstanceLevelSnapshots
  [2]: https://blueprints.launchpad.net/nova/+spec/instance-level-snapshots
 
 
 Hi Jon,
 
 In your specification in the Snapshot Storage section you say it might be
 nice to combine all of the snapshot images into a single OVF file that
 contains all volumes attached to the instance at the time of snapshot. I'd
 love it if, by the time you get to the point of implementing this storage
 part, we have an option available to you in Glance for storing something
 akin to an Instance template. An instance template would be an entity
 stored in Glance with references to each volume or image that was uploaded
 as part of the snapshot. As an example, it could be something like
 
 instance_template: {
/dev/sda: /v2/images/some-imageid,
/dev/sdb: some url for a cinder volume-like entity
 }
 
 Essentially, this kind of storage would bring the OVF metadata up into
 Glance rather than burying it down in an image byte stream where it is
 harder to search or access.
 
 This is an idea that has been discussed several times before, generally
 favorably, and if we move ahead with instance-level snapshots in Nova I'd
 love to move quickly to support it in Glance. Part of the reason for the
 delay of this feature was my worry that if Glance jumps out ahead, we'll
 end up with some instance template format that Nova doesn't really want, so
 this opportunity for collaboration on use cases would be fantastic.
 
 If after a bit more discussion in this thread, folks think these templates
 in Glance would be a good idea, we can try to draw up a proposal for how to
 implement the first cut of this feature in Glance.

This sounds awesome.  If there is a consensus for this feature, then I'm
happy to work with you on the proposal.  I'll circle back once everyone
has had a chance to weight in and we reach agreement.

-- 
Jon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal for instance-level snapshots in Nova

[Jon Bernard]

* Christopher Yeoh cbky...@gmail.com wrote:
 On Tue, Jan 7, 2014 at 7:50 AM, Jon Bernard jbern...@tuxion.com wrote:
 
  Hello all,
 
  I would like to propose instance-level snapshots as a feature for
  inclusion in Nova.  An initial draft of the more official proposal is
  here [1], blueprint is here [2].
 
  In a nutshell, this feature will take the existing create-image
  functionality a few steps further by providing the ability to take
  a snapshot of a running instance that includes all of its attached
  volumes.  A coordinated snapshot of multiple volumes for backup
  purposes.  The snapshot operation should occur while the instance is in
  a paused and quiesced state so that each volume snapshot is both
  consistent within itself and with respect to its sibling snapshots.
 
  I still have some open questions on a few topics:
 
  * API changes, two different approaches come to mind:
 
1. Nova already has a command `createImage` for creating an image of an
   existing instance.  This command could be extended to take an
   additional parameter `all-volumes` that signals the underlying code
   to capture all attached volumes in addition to the root volume.  The
   semantic here is important, `createImage` is used to create
   a template image stored in Glance for later reuse.  If the primary
   intent of this new feature is for backup only, then it may not be
   wise to overlap the two operations in this way.  On the other hand,
   this approach would introduce the least amount of change to the
   existing API, requiring only modification of an existing command
   instead of the addition of an entirely new one.
 
2. If the feature's primary use is for backup purposes, then a new API
   call may be a better approach, and leave `createImage` untouched.
   This new call could be called `createBackup` and take as a parameter
   the name of the instance.  Although it introduces a new member to the
   API reference, it would allow this feature to evolve without
   introducing regressions in any existing calls.  These two calls could
   share code at some point in the future.
 
 
 Note there already is a createBackup/create_backup API call implemented in
 the admin_actions
 extension (in V3 API it is being separated into its own extension
 https://review.openstack.org/#/c/62280/)
 It doesn't do the all volumes snapshot that you want though. There's a
 small window (basically end of icehouse) to make an incompatible change in
 the V3 API if that would be the best way to do it.

Thanks for mentioning this.  If the consensus is to leave createImage as
it is and use a different API call, then createBackup looks like
a really nice fit.

-- 
Jon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev