Re: [Qemu-devel] QEMU bitmap backup usability FAQ
On August 21, 2019 11:19 pm, John Snow wrote: > > > On 8/21/19 10:21 AM, Vladimir Sementsov-Ogievskiy wrote: >> [CC Nikolay] >> >> 21.08.2019 1:25, John Snow wrote: >>> Hi, downstream here at Red Hat I've been fielding some questions about >>> the usability and feature readiness of Bitmaps (and related features) in >>> QEMU. >>> >>> Here are some questions I answered internally that I am copying to the >>> list for two reasons: >>> >>> (1) To make sure my answers are actually correct, and >>> (2) To share this pseudo-reference with the community at large. >>> >>> This is long, and mostly for reference. There's a summary at the bottom >>> with some todo items and observations about the usability of the feature >>> as it exists in QEMU. >>> >>> Before too long, I intend to send a more summarized "roadmap" mail which >>> details all of the current and remaining work to be done in and around >>> the bitmaps feature in QEMU. >>> >>> >>> Questions: >>> "What format(s) is/are required for this functionality?" >>> >>> From the QEMU API, any format can be used to create and author >>> incremental backups. The only known format limitations are: >>> >>> 1. Persistent bitmaps cannot be created on any format except qcow2, >>> although there are hooks to add support to other formats at a later date >>> if desired. >>> >>> DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient >>> bitmaps instead of persistent ones. >>> >>> Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent >>> ones in case they made a mistake. >> >> I doubt, as in my opinion real users of Qemu are not people but libvirt, >> which >> should never make such mistake. >> > > Right, that's largely been the consensus here; but there is some concern > that libvirt might not be the only user of QEMU, so I felt it was worth > documenting some of the crucial moments where it was "easy" to get it wrong. > > I think like it or not, the API that QEMU presents has to be considered > on its own without libvirt because it's not a given that libvirt will > forever and always be the only user of QEMU. > > I do think that any problems of this kind that can be solved in libvirt > are not immediate, crucial action items. libvirt WILL be the major user > of these features. Chiming in with a bit of vacation-induced delay - libvirt is definitely not the only user of QEMU's QMP interface - we at Proxmox use QEMU directly in our Proxmox VE product (usually a rather recent version, currently 4.0 with some cherry-picks and custom patches) and have been doing so for quite a while (the earliest reference in git that I can find is for QEMU 0.11.1, but there was SVN before that..). IIRC, we currently only use the bitmap features for our own custom backup jobs (shipped in our patched QEMU packages[1]), and are planning to integrate differential mirroring on top of storage-level/ZFS snapshots once that has stabilized upstream. That being said, the same basic guidelines apply to us that apply to libvirt - our users are (normally) also not talking QMP manually, our stack does it for them. Misuse of QMP interfaces is thus a bug in our software, and not a mistake made by its user. We do expose HMP over our API, but that is more for convenience of power users than any real use case that I am aware of. 1: patches #20-24 from https://git.proxmox.com/?p=pve-qemu.git;a=tree;f=debian/patches/pve;h=46bd31d60fe2c03571d9d29c7ee80f208206d37e;hb=refs/heads/master
Re: [Qemu-devel] QEMU bitmap backup usability FAQ
22.08.2019 0:19, John Snow wrote: > > > On 8/21/19 10:21 AM, Vladimir Sementsov-Ogievskiy wrote: >> [CC Nikolay] >> >> 21.08.2019 1:25, John Snow wrote: >>> Hi, downstream here at Red Hat I've been fielding some questions about >>> the usability and feature readiness of Bitmaps (and related features) in >>> QEMU. >>> >>> Here are some questions I answered internally that I am copying to the >>> list for two reasons: >>> >>> (1) To make sure my answers are actually correct, and >>> (2) To share this pseudo-reference with the community at large. >>> >>> This is long, and mostly for reference. There's a summary at the bottom >>> with some todo items and observations about the usability of the feature >>> as it exists in QEMU. >>> >>> Before too long, I intend to send a more summarized "roadmap" mail which >>> details all of the current and remaining work to be done in and around >>> the bitmaps feature in QEMU. >>> >>> >>> Questions: >>> "What format(s) is/are required for this functionality?" >>> >>> From the QEMU API, any format can be used to create and author >>> incremental backups. The only known format limitations are: >>> >>> 1. Persistent bitmaps cannot be created on any format except qcow2, >>> although there are hooks to add support to other formats at a later date >>> if desired. >>> >>> DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient >>> bitmaps instead of persistent ones. >>> >>> Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent >>> ones in case they made a mistake. >> >> I doubt, as in my opinion real users of Qemu are not people but libvirt, >> which >> should never make such mistake. >> > > Right, that's largely been the consensus here; but there is some concern > that libvirt might not be the only user of QEMU, so I felt it was worth > documenting some of the crucial moments where it was "easy" to get it wrong. > > I think like it or not, the API that QEMU presents has to be considered > on its own without libvirt because it's not a given that libvirt will > forever and always be the only user of QEMU. > > I do think that any problems of this kind that can be solved in libvirt > are not immediate, crucial action items. libvirt WILL be the major user > of these features. > > However, try as we might, releasing a set of primitive operations that > offer 998 ways to corrupt your data and 2 ways to manage it correctly > are going to provoke some questions from people who are trying to work > with that API, including from libvirt developers. > > It might be the conclusion that it's libvirt's job to safeguard the user > from themselves, but we at least need to present consistent and clear > information about the way we expect/anticipate people to use the APIs, > because people DO keep asking me about several of these issues and the > usability problems they perceive with the QEMU API. > > So this thread was largely in attempt to explore what some "solutions" > to perceived problems look like, mostly to come to the conclusion that > the actual "must-haves" list in QEMU is not very long compared to the > "nice-to-haves?" list. > >>> >>> >>> 2. When using push backups (blockdev-backup, drive-backup), you may use >>> any format as a target format. > >>> DANGER CAVEAT #2: without backing file and/or filesystem-less sparse >>> support, these images will be unusable. >> >> You mean incremental backups of course, as the whole document is about >> bitmaps. >> > > Ah, yes, incremental push backups. Full backups are of course not a > problem. :) > >>> >>> EXAMPLE: Backing up to a raw file loses allocation information, so we >>> can no longer distinguish between zeroes and unallocated regions. The >>> cluster size is also lost. This file will not be usable without >>> additional metadata recorded elsewhere.* >>> >>> (* This is complicated, but it is in theory possible to do a push backup >>> to e.g. an NBD target with custom server code that saves allocation >>> information to a metadata file, which would allow you to reconstruct >>> backups. For instance, recording in a .json file which extents were >>> written out would allow you to -- with a custom binary -- write this >>> information on top of a base file to reconstruct a backup.) >>> >>> >>> 3. Any format can be used for either shared storage or live storage >>> migrations. There are TWO distinct mechanisms for migrating bitmaps: >>> >>> A) The bitmap is flushed to storage and re-opened on the destination. >>> This is only supported for qcow2 and shared-storage migrations. >> >> cons: flushing/reopening is done during migration downtime, so if you have >> a lot of bitmap data (for example, 64k granulared bitmap for 16tb disk is >> ~30MB, and there may be several bitmaps) downtime will become long. >> > > Worth documenting the drawback, yes. > >>> >>> B) The bitmap is live-migrated to the destination. This is supported for >>> any format and can be used for either shared storage or live
Re: [Qemu-devel] QEMU bitmap backup usability FAQ
On 8/21/19 10:21 AM, Vladimir Sementsov-Ogievskiy wrote: > [CC Nikolay] > > 21.08.2019 1:25, John Snow wrote: >> Hi, downstream here at Red Hat I've been fielding some questions about >> the usability and feature readiness of Bitmaps (and related features) in >> QEMU. >> >> Here are some questions I answered internally that I am copying to the >> list for two reasons: >> >> (1) To make sure my answers are actually correct, and >> (2) To share this pseudo-reference with the community at large. >> >> This is long, and mostly for reference. There's a summary at the bottom >> with some todo items and observations about the usability of the feature >> as it exists in QEMU. >> >> Before too long, I intend to send a more summarized "roadmap" mail which >> details all of the current and remaining work to be done in and around >> the bitmaps feature in QEMU. >> >> >> Questions: >> >>> "What format(s) is/are required for this functionality?" >> >> From the QEMU API, any format can be used to create and author >> incremental backups. The only known format limitations are: >> >> 1. Persistent bitmaps cannot be created on any format except qcow2, >> although there are hooks to add support to other formats at a later date >> if desired. >> >> DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient >> bitmaps instead of persistent ones. >> >> Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent >> ones in case they made a mistake. > > I doubt, as in my opinion real users of Qemu are not people but libvirt, which > should never make such mistake. > Right, that's largely been the consensus here; but there is some concern that libvirt might not be the only user of QEMU, so I felt it was worth documenting some of the crucial moments where it was "easy" to get it wrong. I think like it or not, the API that QEMU presents has to be considered on its own without libvirt because it's not a given that libvirt will forever and always be the only user of QEMU. I do think that any problems of this kind that can be solved in libvirt are not immediate, crucial action items. libvirt WILL be the major user of these features. However, try as we might, releasing a set of primitive operations that offer 998 ways to corrupt your data and 2 ways to manage it correctly are going to provoke some questions from people who are trying to work with that API, including from libvirt developers. It might be the conclusion that it's libvirt's job to safeguard the user from themselves, but we at least need to present consistent and clear information about the way we expect/anticipate people to use the APIs, because people DO keep asking me about several of these issues and the usability problems they perceive with the QEMU API. So this thread was largely in attempt to explore what some "solutions" to perceived problems look like, mostly to come to the conclusion that the actual "must-haves" list in QEMU is not very long compared to the "nice-to-haves?" list. >> >> >> 2. When using push backups (blockdev-backup, drive-backup), you may use >> any format as a target format. > >> DANGER CAVEAT #2: without backing file and/or filesystem-less sparse >> support, these images will be unusable. > > You mean incremental backups of course, as the whole document is about > bitmaps. > Ah, yes, incremental push backups. Full backups are of course not a problem. :) >> >> EXAMPLE: Backing up to a raw file loses allocation information, so we >> can no longer distinguish between zeroes and unallocated regions. The >> cluster size is also lost. This file will not be usable without >> additional metadata recorded elsewhere.* >> >> (* This is complicated, but it is in theory possible to do a push backup >> to e.g. an NBD target with custom server code that saves allocation >> information to a metadata file, which would allow you to reconstruct >> backups. For instance, recording in a .json file which extents were >> written out would allow you to -- with a custom binary -- write this >> information on top of a base file to reconstruct a backup.) >> >> >> 3. Any format can be used for either shared storage or live storage >> migrations. There are TWO distinct mechanisms for migrating bitmaps: >> >> A) The bitmap is flushed to storage and re-opened on the destination. >> This is only supported for qcow2 and shared-storage migrations. > > cons: flushing/reopening is done during migration downtime, so if you have > a lot of bitmap data (for example, 64k granulared bitmap for 16tb disk is > ~30MB, and there may be several bitmaps) downtime will become long. > Worth documenting the drawback, yes. >> >> B) The bitmap is live-migrated to the destination. This is supported for >> any format and can be used for either shared storage or live storage >> migrations. >> >> DANGER CAVEAT #3: The second bitmap migration technique there is an >> optional migration capability that must be enabled explicitly. >> Otherwise, some
Re: [Qemu-devel] QEMU bitmap backup usability FAQ
[CC Nikolay] 21.08.2019 1:25, John Snow wrote: > Hi, downstream here at Red Hat I've been fielding some questions about > the usability and feature readiness of Bitmaps (and related features) in > QEMU. > > Here are some questions I answered internally that I am copying to the > list for two reasons: > > (1) To make sure my answers are actually correct, and > (2) To share this pseudo-reference with the community at large. > > This is long, and mostly for reference. There's a summary at the bottom > with some todo items and observations about the usability of the feature > as it exists in QEMU. > > Before too long, I intend to send a more summarized "roadmap" mail which > details all of the current and remaining work to be done in and around > the bitmaps feature in QEMU. > > > Questions: > >> "What format(s) is/are required for this functionality?" > > From the QEMU API, any format can be used to create and author > incremental backups. The only known format limitations are: > > 1. Persistent bitmaps cannot be created on any format except qcow2, > although there are hooks to add support to other formats at a later date > if desired. > > DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient > bitmaps instead of persistent ones. > > Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent > ones in case they made a mistake. I doubt, as in my opinion real users of Qemu are not people but libvirt, which should never make such mistake. > > > 2. When using push backups (blockdev-backup, drive-backup), you may use > any format as a target format. > > DANGER CAVEAT #2: without backing file and/or filesystem-less sparse > support, these images will be unusable. You mean incremental backups of course, as the whole document is about bitmaps. > > EXAMPLE: Backing up to a raw file loses allocation information, so we > can no longer distinguish between zeroes and unallocated regions. The > cluster size is also lost. This file will not be usable without > additional metadata recorded elsewhere.* > > (* This is complicated, but it is in theory possible to do a push backup > to e.g. an NBD target with custom server code that saves allocation > information to a metadata file, which would allow you to reconstruct > backups. For instance, recording in a .json file which extents were > written out would allow you to -- with a custom binary -- write this > information on top of a base file to reconstruct a backup.) > > > 3. Any format can be used for either shared storage or live storage > migrations. There are TWO distinct mechanisms for migrating bitmaps: > > A) The bitmap is flushed to storage and re-opened on the destination. > This is only supported for qcow2 and shared-storage migrations. cons: flushing/reopening is done during migration downtime, so if you have a lot of bitmap data (for example, 64k granulared bitmap for 16tb disk is ~30MB, and there may be several bitmaps) downtime will become long. > > B) The bitmap is live-migrated to the destination. This is supported for > any format and can be used for either shared storage or live storage > migrations. > > DANGER CAVEAT #3: The second bitmap migration technique there is an > optional migration capability that must be enabled explicitly. > Otherwise, some migration combinations may drop bitmaps. Also, bad thing may happen if we try to migrate persistent bitmap to not qcow2. Also, with enabled capability, flushing/reopening is automatically disabled. > > Matrix: > >> migrate = migrate_capability or (persistent and shared_storage) > > Enumerated: > > live storage + raw : transient + no-capability: Dropped > live-storage + raw : transient + bm-capability: Migrated > live-storage + qcow2 : transient + no-capability: Dropped > live-storage + qcow2 : transient + bm-capability: Migrated > live-storage + qcow2 : persistent + no-capability: Dropped (!) > live-storage + qcow2 : persistent + bm-capability: Migrated > > shared-storage + raw : transient - no-capability: Dropped > shared-storage + raw : transient + bm-capability: Migrated > shared-storage + qcow2 : transient + no-capability: Migrated Dropped you mean > shared-storage + qcow2 : transient + bm-capability: Migrated > shared-storage + qcow2 : persistent + no-capability: Migrated > shared-storage + qcow2 : persistent + bm-capability: Migrated > > Enabling the bitmap migration capability will ALWAYS migrate the bitmap. > If it's disabled, we will only migrate the bitmaps for shared storage > migrations where the bitmap is persistent, which is a qcow2-only case. > > There is no warning or error if you attempt to migrate in a manner that > loses your bitmaps. > > (I might be persuaded to add a case for when you are doing a live > storage migration of qcow2 with persistent bitmaps, which is somewhat a > conflicting case: you've asked for the bitmap to be persistent, but it > seems likely that if this ever happens in practice, it's because you > have
[Qemu-devel] QEMU bitmap backup usability FAQ
Hi, downstream here at Red Hat I've been fielding some questions about the usability and feature readiness of Bitmaps (and related features) in QEMU. Here are some questions I answered internally that I am copying to the list for two reasons: (1) To make sure my answers are actually correct, and (2) To share this pseudo-reference with the community at large. This is long, and mostly for reference. There's a summary at the bottom with some todo items and observations about the usability of the feature as it exists in QEMU. Before too long, I intend to send a more summarized "roadmap" mail which details all of the current and remaining work to be done in and around the bitmaps feature in QEMU. Questions: > "What format(s) is/are required for this functionality?" >From the QEMU API, any format can be used to create and author incremental backups. The only known format limitations are: 1. Persistent bitmaps cannot be created on any format except qcow2, although there are hooks to add support to other formats at a later date if desired. DANGER CAVEAT #1: Adding bitmaps to QEMU by default creates transient bitmaps instead of persistent ones. Possible TODO: Allow users to 'upgrade' transient bitmaps to persistent ones in case they made a mistake. 2. When using push backups (blockdev-backup, drive-backup), you may use any format as a target format. DANGER CAVEAT #2: without backing file and/or filesystem-less sparse support, these images will be unusable. EXAMPLE: Backing up to a raw file loses allocation information, so we can no longer distinguish between zeroes and unallocated regions. The cluster size is also lost. This file will not be usable without additional metadata recorded elsewhere.* (* This is complicated, but it is in theory possible to do a push backup to e.g. an NBD target with custom server code that saves allocation information to a metadata file, which would allow you to reconstruct backups. For instance, recording in a .json file which extents were written out would allow you to -- with a custom binary -- write this information on top of a base file to reconstruct a backup.) 3. Any format can be used for either shared storage or live storage migrations. There are TWO distinct mechanisms for migrating bitmaps: A) The bitmap is flushed to storage and re-opened on the destination. This is only supported for qcow2 and shared-storage migrations. B) The bitmap is live-migrated to the destination. This is supported for any format and can be used for either shared storage or live storage migrations. DANGER CAVEAT #3: The second bitmap migration technique there is an optional migration capability that must be enabled explicitly. Otherwise, some migration combinations may drop bitmaps. Matrix: > migrate = migrate_capability or (persistent and shared_storage) Enumerated: live storage + raw : transient + no-capability: Dropped live-storage + raw : transient + bm-capability: Migrated live-storage + qcow2 : transient + no-capability: Dropped live-storage + qcow2 : transient + bm-capability: Migrated live-storage + qcow2 : persistent + no-capability: Dropped (!) live-storage + qcow2 : persistent + bm-capability: Migrated shared-storage + raw : transient - no-capability: Dropped shared-storage + raw : transient + bm-capability: Migrated shared-storage + qcow2 : transient + no-capability: Migrated shared-storage + qcow2 : transient + bm-capability: Migrated shared-storage + qcow2 : persistent + no-capability: Migrated shared-storage + qcow2 : persistent + bm-capability: Migrated Enabling the bitmap migration capability will ALWAYS migrate the bitmap. If it's disabled, we will only migrate the bitmaps for shared storage migrations where the bitmap is persistent, which is a qcow2-only case. There is no warning or error if you attempt to migrate in a manner that loses your bitmaps. (I might be persuaded to add a case for when you are doing a live storage migration of qcow2 with persistent bitmaps, which is somewhat a conflicting case: you've asked for the bitmap to be persistent, but it seems likely that if this ever happens in practice, it's because you have neglected to ask for it to be migrated to the new host.) See iotest 169 for more details on this. At present, these are the only format limitations I am consciously aware of. From a management API/GUI perspective, it makes sense to restrict the feature set to "qcow2 only" to minimize edge cases. > "Is libvirt aware of these 'gotcha' cases?" >From talks I've had with Eric Blake and Peter Krempa, they certainly are now. > "Is it possible to make persistent the default?" Not quickly. In QEMU, not without a deprecation period or some other incompatibility. Default values are not (yet?) introspectable via the schema. We need (possibly) up to two QAPI extensions: I) The ability to return deprecation warnings when issuing a command that will cease to work in the future. This has been being discussed somewhat on-list recently.
