On 02/23/2017 09:29 AM, Peter Lieven wrote: > Am 22.02.2017 um 22:17 schrieb John Snow: >> >> On 02/22/2017 03:45 AM, Peter Lieven wrote: >>> Am 21.02.2017 um 22:13 schrieb John Snow: >>>> On 02/21/2017 07:43 AM, Peter Lieven wrote: >>>>> Hi, >>>>> >>>>> >>>>> is there anyone ever thought about implementing something like VMware >>>>> CBT in Qemu? >>>>> >>>>> >>>>> https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020128 >>>>> >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> Peter >>>>> >>>>> >>>> A bit outdated now, but: >>>> http://wiki.qemu-project.org/Features/IncrementalBackup >>>> >>>> and also a summary I wrote not too far back (PDF): >>>> https://drive.google.com/file/d/0B3CFr1TuHydWalVJaEdPaE5PbFE >>>> >>>> and I'm sure the Virtuozzo developers could chime in on this subject, >>>> but basically we do have something similar in the works, as eblake >>>> says. >>> Hi John, Hi Erik, >>> >>> thanks for your feedback. Are you both the ones working primary on >>> this topic? >>> If there is anything to review or help needed, please let me know. >>> >> I've been working on incremental backups; Fam and I now co-maintain >> block/dirty-bitmap.c. >> >> Vladimir Sementsov-Ogievskiy has been working on bitmap persistence and >> migration from Virtuozzo; as well as the NBD specification amendment to >> allow us to fleece images with dirty bitmaps. >> >> (Check the wiki and the whitepaper I linked!) >> >> Eric has been guiding the review process for the NBD side of things. >> >>> My 2 cents: >>> I thing I had in mind if there is no image fleecing available, but >>> fetching the dirty bitmap >>> from external would be a feauture to put a write lock on a block device. >>> Write lock means, drain all pending writes and queue all further >>> writes until unlock (as if they >>> were throttled to zero). This could help fetch consistent backups >>> from storage device (thinking of iSCSI SAN) without >>> the help of the hypervisor to actually transfer data (no load in the >>> frontend network or the host). What would further >>> be needed is a write generation for each block, not just only a dirty >>> bitmap. >>> >>> In this case something like this via QMP (and external software) >>> should work: >>> ---8<--- >>> gen = write generation of last backup (or 0 for full backup) >>> do { >>> nextgen = fetch current write generation (via QMP) >> As Eric said, there's a lot of hostility to using QMP as a metadata >> transmission protocol. >> >>> dirtymap = send all block whose write generation is greater >>> than 'gen' (via QMP) >>> dirtycnt = 0 >>> foreach block in dirtymap { >>> copy to backup via external software >>> dirtycnt++ >>> } >>> gen = nextgen >>> } while (dirtycnt < X) <--- to achieve this a thorttling or >>> similar might be needed >>> >>> fsfreeze (optional) >>> write lock (via QMP) >>> backupgen = fetch current write generation (via QMP) >>> dirtymap = send all block whose write generation is greater than >>> 'gen' (via QMP) >>> foreach block in dirtymap { >>> copy to backup via external software >>> } >>> unlock (via QMP) >>> fsthaw (optional) >>> --->8--- >>> >>> As far as I understand CBT in VMware is not just only a dirty bitmap, >>> but also a write generation tracking for blocks (size 64kb or whatever) >>> >> I think at the moment I'm worried about getting the basic features out >> the door, but I'm not opposed to adding fancier features if there's >> justification or demand for them. > > Sure, the basic features are most important. I was just thinking of the > above scenario to interact with a NAS and have Qemu's "help" > to create incremental backups. > > Peter
If you get the chance to read the white paper I linked to you, please let me know which use cases we might not be able to cover that you feel other programs might offer. I can also make a point to CC you on future upstream discussions as they happen. Thanks, --js
