Re: [Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series
On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote:
It's spring! The winter snow has thawed and so here is a new
patch series to enter your life and warm your heart.
This patchset enables the in-memory part of the incremental backup
feature, without transactional support.
Support for transactions was separated into a separate series which
is also now available on-list. Getting this portion of the series
committed will help stabilize work on bitmap persistence and bitmap
migration.
Thanks to Fam Zheng for the original versions of this patchset,
And thanks to Max and Eric for reviewing 2,396 versions of it since.
===
v6:
===
01: s/underlaying/underlying/
Removed a reference to 'disabled' bitmaps.
Touching up inconsistent list indentation.
Added FreeBSD Documentation License, primarily to be difficult
07: More in-line documentation for hbitmap_merge, for return value.
Fix size cache index to be uint64_t.
09: Grammar fixes from Eric Blake, kept R-Bs.
10: Moved yield into the do{}while(). Now we check to see if we should
yield/cancel after each unsuccessful sector we transfer.
Some documentation additions for Eric Blake.
15: corrected 'num_elements' to 'start'
18: Refactored qmp.py event functions,
Added in more explicit exception classes.
No changes to iotests.py, just qmp.py.
Key:
[] : patches are identical
[] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences,
respectively
001/21:[0057] [FC] 'docs: incremental backup documentation'
002/21:[] [--] 'qapi: Add optional field name to block dirty bitmap'
003/21:[] [--] 'qmp: Ensure consistent granularity type'
004/21:[] [--] 'qmp: Add block-dirty-bitmap-add and
block-dirty-bitmap-remove'
005/21:[] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
006/21:[] [--] 'hbitmap: cache array lengths'
007/21:[0008] [FC] 'hbitmap: add hbitmap_merge'
008/21:[] [--] 'block: Add bitmap disabled status'
009/21:[0008] [FC] 'block: Add bitmap successors'
010/21:[0013] [FC] 'qmp: Add support of dirty-bitmap sync mode for
drive-backup'
011/21:[] [--] 'qmp: add block-dirty-bitmap-clear'
012/21:[] [--] 'qmp: Add dirty bitmap status field in query-block'
013/21:[] [--] 'block: add BdrvDirtyBitmap documentation'
014/21:[] [--] 'block: Ensure consistent bitmap function prototypes'
015/21:[0002] [FC] 'block: Resize bitmaps on bdrv_truncate'
016/21:[] [--] 'hbitmap: truncate tests'
017/21:[] [-C] 'iotests: add invalid input incremental backup tests'
018/21:[0095] [FC] 'iotests: add QMP event waiting queue'
019/21:[] [--] 'iotests: add simple incremental backup case'
020/21:[] [--] 'iotests: add incremental backup failure recovery test'
021/21:[] [--] 'iotests: add incremental backup granularity tests'
===
v5:
===
10: Code has been moved into backup_run_incremental()
'polyrhythm' check is removed,
clusters_per_iter variable is introduced instead.
If the bitmap granularity is larger than the backup granularity,
loop over the backup_do_cow call multiple times.
If the bitmap granularity is smaller, skip the iterator ahead as
we had been doing previously.
14: Only whitespace changes caused by patch 10.
15: Changed my approach for clearing out data for the hbitmap
truncate shrink case, as suggested by Stefan
18: Added a proper timeout mechanism to qmp.pull_event():
wait=False or wait=0.0 implies non-blocking.
wait=True implies blocking.
wait=60.0 implies a 60 second timeout.
VM.event_wait() now uses a 60 second timeout by default.
19: Many things:
The big picture is to add a set of full backups alongside the
incremental backups created during the test to be able to test
the validity of each incremental at the conclusion of the test
when we can shut the VM down.
To do this, there are two basic changes:
(1) Keep a list of pairs of backup filenames (incremental, reference);
create a full reference backup for every incremental created.
(2) Refactor the backup helper functions a bit.
20: Naming fallout from 19
Added calls to vm.shutdown() and check_backups().
21: NEW, adds granularity tests that cover the changes in patch 10.
===
v4:
===
04: Some in-line documentation for block_dirty_bitmap_lookup
Changed behavior with respect to aio_context
(always acquire, release if pbs == null, give to user otherwise)
10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
Kept R-B.
16: Added some comments to test_check_boundary_bits.
Kept R-B.
17: Folded in refactor from incremental transactions v1 (Poor Kitty)
18: Pulled forward from incremental transactions v1
Kept R-B from that series.
19: Folded in refactor from
Re: [Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series
On 04/23/2015 09:19 AM, Stefan Hajnoczi wrote: On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote: === v6: === 01: s/underlaying/underlying/ Removed a reference to 'disabled' bitmaps. Touching up inconsistent list indentation. Added FreeBSD Documentation License, primarily to be difficult Please stick to the currently used set of licenses in the future, unless you have a strong reason. It's not a good use of anyone's time to fuss with licenses when we have enough of them in the codebase already. In my non-lawyer opinion the license you chose seems okay but I'd rather avoid the risk and hassle. Thanks, Stefan I know I said primarily to be difficult but I was just being facetious. I didn't find the GPL2+ to be suitable for documentation, strictly, so I went to read up on the documentation licenses that the fsf support/recommend. There's the GNU documentation license, but I found that unsuitable for a couple reasons -- one of them was that you are forbidden(!) from changing the text of the license, and there are some provisions in there I didn't like, such as requiring the full text of the license to be included with compiled copies of the document. That's not something I care about -- a reference in source, for instance, is sufficient, or a copy of the license being distributed *with* the compiled source is fine, but I have no need for the full license to be copied with the compiled version. The other documentation license the fsf recommends is the FreeBSD one, and that one looked appealing, short, and to the point, so it is the one I chose. It is essentially the FreeBSD license with words altered to clarify what code and source means with respect to a document. Sorry for /actually/ being difficult; but Eric Blake was urging me to select a license instead of relying on the implicit GPL, so I did go out of my way to choose one I found appropriate. I stand by my pick. --js
Re: [Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series
On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote: === v6: === 01: s/underlaying/underlying/ Removed a reference to 'disabled' bitmaps. Touching up inconsistent list indentation. Added FreeBSD Documentation License, primarily to be difficult Please stick to the currently used set of licenses in the future, unless you have a strong reason. It's not a good use of anyone's time to fuss with licenses when we have enough of them in the codebase already. In my non-lawyer opinion the license you chose seems okay but I'd rather avoid the risk and hassle. Thanks, Stefan pgpPyTMJK68Iw.pgp Description: PGP signature
[Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series
It's spring! The winter snow has thawed and so here is a new
patch series to enter your life and warm your heart.
This patchset enables the in-memory part of the incremental backup
feature, without transactional support.
Support for transactions was separated into a separate series which
is also now available on-list. Getting this portion of the series
committed will help stabilize work on bitmap persistence and bitmap
migration.
Thanks to Fam Zheng for the original versions of this patchset,
And thanks to Max and Eric for reviewing 2,396 versions of it since.
===
v6:
===
01: s/underlaying/underlying/
Removed a reference to 'disabled' bitmaps.
Touching up inconsistent list indentation.
Added FreeBSD Documentation License, primarily to be difficult
07: More in-line documentation for hbitmap_merge, for return value.
Fix size cache index to be uint64_t.
09: Grammar fixes from Eric Blake, kept R-Bs.
10: Moved yield into the do{}while(). Now we check to see if we should
yield/cancel after each unsuccessful sector we transfer.
Some documentation additions for Eric Blake.
15: corrected 'num_elements' to 'start'
18: Refactored qmp.py event functions,
Added in more explicit exception classes.
No changes to iotests.py, just qmp.py.
Key:
[] : patches are identical
[] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/21:[0057] [FC] 'docs: incremental backup documentation'
002/21:[] [--] 'qapi: Add optional field name to block dirty bitmap'
003/21:[] [--] 'qmp: Ensure consistent granularity type'
004/21:[] [--] 'qmp: Add block-dirty-bitmap-add and
block-dirty-bitmap-remove'
005/21:[] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
006/21:[] [--] 'hbitmap: cache array lengths'
007/21:[0008] [FC] 'hbitmap: add hbitmap_merge'
008/21:[] [--] 'block: Add bitmap disabled status'
009/21:[0008] [FC] 'block: Add bitmap successors'
010/21:[0013] [FC] 'qmp: Add support of dirty-bitmap sync mode for
drive-backup'
011/21:[] [--] 'qmp: add block-dirty-bitmap-clear'
012/21:[] [--] 'qmp: Add dirty bitmap status field in query-block'
013/21:[] [--] 'block: add BdrvDirtyBitmap documentation'
014/21:[] [--] 'block: Ensure consistent bitmap function prototypes'
015/21:[0002] [FC] 'block: Resize bitmaps on bdrv_truncate'
016/21:[] [--] 'hbitmap: truncate tests'
017/21:[] [-C] 'iotests: add invalid input incremental backup tests'
018/21:[0095] [FC] 'iotests: add QMP event waiting queue'
019/21:[] [--] 'iotests: add simple incremental backup case'
020/21:[] [--] 'iotests: add incremental backup failure recovery test'
021/21:[] [--] 'iotests: add incremental backup granularity tests'
===
v5:
===
10: Code has been moved into backup_run_incremental()
'polyrhythm' check is removed,
clusters_per_iter variable is introduced instead.
If the bitmap granularity is larger than the backup granularity,
loop over the backup_do_cow call multiple times.
If the bitmap granularity is smaller, skip the iterator ahead as
we had been doing previously.
14: Only whitespace changes caused by patch 10.
15: Changed my approach for clearing out data for the hbitmap
truncate shrink case, as suggested by Stefan
18: Added a proper timeout mechanism to qmp.pull_event():
wait=False or wait=0.0 implies non-blocking.
wait=True implies blocking.
wait=60.0 implies a 60 second timeout.
VM.event_wait() now uses a 60 second timeout by default.
19: Many things:
The big picture is to add a set of full backups alongside the
incremental backups created during the test to be able to test
the validity of each incremental at the conclusion of the test
when we can shut the VM down.
To do this, there are two basic changes:
(1) Keep a list of pairs of backup filenames (incremental, reference);
create a full reference backup for every incremental created.
(2) Refactor the backup helper functions a bit.
20: Naming fallout from 19
Added calls to vm.shutdown() and check_backups().
21: NEW, adds granularity tests that cover the changes in patch 10.
===
v4:
===
04: Some in-line documentation for block_dirty_bitmap_lookup
Changed behavior with respect to aio_context
(always acquire, release if pbs == null, give to user otherwise)
10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
Kept R-B.
16: Added some comments to test_check_boundary_bits.
Kept R-B.
17: Folded in refactor from incremental transactions v1 (Poor Kitty)
18: Pulled forward from incremental transactions v1
Kept R-B from that series.
19: Folded in refactor from incremental transactions v1
Added offset assertions into wait_incremental
20: Removed error tolerance from wait_until_completed, as
these patches no longer make use
