This is a continuation of previous versions of this series. v2's cover letter was the most elaborate, which you can find here (includes benchmarks): http://lists.nongnu.org/archive/html/qemu-devel/2014-11/msg03430.html
See patch 1 for an explanation of why this series exists and what it does. Patch 1 is basically the core of this series, the rest just employs the functions introduced there. The new thing about v4 is that there now is a way to limit the total combined size of all data structures used for the new overlap prevention algorithm. Bad news: The limit is disabled by default. This is mainly because for image creation, it is pretty hard to find a sane default (it depends on the desired image size; but the image is not created with that size, but rather with size 0 and then truncated; inferring the default from the image size seems to make sense, but we cannot do this for images that are going to be truncated). RFC: An alternative would be to infer the limit like so: MAX(1M, guest_image_size / 1M). It seems sane to me and I guess it should work even for images with very, very large guest sizes. But I think it will break if you're trying to preallocate like a 1 PB image (which is not really a concern right now, but let's see about it in ten years)... On the other hand, the worst that happens is that the event is generated and the overlap checks won't be performed for some parts of the image during its creation. Doesn't sound too bad to me. What do you think, dears reviewers? In a later patch, we may want to change the meaning of the "constant" overlap checking option to mean the same as "cached", which is everything except for inactive L2 tables. This series does make checking for overlaps with inactive L2 tables at runtime just as cheap as everything else (constant time plus caching), but using these checks means qemu has to read all the snapshot L1 tables when opening a qcow2 file. This does not take long, of course, but it does result in a bit of overhead so I did not want to enable it by default. v4: - Patch 1: - Use the length of bs->file instead of bs itself as a basis on how many windows should be allocated initially - If the fragment list would take up more memory than the bitmap (which is conceivable, considering that every cluster in the bitmap/bytemap uses exactly one byte, whereas every cluster may use up to one whole entry in the fragment list (which takes up four bytes)), do not drop the bitmap on cache eviction, but rather just abort the generation of the fragment list - The cache_size parameter should be divided by WINDOW_SIZE only instead of WINDOW_SIZE + sizeof(*s->metadata_list->cached_windows); the user is more likely to specify multiples of WINDOW_SIZE, so we should take that into account (the sizeof(...) part is really negligible, and also, if the user really wants to limit the memory usage, there are now four new patches in this series to do that) - Just drop the bitmap when the metadata list is destroyed; do not use destroy_window_bitmap() to do so, because this will (unneccessarily) generate a fragment list (which is going to be dropped, too, anyway) - Patch 13: Added; adds a new QMP event to be used by patch 14 - Patch 14: Added; adds a control mechanism to limit the amount of memory used for the overlap structures (and issues the event from patch 13 if that limit is hit) - Patch 15: Added; adds runtime options to qcow2 to control the cache size and the total memory limit for the overlap checking - Patch 16: Added; exposes these options via QMP - Patch 17: Added; some tests for these new options git-backport-diff against v3: Key: [----] : patches are identical [####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively 001/17:[0071] [FC] 'qcow2: Add new overlap check functions' 002/17:[----] [--] 'qcow2: Pull up overlap check option evaluation' 003/17:[----] [-C] 'qcow2: Create metadata list' 004/17:[----] [-C] 'qcow2/overlaps: Protect image header' 005/17:[----] [--] 'qcow2/overlaps: Protect refcount table' 006/17:[----] [-C] 'qcow2/overlaps: Protect refcount blocks' 007/17:[----] [--] 'qcow2/overlaps: Protect active L1 table' 008/17:[----] [-C] 'qcow2/overlaps: Protect active L2 tables' 009/17:[----] [-C] 'qcow2/overlaps: Protect snapshot table' 010/17:[----] [--] 'qcow2/overlaps: Protect inactive L1 tables' 011/17:[----] [-C] 'qcow2/overlaps: Protect inactive L2 tables' 012/17:[----] [--] 'qcow2: Use new metadata overlap check function' 013/17:[down] 'qcow2/overlaps: Add "memory limit reached" event' 014/17:[down] 'qcow2/overlaps: Add memory usage limit' 015/17:[down] 'qcow2: Add overlap structure memory size options' 016/17:[down] 'qapi: Expose new qcow2 overlap check options' 017/17:[down] 'iotests: Test qcow2's overlap check memory limit' Max Reitz (17): qcow2: Add new overlap check functions qcow2: Pull up overlap check option evaluation qcow2: Create metadata list qcow2/overlaps: Protect image header qcow2/overlaps: Protect refcount table qcow2/overlaps: Protect refcount blocks qcow2/overlaps: Protect active L1 table qcow2/overlaps: Protect active L2 tables qcow2/overlaps: Protect snapshot table qcow2/overlaps: Protect inactive L1 tables qcow2/overlaps: Protect inactive L2 tables qcow2: Use new metadata overlap check function qcow2/overlaps: Add "memory limit reached" event qcow2/overlaps: Add memory usage limit qcow2: Add overlap structure memory size options qapi: Expose new qcow2 overlap check options iotests: Test qcow2's overlap check memory limit block/Makefile.objs | 3 +- block/qcow2-cluster.c | 13 ++ block/qcow2-overlap.c | 572 +++++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 202 +++++++--------- block/qcow2-snapshot.c | 105 ++++++++- block/qcow2.c | 162 +++++++++---- block/qcow2.h | 15 ++ docs/qmp/qmp-events.txt | 28 +++ qapi/block-core.json | 37 +++ qapi/event.json | 27 +++ tests/qemu-iotests/060 | 222 ++++++++++++++++++ tests/qemu-iotests/060.out | 47 ++++ tests/qemu-iotests/group | 2 +- 13 files changed, 1262 insertions(+), 173 deletions(-) create mode 100644 block/qcow2-overlap.c -- 2.3.7