This 3 step patchset implements deduplication in QCOW2. First patchset create the core infrastructure for deduplication and enable it in QCOW2 image. It ends at "qcow2: Enable the deduplication feature."
Second patchset implements some metrics in QMP. It ends at "qapi: Return virtual block device deduplication metrics in QMP" Third patchset implements asynchronous deduplication. It's a work in progress patchset that is included in this post so reviewers can have a grasp of where the feature is heading. One can compile and install https://github.com/wernerd/Skein3Fish and use the --enable-skein-dedup configure option in order to use the faster skein HASH. Images must be created with "-o dedup=[skein|sha256]" in order to activate the deduplication in the image. Deduplication is now fast enough to be usable. Nice side effect is that duplicated writes are faster than native QCOW2: v5: Move qemu-io-test dedup patch [Eric] Reserve some room at the end of the QCOW header extensions. [Eric] Fix the specification. [Eric] Now overflow deduplication refcount at 2^16/2 [Stefan] Implements metrics. Implement asynchronous deduplication. Increase L2 table size and deduplication block hash size. Random cleanups v4: Fix and complete qcow2 spec [Stefan] Hash the hash_algo field in the header extension [Stefan] Fix qcow2 spec [Eric] Remove pointer to hash and simplify hash memory management [Stefan] Rename and move qcow2_read_cluster_data to qcow2.c [Stefan] Document lock dropping behaviour of the previous function [Stefan] cleanup qcow2_dedup_read_missing_cluster_data [Stefan] rename *_offset to *_sect [Stefan] add a ./configure check for ssl [Stefan] Replace openssl by gnutls [Stefan] Implement Skein hashes Rewrite pretty every qcow2-dedup.c commits after Add qcow2_dedup_read_missing_and_concatenate to simplify the code Use 64KB deduplication hash block to reduce allocation flushes Use 64KB l2 tables to reduce allocation flushes [breaks compatibility] Use lazy refcounts to avoid qcow2_cache_set_dependency loops resultings in frequent caches flushes Do not create and load dedup RAM structures when bdrs->read_only is true v3: make it work barely replace kernel red black trees by gtree. Benoît Canet (62): qcow2: Add deduplication to the qcow2 specification. qcow2: Add deduplication structures and fields. qcow2: Add qcow2_dedup_read_missing_and_concatenate qcow2: Make update_refcount public. qcow2: Create a way to link to l2 tables when deduplicating. qcow2: Add qcow2_dedup and related functions qcow2: Add qcow2_dedup_store_new_hashes. qcow2: Implement qcow2_compute_cluster_hash. qcow2: Extract qcow2_dedup_grow_table qcow2: Add qcow2_dedup_grow_table and use it. qcow2: Makes qcow2_alloc_cluster_link_l2 mark to deduplicate clusters. qcow2: make the deduplication forget a cluster hash when a cluster is to dedupe qcow2: Create qcow2_is_cluster_to_dedup. qcow2: Load and save deduplication table header extension. qcow2: Extract qcow2_do_table_init. qcow2-cache: Allow to choose table size at creation. qcow2: Extract qcow2_add_feature and qcow2_remove_feature. block: Add qemu-img dedup create option. qcow2: Add a deduplication boolean to update_refcount. qcow2: Drop hash for a given cluster when dedup makes refcount > 2^16/2. qcow2: Remove hash when cluster is deleted. qcow2: Add qcow2_dedup_is_running to probe if dedup is running. qcow2: Integrate deduplication in qcow2_co_writev loop. qcow2: Serialize write requests when deduplication is activated. qcow2: Add verification of dedup table. qcow2: Adapt checking of QCOW_OFLAG_COPIED for dedup. qcow2: Add check_dedup_l2 in order to check l2 of dedup table. qcow2: Do not overwrite existing entries with QCOW_OFLAG_COPIED. qcow2: Integrate SKEIN hash algorithm in deduplication. qcow2: Add lazy refcounts to deduplication to prevent qcow2_cache_set_dependency loops qcow2: Use large L2 table for deduplication. qcow: Set large dedup hash block size. qemu-iotests: Filter dedup=on/off so existing tests don't break. qcow2: Add qcow2_dedup_init and qcow2_dedup_close. qcow2: Add qcow2_co_dedup_resume to restart deduplication. qcow2: Enable the deduplication feature. qcow2: Add deduplication metrics structures. qcow2: Initialize deduplication metrics. qcow2: Collect unaligned writes missing data reads metric. qcow2: Collect deduplicated cluster metric. qcow2: Collect undeduplicated cluster metric. qcow2: Count QCowHashNode creation metrics. qcow2: Count QCowHashNode removal from tree for metrics. qcow2: Count cluster deleted metric qcow2: Count deduplication refcount overflow metric. qapi: Add support for deduplication infos in qapi-schema.json. block: Add deduplication metrics to BlockDriverInfo. qcow2: Add qcow2_dedup_update_metrics to compute dedup RAM usage. qcow2: returns deduplication metrics and status via bdrv_get_info() qapi: Return virtual block device deduplication metrics in QMP block: Add BlockDriver function prototype to pause and resume deduplication. qcow2: Add code to deduplicate cluster flagged with QCOW_OFLAG_TO_DEDUP. block: Add bdrv_has_dedup. block: Add bdrv_is_dedup_running. block: Add bdrv_resume_dedup. block: Add bdrv_pause_dedup. qcow2: Add qcow2_pause_dedup. qcow2: Add qcow2_resume_dedup. qcow2: Make dedup status persists. qerror: Add QERR_DEVICE_NOT_DEDUPLICATED. qmp: Add block-pause-dedup. qmp: Add block_resume_dedup. block.c | 108 +++ block/Makefile.objs | 1 + block/qcow2-cache.c | 12 +- block/qcow2-cluster.c | 182 ++++-- block/qcow2-dedup.c | 1492 ++++++++++++++++++++++++++++++++++++++++++ block/qcow2-refcount.c | 175 +++-- block/qcow2.c | 378 +++++++++-- block/qcow2.h | 149 ++++- blockdev.c | 36 + configure | 55 ++ docs/specs/qcow2.txt | 104 ++- include/block/block.h | 18 + include/block/block_int.h | 5 + include/qapi/qmp/qerror.h | 3 + qapi-schema.json | 76 ++- qmp-commands.hx | 46 ++ tests/qemu-iotests/common.rc | 3 +- 17 files changed, 2708 insertions(+), 135 deletions(-) create mode 100644 block/qcow2-dedup.c -- 1.7.10.4
