Following the discussion of these two patches in the April OpenZFS Leadership meeting, Matt suggested I solicit additional review from this list.
The first patch: Dedup DDT load https://github.com/zfsonlinux/zfs/pull/9464/ Adds a new zpool command: `zpool prefetch -t ddt <poolname>` that basically loads the entire DDT into memory. This main use case of this is customers with large DDTs, after a reboot experience extremely pool write speeds because of random reads from the DDT. It often took days for the majority of the DDT ZAP to be cached in the ARC. With this change, after a reboot the entire DDT can be read quickly into the ARC and restore performance. It extends `zpool status -D` such that specifying -DD returns more detail, including the % of the DDT that is currently in the ARC. This calculation can be a little slow, that is why it is not the default. The second patch is here: Dedup Ceiling https://github.com/openzfs/zfs/pull/10169 This adds a new pair of new properties: dedup_table_quota - To limit the on-disk size of the DDT dedup_table_size - [readonly] the on-disk size of the DDT Once the quota is reached, newly written blocks that do not match an existing hash in the DDT are written as if dedup=off. Blocks that match the hash of an existing entry in the DDT will still dedup (by incrementing the refcount). This allows the administrator to avoid overflowing a special or dedup class vdev, and provides some measure of control over the memory footprint of dedup (generally somewhere between 1.4x and 3x less than the on-disk size of the DDT, depending on hash fill rates and compression etc) As discussed in the meeting, we think both of these concepts will still apply to future iterations of dedup, such as the one Panzura is hoping to offer soon. The goal is to make sure the naming of these features, commands, and properties will still make sense if their implementations are replaced with different ones in an eventual enhancement (ddt log) or replacement (panzura dedup) of the dedup feature. Thank you. -- Allan Jude -- VP Engineering ** Klara Inc. | al...@klarasystems.com | +1 289-260-5944 ** ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/T22f59df09864c0fb-Mb3064411930dcbe9c14fe60d Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
