On Fri, Apr 11, 2014 at 11:28:48AM +0200, Martin Steigerwald wrote: > Hi Liu, > > Am Donnerstag, 10. April 2014, 23:55:21 schrieb Liu Bo: > > Hi, > > > > Just FYI, these patches are also available on the following site, > > > > kernel: > > https://github.com/liubogithub/btrfs-work.git dedup-on-3.14-linux > > > > progs: > > https://github.com/liubogithub/btrfs-progs.git dedup > > I bet its good to only test it with test data so far or would you consider it > safe enough to test on production data already?
Definitely at an experimental stage(still has known bugs), that's why I tagged a 'RFC' on it. > > Fortunately since I added that additional mSATA SSD I have some spare storage > to put some test setup into. Thanks for testing it! Thanks, -liubo > > Thanks, > Martin > > > thanks, > > -liubo > > > > On Thu, Apr 10, 2014 at 11:48:30AM +0800, Liu Bo wrote: > > > Hello, > > > > > > This the 10th attempt for in-band data dedupe, based on Linux _3.14_ > > > kernel. > > > > > > Data deduplication is a specialized data compression technique for > > > eliminating duplicate copies of repeating data.[1] > > > > > > This patch set is also related to "Content based storage" in project > > > ideas[2], it introduces inband data deduplication for btrfs and > > > dedup/dedupe is for short. > > > > > > * PATCH 1 is a speed-up improvement, which is about dedup and quota. > > > > > > * PATCH 2-5 is the preparation work for dedup implementation. > > > > > > * PATCH 6 shows how we implement dedup feature. > > > > > > * PATCH 7 fixes a backref walking bug with dedup. > > > > > > * PATCH 8 fixes a free space bug of dedup extents on error handling. > > > > > > * PATCH 9 adds the ioctl to control dedup feature. > > > > > > * PATCH 10 targets delayed refs' scalability problem of deleting refs, > > > which is> > > > uncovered by the dedup feature. > > > > > > * PATCH 11-16 fixes bugs of dedupe including race bug, deadlock, abnormal > > > > > > transaction abortion and crash. > > > > > > * btrfs-progs patch(PATCH 17) offers all details about how to control the > > > > > > dedup feature on progs side. > > > > > > I've tested this with xfstests by adding a inline dedup 'enable & on' in > > > xfstests' mount and scratch_mount. > > > > > > > > > ***NOTE*** > > > Known bugs: > > > * Mounting with options "flushoncommit" and enabling dedupe feature will > > > end up> > > > with _deadlock_. > > > > > > TODO: > > > * a bit-to-bit comparison callback. > > > > > > All comments are welcome! > > > > > > > > > [1]: http://en.wikipedia.org/wiki/Data_deduplication > > > [2]: > > > https://btrfs.wiki.kernel.org/index.php/Project_ideas#Content_based_stora > > > ge > > > > > > v10: > > > - fix a typo in the subject line. > > > - update struct 'btrfs_ioctl_dedup_args' in the kernel side to fix > > > > > > 'Inappropriate ioctl for device'. > > > > > > v9: > > > - fix a deadlock and a crash reported by users. > > > - fix the metadata ENOSPC problem with dedup again. > > > > > > v8: > > > - fix the race crash of dedup ref again. > > > - fix the metadata ENOSPC problem with dedup. > > > > > > v7: > > > - rebase onto the lastest btrfs > > > - break a big patch into smaller ones to make reviewers happy. > > > - kill mount options of dedup and use ioctl method instead. > > > - fix two crash due to the special dedup ref > > > > > > For former patch sets: > > > v6: http://thread.gmane.org/gmane.comp.file-systems.btrfs/27512 > > > v5: http://thread.gmane.org/gmane.comp.file-systems.btrfs/27257 > > > v4: http://thread.gmane.org/gmane.comp.file-systems.btrfs/25751 > > > v3: http://comments.gmane.org/gmane.comp.file-systems.btrfs/25433 > > > v2: http://comments.gmane.org/gmane.comp.file-systems.btrfs/24959 > > > > > > Liu Bo (16): > > > Btrfs: disable qgroups accounting when quota_enable is 0 > > > Btrfs: introduce dedup tree and relatives > > > Btrfs: introduce dedup tree operations > > > Btrfs: introduce dedup state > > > Btrfs: make ordered extent aware of dedup > > > Btrfs: online(inband) data dedup > > > Btrfs: skip dedup reference during backref walking > > > Btrfs: don't return space for dedup extent > > > Btrfs: add ioctl of dedup control > > > Btrfs: improve the delayed refs process in rm case > > > Btrfs: fix a crash of dedup ref > > > Btrfs: fix deadlock of dedup work > > > Btrfs: fix transactin abortion in __btrfs_free_extent > > > Btrfs: fix wrong pinned bytes in __btrfs_free_extent > > > Btrfs: use total_bytes instead of bytes_used for global_rsv > > > Btrfs: fix dedup enospc problem > > > > > > fs/btrfs/backref.c | 9 + > > > fs/btrfs/ctree.c | 2 +- > > > fs/btrfs/ctree.h | 86 ++++++ > > > fs/btrfs/delayed-ref.c | 26 +- > > > fs/btrfs/delayed-ref.h | 3 + > > > fs/btrfs/disk-io.c | 37 +++ > > > fs/btrfs/extent-tree.c | 235 +++++++++++++--- > > > fs/btrfs/extent_io.c | 22 +- > > > fs/btrfs/extent_io.h | 16 ++ > > > fs/btrfs/file-item.c | 244 +++++++++++++++++ > > > fs/btrfs/inode.c | 635 > > > ++++++++++++++++++++++++++++++++++++++----- fs/btrfs/ioctl.c > > > | 167 ++++++++++++ > > > fs/btrfs/ordered-data.c | 44 ++- > > > fs/btrfs/ordered-data.h | 13 +- > > > fs/btrfs/qgroup.c | 3 + > > > fs/btrfs/relocation.c | 3 + > > > fs/btrfs/transaction.c | 41 +++ > > > fs/btrfs/transaction.h | 1 + > > > include/trace/events/btrfs.h | 3 +- > > > include/uapi/linux/btrfs.h | 12 + > > > 20 files changed, 1471 insertions(+), 131 deletions(-) > > -- > Martin 'Helios' Steigerwald - http://www.Lichtvoll.de > GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
