Re: [PATCH v13 00/15] Btrfs In-band De-duplication

2016-10-19 Thread Qu Wenruo



At 10/17/2016 10:30 PM, David Sterba wrote:

On Thu, Sep 08, 2016 at 03:12:49PM +0800, Qu Wenruo wrote:

This patchset can be fetched from github:
https://github.com/adam900710/linux.git wang_dedupe_20160907


Can you please publish the patchset in a branch that does not change
name and is not based on for-next? I'm' going to do less manual merge
tests with Chandan's patchset. A 'dedupe-latest' would work for me,
otherwise feel free to keep the timestamped branches for your needs. The
base branch should be chris' integration (not necessarily the latest
one). Thanks.



Not a problem.

Since now the prepare patches are all in mainline, I can rebase them to 
either mainline non-rc release or Chris' for-linux-4.9.


Thanks,
Qu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v13 00/15] Btrfs In-band De-duplication

2016-10-17 Thread David Sterba
On Thu, Sep 08, 2016 at 03:12:49PM +0800, Qu Wenruo wrote:
> This patchset can be fetched from github:
> https://github.com/adam900710/linux.git wang_dedupe_20160907

Can you please publish the patchset in a branch that does not change
name and is not based on for-next? I'm' going to do less manual merge
tests with Chandan's patchset. A 'dedupe-latest' would work for me,
otherwise feel free to keep the timestamped branches for your needs. The
base branch should be chris' integration (not necessarily the latest
one). Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v13 00/15] Btrfs In-band De-duplication

2016-09-10 Thread Qu Wenruo



On 09/10/2016 08:22 AM, Mark Fasheh wrote:

On Thu, Sep 08, 2016 at 03:12:49PM +0800, Qu Wenruo wrote:

This patchset can be fetched from github:
https://github.com/adam900710/linux.git wang_dedupe_20160907

This version is just another small update, rebased to David's
for-next-20160906 branch.

This updates only includes one small fix, which is exposed by recent
commits which checks space_info->bytes_may_use at umount time.
Caused by that we only free quota reserved space at hash hit, but doesn't
free space_info->bytes_may_use.

Other rebase changes are all related to recent infrastructure change,
like io_tree and quota flags change.

We ran xfstests with dedupe enabled.


Is there an xfstests patch for this I can look at? We want to be able to run
and reproduce the same tests as you.


Any btrfs test which enable dedupe and trigger dedupe will cause kernel 
warning at umount time.
So, no new test cases, either submitted xfstest dedupe test cases, or 
enabling dedupe by modifying xfstests.


We were testing it by modifying _scratch_mount() and _test_mount() to 
enable dedupe for *EVERY* test case.


But now we are adding support to enable quota/dedupe for any test case 
if FSTYP is "btrfs" and environment variant 
"ALWAYS_ENABLE_BTRFS_FEATURE" contains "dedupe".


We will submit such patch to help increase the coverage soon and will CC 
anyone who is interested in inband dedupe and quota.




Also where are the disk portion patches or did I miss them somehow?


Disk format changes are planned for later merge, just as changelog 
version 11 mentioned, we are focusing on the in-memory backend for now.


Only after in-memory backend is stable and merged then we will consider 
other backend.


Thank,
Qu


--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v13 00/15] Btrfs In-band De-duplication

2016-09-09 Thread Mark Fasheh
On Thu, Sep 08, 2016 at 03:12:49PM +0800, Qu Wenruo wrote:
> This patchset can be fetched from github:
> https://github.com/adam900710/linux.git wang_dedupe_20160907
> 
> This version is just another small update, rebased to David's
> for-next-20160906 branch.
> 
> This updates only includes one small fix, which is exposed by recent
> commits which checks space_info->bytes_may_use at umount time.
> Caused by that we only free quota reserved space at hash hit, but doesn't
> free space_info->bytes_may_use.
> 
> Other rebase changes are all related to recent infrastructure change,
> like io_tree and quota flags change.
> 
> We ran xfstests with dedupe enabled.

Is there an xfstests patch for this I can look at? We want to be able to run
and reproduce the same tests as you.

Also where are the disk portion patches or did I miss them somehow?
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


Re: [PATCH v13 00/15] Btrfs In-band De-duplication

2016-09-09 Thread David Sterba
On Thu, Sep 08, 2016 at 03:12:49PM +0800, Qu Wenruo wrote:
> This patchset can be fetched from github:
> https://github.com/adam900710/linux.git wang_dedupe_20160907
> 
> This version is just another small update, rebased to David's
> for-next-20160906 branch.

I've rebased it locally to the 4.9 patch queue and Josef's btree-inode
branch, now pushed to for-next-test. It's really for testing only
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[PATCH v13 00/15] Btrfs In-band De-duplication

2016-09-08 Thread Qu Wenruo
This patchset can be fetched from github:
https://github.com/adam900710/linux.git wang_dedupe_20160907

This version is just another small update, rebased to David's
for-next-20160906 branch.

This updates only includes one small fix, which is exposed by recent
commits which checks space_info->bytes_may_use at umount time.
Caused by that we only free quota reserved space at hash hit, but doesn't
free space_info->bytes_may_use.

Other rebase changes are all related to recent infrastructure change,
like io_tree and quota flags change.

We ran xfstests with dedupe enabled.
While we encountered several bugs, but it's unrelated to dedupe, but
the base branch.

We'll keep digging to fix these non-dedupe bugs.

Changelog:
v2:
  Totally reworked to handle multiple backends
v3:
  Fix a stupid but deadly on-disk backend bug
  Add handle for multiple hash on same bytenr corner case to fix abort
  trans error
  Increase dedup rate by enhancing delayed ref handler for both backend.
  Move dedup_add() to run_delayed_ref() time, to fix abort trans error.
  Increase dedup block size up limit to 8M.
v4:
  Add dedup prop for disabling dedup for given files/dirs.
  Merge inmem_search() and ondisk_search() into generic_search() to save
  some code
  Fix another delayed_ref related bug.
  Use the same mutex for both inmem and ondisk backend.
  Move dedup_add() back to btrfs_finish_ordered_io() to increase dedup
  rate.
v5:
  Reuse compress routine for much simpler dedup function.
  Slightly improved performance due to above modification.
  Fix race between dedup enable/disable
  Fix for false ENOSPC report
v6:
  Further enable/disable race window fix.
  Minor format change according to checkpatch.
v7:
  Fix one concurrency bug with balance.
  Slightly modify return value from -EINVAL to -EOPNOTSUPP for
  btrfs_dedup_ioctl() to allow progs to distinguish unsupported commands
  and wrong parameter.
  Rebased to integration-4.6.
v8:
  Rename 'dedup' to 'dedupe'.
  Add support to allow dedupe and compression work at the same time.
  Fix several balance related bugs. Special thanks to Satoru Takeuchi,
  who exposed most of them.
  Small dedupe hit case performance improvement.
v9:
  Re-order the patchset to completely separate pure in-memory and any
  on-disk format change.
  Fold bug fixes into its original patch.
v10:
  Adding back missing bug fix patch.
  Reduce on-disk item size.
  Hide dedupe ioctl under CONFIG_BTRFS_DEBUG.
v11:
  Remove other backend and props support to focus on the framework and
  in-memory backend. Suggested by David.
  Better disable and buffered write race protection.
  Comprehensive fix to dedupe metadata ENOSPC problem.
v12:
  Stateful 'enable' ioctl and new 'reconf' ioctl
  New FORCE flag for enable ioctl to allow stateless ioctl
  Precise error report and extendable ioctl structure.
v12.1
  Rebase to David's for-next-20160704 branch
  Add co-ordinate patch for subpage and dedupe patchset. 
v12.2
  Rebase to David's for-next-20160715 branch
  Add co-ordinate patch for other patchset.
v13
  Rebase to David's for-next-20160906 branch
  Fix a reserved space leak bug, which only frees quota reserved space
  but not space_info->byte_may_use.

Qu Wenruo (5):
  btrfs: expand btrfs_set_extent_delalloc() and its friends to support
in-band dedupe and subpage size patchset
  btrfs: delayed-ref: Add support for increasing data ref under spinlock
  btrfs: dedupe: Inband in-memory only de-duplication implement
  btrfs: relocation: Enhance error handling to avoid BUG_ON
  btrfs: dedupe: Introduce new reconfigure ioctl

Wang Xiaoguang (10):
  btrfs: dedupe: Introduce dedupe framework and its header
  btrfs: dedupe: Introduce function to initialize dedupe info
  btrfs: dedupe: Introduce function to add hash into in-memory tree
  btrfs: dedupe: Introduce function to remove hash from in-memory tree
  btrfs: dedupe: Introduce function to search for an existing hash
  btrfs: dedupe: Implement btrfs_dedupe_calc_hash interface
  btrfs: ordered-extent: Add support for dedupe
  btrfs: dedupe: Add ioctl for inband dedupelication
  btrfs: improve inode's outstanding_extents computation
  btrfs: dedupe: fix false ENOSPC

 fs/btrfs/Makefile|   2 +-
 fs/btrfs/ctree.h |  25 +-
 fs/btrfs/dedupe.c| 820 +++
 fs/btrfs/dedupe.h| 201 +-
 fs/btrfs/delayed-ref.c   |  30 +-
 fs/btrfs/delayed-ref.h   |   8 +
 fs/btrfs/disk-io.c   |   4 +
 fs/btrfs/extent-tree.c   |  82 +++-
 fs/btrfs/extent_io.c |  65 +++-
 fs/btrfs/extent_io.h |  17 +-
 fs/btrfs/file.c  |  26 +-
 fs/btrfs/free-space-cache.c  |   5 +-
 fs/btrfs/inode-map.c |   4 +-
 fs/btrfs/inode.c | 463 +-
 fs/btrfs/ioctl.c |  93 -
 fs/btrfs/ordered-data.c  |  46 ++-
 fs/btrfs/ordered-data.h  |  14 +