Re: [PATCH] Btrfs: remove redundant btrfs_trans_release_metadata"

On 5.09.2018 04:14, Liu Bo wrote: > __btrfs_end_transaction() has done the metadata release twice, > probably because it used to process delayed refs in between, but now > that we don't process delayed refs any more, the 2nd release is always > a noop. > > Signed-off-by: Liu Bo Reviewed-by:

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

On 2018/9/5 下午1:42, Nikolay Borisov wrote: > > > On 5.09.2018 05:10, Qu Wenruo wrote: >> >> >> On 2018/8/16 下午9:10, Nikolay Borisov wrote: >>> This commit enables the delayed refs infrastructures. This entails doing >>> the following: >>> >>> 1. Replacing existing calls of

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

On 5.09.2018 05:10, Qu Wenruo wrote: > > > On 2018/8/16 下午9:10, Nikolay Borisov wrote: >> This commit enables the delayed refs infrastructures. This entails doing >> the following: >> >> 1. Replacing existing calls of btrfs_extent_post_op (which is the >> equivalent of delayed refs) with the

[PATCH] btrfs: qgroup: Don't trace subtree if we're dropping tree reloc tree

Tree reloc tree doesn't contribute to qgroup numbers, as we have accounted them at balance time (check replace_path()). Skip such unneeded subtree trace should reduce some performance overhead. Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 8 +++- 1 file changed, 7 insertions(+), 1

[PATCH] btrfs: defrag: use btrfs_mod_outstanding_extents in cluster_pages_for_defrag

Since commit 8b62f87bad9c ("Btrfs: rework outstanding_extents"), manual operations of outstanding_extent in btrfs_inode are replaced by btrfs_mod_outstanding_extents(). The one in cluster_pages_for_defrag seems to be lost, so replace it of btrfs_mod_outstanding_extents(). Fixes: 8b62f87bad9c

Re: [PATCH 5/8] btrfs-progs: Wire up delayed refs

On 2018/8/16 下午9:10, Nikolay Borisov wrote: > This commit enables the delayed refs infrastructures. This entails doing > the following: > > 1. Replacing existing calls of btrfs_extent_post_op (which is the > equivalent of delayed refs) with the proper btrfs_run_delayed_refs. > As well as

[PATCH v2] Btrfs: remove confusing tracepoint in btrfs_add_reserved_bytes

Here we're not releasing any space, but transferring bytes from ->bytes_may_use to ->bytes_reserved. Signed-off-by: Liu Bo --- v2: Add missing commit log. fs/btrfs/extent-tree.c | 4 1 file changed, 4 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index

nbdkit as a flexible alternative to loopback mounts

https://rwmj.wordpress.com/2018/09/04/nbdkit-as-a-flexible-alternative-to-loopback-mounts/ This is a pretty cool writeup. I can vouch Btrfs will format mount, write to, scrub, and btrfs check works on an 8EiB (virtual) disk. The one thing I thought might cause a problem is the ndb device has a

Re: [PATCH 20/35] btrfs: reset max_extent_size on clear in a bitmap

On Thu, Aug 30, 2018 at 10:42 AM, Josef Bacik wrote: > From: Josef Bacik > > We need to clear the max_extent_size when we clear bits from a bitmap > since it could have been from the range that contains the > max_extent_size. > Looks OK. Reviewed-by: Liu Bo thanks, liubo > Signed-off-by:

Re: [PATCH 08/35] btrfs: release metadata before running delayed refs

On Thu, Aug 30, 2018 at 10:41 AM, Josef Bacik wrote: > We want to release the unused reservation we have since it refills the > delayed refs reserve, which will make everything go smoother when > running the delayed refs if we're short on our reservation. > Looks good. Reviewed-by: Liu Bo

[PATCH] Btrfs: remove confusing tracepoint in btrfs_add_reserved_bytes

Signed-off-by: Liu Bo --- fs/btrfs/extent-tree.c | 4 1 file changed, 4 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 41a02cbb5a4a..76ee5ebef2b9 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -6401,10 +6401,6 @@ static int

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

On 2018/9/5 上午4:37, Chris Murphy wrote: > On Tue, Sep 4, 2018 at 10:22 AM, Etienne Champetier > wrote: > >> Do you have a procedure to copy all subvolumes & skip error ? (I have >> ~200 snapshots) > > If they're already read-only snapshots, then script an iteration of > btrfs send receive to

[PATCH] Btrfs: remove redundant btrfs_trans_release_metadata"

__btrfs_end_transaction() has done the metadata release twice, probably because it used to process delayed refs in between, but now that we don't process delayed refs any more, the 2nd release is always a noop. Signed-off-by: Liu Bo --- fs/btrfs/transaction.c | 6 -- 1 file changed, 6

Re: [PATCH 02/35] btrfs: add cleanup_ref_head_accounting helper

On Thu, Aug 30, 2018 at 10:41 AM, Josef Bacik wrote: > From: Josef Bacik > > We were missing some quota cleanups in check_ref_cleanup, so break the > ref head accounting cleanup into a helper and call that from both > check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that > we

[PATCH v5 2/2] vfs: dedupe should return EPERM if permission is not granted

Right now we return EINVAL if a process does not have permission to dedupe a file. This was an oversight on my part. EPERM gives a true description of the nature of our error, and EINVAL is already used for the case that the filesystem does not support dedupe. Signed-off-by: Mark Fasheh

[PATCH v5 1/2] vfs: allow dedupe of user owned read-only files

The permission check in vfs_dedupe_file_range() is too coarse - We only allow dedupe of the destination file if the user is root, or they have the file open for write. This effectively limits a non-root user from deduping their own read-only files. In addition, the write file descriptor that the

[RESEND][PATCH v5 0/2] vfs: fix dedupe permission check

Hi Andrew/Al, Could I please have these patches put in a tree for more public testing? They've hit fsdevel a few times now, I have links to the discussions in the change log below. The following patches fix a couple of issues with the permission check we do in vfs_dedupe_file_range(). The

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

On Tue, Sep 4, 2018 at 10:22 AM, Etienne Champetier wrote: > Do you have a procedure to copy all subvolumes & skip error ? (I have > ~200 snapshots) If they're already read-only snapshots, then script an iteration of btrfs send receive to a new volume. Btrfs seed-sprout would be ideal, however

WTS: Panasonic Toughbook

Hello, We have Grade A Panasonic Toughbook, tested working in good condition and looks like new. Grade A cf-c2ccezxcm -Panasonic Toughbook CF-C2 Convertible Laptop Intel Core i5-4300U 1.90GHz CPU 8GB RAM 128GB HDD Win 8.1 Pro. QTY 52 @ $110 each. Contact us if you are interested, pictures are

Re: [PATCH 12/35] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

On 4.09.2018 20:57, Josef Bacik wrote: > On Mon, Sep 03, 2018 at 05:19:19PM +0300, Nikolay Borisov wrote: >> >> >> On 30.08.2018 20:42, Josef Bacik wrote: >>> + if (flush_state == ALLOC_CHUNK_FORCE && !commit_cycles) >>> + flush_state++; >> >> This is a bit obscure.

Re: [PATCH 05/35] btrfs: introduce delayed_refs_rsv

On Tue, Sep 04, 2018 at 06:21:23PM +0300, Nikolay Borisov wrote: > > > On 30.08.2018 20:41, Josef Bacik wrote: > > From: Josef Bacik > > > > Traditionally we've had voodoo in btrfs to account for the space that > > delayed refs may take up by having a global_block_rsv. This works most > > of

Re: [PATCH 21/35] btrfs: only run delayed refs if we're committing

On Tue, Sep 04, 2018 at 01:54:13PM -0400, Josef Bacik wrote: > On Fri, Aug 31, 2018 at 05:28:09PM -0700, Omar Sandoval wrote: > > On Thu, Aug 30, 2018 at 01:42:11PM -0400, Josef Bacik wrote: > > > I noticed in a giant dbench run that we spent a lot of time on lock > > > contention while running

Re: [PATCH 12/35] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

On Mon, Sep 03, 2018 at 05:19:19PM +0300, Nikolay Borisov wrote: > > > On 30.08.2018 20:42, Josef Bacik wrote: > > + if (flush_state == ALLOC_CHUNK_FORCE && !commit_cycles) > > + flush_state++; > > This is a bit obscure. So if we allocated a chunk and !commit_cycles

Re: [PATCH 21/35] btrfs: only run delayed refs if we're committing

On Fri, Aug 31, 2018 at 05:28:09PM -0700, Omar Sandoval wrote: > On Thu, Aug 30, 2018 at 01:42:11PM -0400, Josef Bacik wrote: > > I noticed in a giant dbench run that we spent a lot of time on lock > > contention while running transaction commit. This is because dbench > > results in a lot of

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

Thanks Qu, one last question I think Le mar. 4 sept. 2018 à 08:33, Qu Wenruo a écrit : > > On 2018/9/4 下午7:53, Etienne Champetier wrote: > > Hi Qu, > > > > Le lun. 3 sept. 2018 à 20:27, Qu Wenruo a écrit : > >> > >> On 2018/9/3 下午10:18, Etienne Champetier wrote: > >>> Hello btfrs hackers, > >>>

Re: [PATCH 05/35] btrfs: introduce delayed_refs_rsv

On 30.08.2018 20:41, Josef Bacik wrote: > From: Josef Bacik > > Traditionally we've had voodoo in btrfs to account for the space that > delayed refs may take up by having a global_block_rsv. This works most > of the time, except when it doesn't. We've had issues reported and seen > in

[PATCH] btrfs-progs: calibrate extent_end when found a gap

The extent_end will be used to check whether there is gap between this extent and next extent. If it is not calibrated, check_file_extent will mistake that there are gaps between the remaining extents. Signed-off-by: Lu Fengqi --- check/mode-lowmem.c | 1 + 1 file changed, 1 insertion(+) diff

[PATCH] btrfs-progs: Continue checking even we found something wrong in free space cache

No need to abort checking, especially for RO check free space cache is meaningless, the errors in fs/extent tree is more interesting for developers. So continue checking even something in free space cache is wrong. Reported-by: Etienne Champetier Signed-off-by: Qu Wenruo --- check/main.c | 1

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

On 2018/8/23 下午3:45, Qu Wenruo wrote: > > > On 2018/8/23 下午3:36, Nikolay Borisov wrote: >> >> >> On 23.08.2018 10:31, Qu Wenruo wrote: >>> Introduce --breadth-first option to do breadth-first tree dump. >>> This is especially handy to inspect high level trees, e.g. comparing >>> tree reloc

Re: RAID1 & BTRFS critical (device sda2): corrupt leaf, bad key order

On 2018/9/4 下午7:53, Etienne Champetier wrote: > Hi Qu, > > Le lun. 3 sept. 2018 à 20:27, Qu Wenruo a écrit : >> >> On 2018/9/3 下午10:18, Etienne Champetier wrote: >>> Hello btfrs hackers, >>> >>> I have a computer acting as backup server with BTRFS RAID1, and I >>> would like to know the

Re: fsck lowmem mode only: ERROR: errors found in fs roots

On Tue, 2018-09-04 at 17:14 +0800, Qu Wenruo wrote: > However the backtrace can't tell which process caused such fsync > call. > (Maybe LVM user space code?) Well it was just literally before btrfs-check exited... so I blindly guesses... but arguably it could be just some coincidence. LVM tools

Re: fsck lowmem mode only: ERROR: errors found in fs roots

On 2018/9/4 上午4:24, Christoph Anton Mitterer wrote: > Hey. > > > On Fri, 2018-08-31 at 10:33 +0800, Su Yue wrote: >> Can you please fetch btrfs-progs from my repo and run lowmem check >> in readonly? >> Repo: https://github.com/Damenly/btrfs-progs/tree/lowmem_debug >> It's based on v4.17.1

[PATCH v10.5 2/5] btrfs-progs: dedupe: Add enable command for dedupe command group

From: Qu Wenruo Add enable subcommand for dedupe commmand group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- Documentation/btrfs-dedupe-inband.asciidoc | 114 +- btrfs-completion | 6 +- cmds-dedupe-ib.c | 238

[PATCH v10.5 5/5] btrfs-progs: dedupe: introduce reconfigure subcommand

From: Qu Wenruo Introduce reconfigure subcommand to co-operate with new kernel ioctl modification. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- Documentation/btrfs-dedupe-inband.asciidoc | 7 +++ btrfs-completion | 2 +- cmds-dedupe-ib.c

[PATCH v10.5 3/5] btrfs-progs: dedupe: Add disable support for inband dedupelication

From: Qu Wenruo Add disable subcommand for dedupe command group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- Documentation/btrfs-dedupe-inband.asciidoc | 5 +++ btrfs-completion | 2 +- cmds-dedupe-ib.c | 41

[PATCH v10.5 4/5] btrfs-progs: dedupe: Add status subcommand

From: Qu Wenruo Add status subcommand for dedupe command group. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- Documentation/btrfs-dedupe-inband.asciidoc | 3 + btrfs-completion | 2 +- cmds-dedupe-ib.c | 80 ++

[PATCH v10.5 0/5] In-band de-duplication for btrfs-progs

Patchset can be fetched from github: https://github.com/littleroad/btrfs-progs.git dedupe_latest Inband dedupe(in-memory backend only) ioctl support for btrfs-progs. v7 changes: Update ctree.h to follow kernel structure change Update print-tree to follow kernel structure change V8 changes:

[PATCH v10.5 1/5] btrfs-progs: Basic framework for dedupe-inband command group

From: Qu Wenruo Add basic ioctl header and command group framework for later use. Alone with basic man page doc. Signed-off-by: Qu Wenruo Signed-off-by: Lu Fengqi --- Documentation/Makefile.in | 1 + Documentation/btrfs-dedupe-inband.asciidoc | 40 ++

[PATCH v15 09/13] btrfs: introduce type based delalloc metadata reserve

From: Wang Xiaoguang Introduce type based metadata reserve parameter for delalloc space reservation/freeing function. The problem we are going to solve is, btrfs use different max extent size for different mount options. For de-duplication, the max extent size can be set by the dedupe ioctl,

[PATCH v15 03/13] btrfs: dedupe: Introduce function to add hash into in-memory tree

From: Wang Xiaoguang Introduce static function inmem_add() to add hash into in-memory tree. And now we can implement the btrfs_dedupe_add() interface. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik Signed-off-by: Lu Fengqi --- fs/btrfs/dedupe.c | 150

[PATCH v15 00/13] Btrfs In-band De-duplication

This patchset can be fetched from github: https://github.com/littleroad/linux.git dedupe_latest Now the new base is v4.19-rc2, and drop the patch about compression which conflict with compression heuristic. Normal test cases from auto group exposes no regression, and ib-dedupe group can pass

[PATCH v15 13/13] btrfs: dedupe: Introduce new reconfigure ioctl

From: Qu Wenruo Introduce new reconfigure ioctl and new FORCE flag for in-band dedupe ioctls. Now dedupe enable and reconfigure ioctl are stateful. | Current state | Ioctl| Next state | | Disabled

[PATCH v15 12/13] btrfs: relocation: Enhance error handling to avoid BUG_ON

From: Qu Wenruo Since the introduction of btrfs dedupe tree, it's possible that balance can race with dedupe disabling. When this happens, dedupe_enabled will make btrfs_get_fs_root() return PTR_ERR(-ENOENT). But due to a bug in error handling branch, when this happens backref_cache->nr_nodes

[PATCH v15 07/13] btrfs: dedupe: Implement btrfs_dedupe_calc_hash interface

From: Wang Xiaoguang Unlike in-memory or on-disk dedupe method, only SHA256 hash method is supported yet, so implement btrfs_dedupe_calc_hash() interface using SHA256. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik Signed-off-by: Lu Fengqi ---

[PATCH v15 11/13] btrfs: dedupe: Add ioctl for inband deduplication

From: Wang Xiaoguang Add ioctl interface for inband deduplication, which includes: 1) enable 2) disable 3) status And a pseudo RO compat flag, to imply that btrfs now supports inband dedup. However we don't add any ondisk format change, it's just a pseudo RO compat flag. All these ioctl

[PATCH v15 04/13] btrfs: dedupe: Introduce function to remove hash from in-memory tree

From: Wang Xiaoguang Introduce static function inmem_del() to remove hash from in-memory dedupe tree. And implement btrfs_dedupe_del() and btrfs_dedup_disable() interfaces. Also for btrfs_dedupe_disable(), add new functions to wait existing writer and block incoming writers to eliminate all

[PATCH v15 02/13] btrfs: dedupe: Introduce function to initialize dedupe info

From: Wang Xiaoguang Add generic function to initialize dedupe info. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik Signed-off-by: Lu Fengqi --- fs/btrfs/Makefile | 2 +- fs/btrfs/dedupe.c | 169 +

[PATCH v15 10/13] btrfs: dedupe: Inband in-memory only de-duplication implement

From: Qu Wenruo Core implement for inband de-duplication. It reuses the async_cow_start() facility to do the calculate dedupe hash. And use dedupe hash to do inband de-duplication at extent level. The workflow is as below: 1) Run delalloc range for an inode 2) Calculate hash for the delalloc

[PATCH v15 05/13] btrfs: delayed-ref: Add support for increasing data ref under spinlock

From: Qu Wenruo For in-band dedupe, btrfs needs to increase data ref with delayed_ref locked, so add a new function btrfs_add_delayed_data_ref_lock() to increase extent ref with delayed_refs already locked. Export init_delayed_ref_head and init_delayed_ref_common for inband dedupe.

[PATCH v15 08/13] btrfs: ordered-extent: Add support for dedupe

From: Wang Xiaoguang Add ordered-extent support for dedupe. Note, current ordered-extent support only supports non-compressed source extent. Support for compressed source extent will be added later. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik ---

[PATCH v15 01/13] btrfs: dedupe: Introduce dedupe framework and its header

From: Wang Xiaoguang Introduce the header for btrfs in-band(write time) de-duplication framework and needed header. The new de-duplication framework is going to support 2 different dedupe methods and 1 dedupe hash. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Signed-off-by: Lu

[PATCH v15 06/13] btrfs: dedupe: Introduce function to search for an existing hash

From: Wang Xiaoguang Introduce static function inmem_search() to handle the job for in-memory hash tree. The trick is, we must ensure the delayed ref head is not being run at the time we search the for the hash. With inmem_search(), we can implement the btrfs_dedupe_search() interface.