Re: Recovery from full metadata with all device space consumed?

On Thu, Apr 19, 2018 at 04:12:39PM -0700, Drew Bloechl wrote: > On Thu, Apr 19, 2018 at 10:43:57PM +, Hugo Mills wrote: > >Given that both data and metadata levels here require paired > > chunks, try adding _two_ temporary devices so that it can allocate a > > new block group. > > Thank

Re: Recovery from full metadata with all device space consumed?

On Thu, Apr 19, 2018 at 10:43:57PM +, Hugo Mills wrote: >Given that both data and metadata levels here require paired > chunks, try adding _two_ temporary devices so that it can allocate a > new block group. Thank you very much, that seems to have done the trick: # fallocate -l 4GiB

Re: Recovery from full metadata with all device space consumed?

On Thu, Apr 19, 2018 at 03:08:48PM -0700, Drew Bloechl wrote: > I've got a btrfs filesystem that I can't seem to get back to a useful > state. The symptom I started with is that rename() operations started > dying with ENOSPC, and it looks like the metadata allocation on the > filesystem is full:

Re: Recovery from full metadata with all device space consumed?

2018-04-20 1:08 GMT+03:00 Drew Bloechl : > I've got a btrfs filesystem that I can't seem to get back to a useful > state. The symptom I started with is that rename() operations started > dying with ENOSPC, and it looks like the metadata allocation on the > filesystem is full: >

Recovery from full metadata with all device space consumed?

I've got a btrfs filesystem that I can't seem to get back to a useful state. The symptom I started with is that rename() operations started dying with ENOSPC, and it looks like the metadata allocation on the filesystem is full: # btrfs fi df /broken Data, RAID0: total=3.63TiB, used=67.00GiB

Re: Hard link not persisted on fsync

On Mon, Apr 16, 2018 at 09:35:24AM -0500, Jayashree Mohan wrote: > Hi, > > The following seems to be a crash consistency bug on btrfs, where in > the link count is not persisted even after a fsync on the original > file. > > Consider the following workload : > creat foo > link (foo, A/bar) >

[PATCH v2 09/16] btrfs: cleanup helpers that reset balance state

The function __cancel_balance name is confusing with the cancel operation of balance and it really resets the state of balance back to zero. The unset_balance_control helper is called only from one place and simple enough to be inlined. Signed-off-by: David Sterba ---

[PATCH v2 12/16] btrfs: track running balance in a simpler way

Currently fs_info::balance_running is 0 or 1 and does not use the semantics of atomics. The pause and cancel check for 0, that can happen only after __btrfs_balance exits for whatever reason. Parallel calls to balance ioctl may enter btrfs_ioctl_balance multiple times but will block on the

[PATCH v2 16/16] btrfs: open code set_balance_control

The helper is quite simple and I'd like to see the locking in the caller. Reviewed-by: Anand Jain Signed-off-by: David Sterba --- fs/btrfs/volumes.c | 25 - 1 file changed, 8 insertions(+), 17 deletions(-) diff --git

[PATCH v2 13/16] btrfs: move and comment read-only check in btrfs_cancel_balance

Balance cannot be started on a read-only filesystem and will have to finish/exit before eg. going to read-only via remount. In case the filesystem is forcibly set to read-only after an error, balance will finish anyway and if the cancel call is too fast it will just wait for that to happen. The

[PATCH v2 15/16] btrfs: use mutex in btrfs_resume_balance_async

While the spinlock does not cause problems, using the mutex is more correct and consistent with others. The global status of balance is eg. checked from btrfs_pause_balance or btrfs_cancel_balance with mutex. Resuming balance happens during mount or ro->rw remount. In the former case, no other

[PATCH v2 08/16] btrfs: add sanity check when resuming balance after mount

Replace a WARN_ON with a proper check and message in case something goes really wrong and resumed balance cannot set up its exclusive status. The check is a user friendly assertion, I don't expect to ever happen under normal circumstances. Also document that the paused balance starts here and

[PATCH v2 11/16] btrfs: kill btrfs_fs_info::volume_mutex

Mutual exclusion of device add/rm and balance was done by the volume mutex up to version 3.7. The commit 5ac00addc7ac091109 ("Btrfs: disallow mutually exclusive admin operations from user mode") added a bit that essentially tracked the same information. The status bit has an advantage over a

[PATCH v2 10/16] btrfs: remove wrong use of volume_mutex from btrfs_dev_replace_start

The volume mutex does not protect against anything in this case, the comment about scrub is right but not related to locking and looks confusing. The comment in btrfs_find_device_missing_or_by_path is wrong and confusing too. The device_list_mutex is not held here to protect device lookup, but in

[PATCH v2 14/16] btrfs: drop lock parameter from update_ioctl_balance_args and rename

The parameter controls locking of the stats part but we can lock it unconditionally, as this only happens once when balance starts. This is not performance critical. Add the prefix for an exported function. Signed-off-by: David Sterba --- fs/btrfs/ctree.h | 2 +-

[PATCH v2 07/16] btrfs: add proper safety check before resuming dev-replace

The device replace is paused by unmount or read only remount, and resumed on next mount or write remount. The exclusive status should be checked properly as it's a global invariant and we must not allow 2 operations run. In this case, the balance can be also paused and resumed under same

[PATCH v2 06/16] btrfs: move clearing of EXCL_OP out of __cancel_balance

Make the clearning visible in the callers so we can pair it with the test_and_set part. Signed-off-by: David Sterba --- fs/btrfs/ioctl.c | 2 +- fs/btrfs/volumes.c | 13 +++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/fs/btrfs/ioctl.c

[PATCH v2 05/16] btrfs: move volume_mutex to callers of btrfs_rm_device

Move locking and unlocking next to the BTRFS_FS_EXCL_OP bit manipulation so it's obvious that the two happen at the same time. Reviewed-by: Anand Jain Signed-off-by: David Sterba --- fs/btrfs/ioctl.c | 4 fs/btrfs/volumes.c | 2 -- 2 files

[PATCH v2 01/16] btrfs: squeeze btrfs_dev_replace_continue_on_mount to its caller

The function is called once and is fairly small, we can merge it with the caller. Reviewed-by: Anand Jain Signed-off-by: David Sterba --- fs/btrfs/dev-replace.c | 16 +++- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git

[PATCH v2 04/16] btrfs: move btrfs_init_dev_replace_tgtdev to dev-replace.c and make static

The function logically belongs there and there's only a single caller, no need to export it. No code changes. Reviewed-by: Anand Jain Signed-off-by: David Sterba --- fs/btrfs/dev-replace.c | 99 ++

[PATCH v2 03/16] btrfs: export and rename free_device

The function will be used outside of volumes.c, the allocation btrfs_alloc_device is also exported. Reviewed-by: Anand Jain Signed-off-by: David Sterba --- fs/btrfs/volumes.c | 24 fs/btrfs/volumes.h | 1 + 2 files changed, 13

[PATCH v2 00/16] Kill fs_info::volume_mutex

This series gets rid of the volume mutex because it's redundant. Updated branch: git://github.com/kdave/btrfs-devel dev/remove-volume-mutex Changes for v2: - sanity check in balance resume is only a warning - read-only check in balance cancel remains and is only moved - typo fixes The fstests

[PATCH v2 02/16] btrfs: make success path out of btrfs_init_dev_replace_tgtdev more clear

This is a preparatory cleanup that will make clear that the only successful way out of btrfs_init_dev_replace_tgtdev will also set the device_out to a valid pointer. With this guarantee, the callers can be simplified. Reviewed-by: Anand Jain Signed-off-by: David Sterba

Re: [PATCH 1/3] btrfs: Add incompat flags check for btrfs_check_super_valid()

On 19.04.2018 18:31, David Sterba wrote: > On Thu, Apr 19, 2018 at 07:10:30PM +0800, Qu Wenruo wrote: >> >> >> On 2018年04月19日 18:59, Nikolay Borisov wrote: >>> >>> >>> On 19.04.2018 12:38, Qu Wenruo wrote: Although we have already checked incompat flags manually before really mounting

status page

Hi to all, as kernel 4.16 is out and 4.17 in in RC, would be possible to update BTRFS status page https://btrfs.wiki.kernel.org/index.php/Status to reflect 4.16 stability ? That page is still based on kernel 4.15 (marked as EOL here: https://www.kernel.org/) Thank you -- To unsubscribe from this

[PATCH] fstests: btrfs/011 cleanup the functions

Use common helper functions where needed. By doing this it improves code readability and debugging of it. Signed-off-by: Anand Jain --- tests/btrfs/011 | 197 ++-- 1 file changed, 63 insertions(+), 134 deletions(-) diff

Re: [PATCH 1/2] btrfs: Factor out read portion of btrfs_get_blocks_direct

On Wed, Apr 11, 2018 at 10:22:58AM +0300, Nikolay Borisov wrote: > >> + free_extent_map(em); > >> + goto unlock_err; > >> + } > >> + /* > >> + * We need to unlock only the end area that we aren't using > >> + * if there is any

Re: [PATCH 1/3] btrfs: Add incompat flags check for btrfs_check_super_valid()

On Thu, Apr 19, 2018 at 07:10:30PM +0800, Qu Wenruo wrote: > > > On 2018年04月19日 18:59, Nikolay Borisov wrote: > > > > > > On 19.04.2018 12:38, Qu Wenruo wrote: > >> Although we have already checked incompat flags manually before really > >> mounting it, we could still enhance

Re: [PATCH 0/15] Review uuid_mutex usage

On Thu, Apr 12, 2018 at 10:29:23AM +0800, Anand Jain wrote: > uuid_mutex lock is not a per-fs lock but a global lock. The main aim of > this patch-set is to critically review the usage of this lock, and delete > the unnecessary once. By doing this we improve the concurrency of > device operations

Re: [PATCH 1/3] btrfs: Drop add_delayed_ref_head fs_info parameter

On Thu, Apr 19, 2018 at 11:09:09AM +0300, Nikolay Borisov wrote: > You said that while merging the deleayed refs cleanup/streamlining > series there were some conflicts. It turns out they are due to these 3 > patches missing. I'm sending them as a separate set or I can also send > them as part of

Re: [PATCH 1/6] btrfs: Sink extent_tree arguments in try_release_extent_mapping

On Thu, Apr 19, 2018 at 10:46:34AM +0300, Nikolay Borisov wrote: > This function already gets the page from which the two extent trees > are referenced. Simplify its signature by moving the code getting the > trees inside the function. No functional changes. > > Signed-off-by: Nikolay Borisov

Re: [PATCH 6/6] btrfs: Remove btrfs_wait_and_free_delalloc_work

On Thu, Apr 19, 2018 at 10:46:39AM +0300, Nikolay Borisov wrote: > This function is called from only 1 place and is effectively a wrapper > over wait_completion/kfree. It doesn't really bring any value having > those two calls in a separate function. Just open code it and remove it. > No

Re: [PATCH 4/6] btrfs: Use list_empty instead of list_empty_careful

On Thu, Apr 19, 2018 at 10:46:37AM +0300, Nikolay Borisov wrote: > list_empty_careful usually is a signal of something tricky going on. Its > usage in btrfs is actually not needed since both lists it's used on are > local to a function and cannot be modified concurrently. So switch to > plain

Re: [PATCH 1/2] btrfs: print-tree: output enhancement

On Thu, Apr 19, 2018 at 08:57:30PM +0800, Qu Wenruo wrote: > > > On 2018年04月19日 20:49, David Sterba wrote: > > On Wed, Apr 11, 2018 at 05:08:12PM +0800, Qu Wenruo wrote: > >> This patch enhance the following things: > >> > >> - tree block header > >> * add generation and owner output for node

Re: [PATCH 2/2] btrfs: print-tree: Add locking status output for debug build

On Wed, Apr 11, 2018 at 05:08:13PM +0800, Qu Wenruo wrote: > It's pretty handy if we can get debug output for locking status of an > extent buffer, specially for race related debugging. > > So add the following output for btrfs_print_tree() and > btrfs_print_leaf(): > - refs > - write_locks (as

Re: [PATCH 1/2] btrfs: print-tree: output enhancement

On 2018年04月19日 20:49, David Sterba wrote: > On Wed, Apr 11, 2018 at 05:08:12PM +0800, Qu Wenruo wrote: >> This patch enhance the following things: >> >> - tree block header >> * add generation and owner output for node and leaf >> - node pointer generation output >> - allow btrfs_print_tree()

Re: [PATCH 1/2] btrfs: print-tree: output enhancement

On Wed, Apr 11, 2018 at 05:08:12PM +0800, Qu Wenruo wrote: > This patch enhance the following things: > > - tree block header > * add generation and owner output for node and leaf > - node pointer generation output > - allow btrfs_print_tree() to not follow nodes > * just like btrfs-progs >

Re: [PATCH v2 1/3] btrfs: clean up le_bitmap_{set, clear}()

On Wed, Apr 18, 2018 at 06:02:35PM -0700, Howard McLauchlan wrote: > le_bitmap_set() is only used by free-space-tree, so move it there and > make it static. le_bitmap_clear() is not used, so remove it. > > Signed-off-by: Howard McLauchlan Added to the queue, thanks. -- To

Re: Filesystem full with unallocated space?

Hi Nikolay, Updating to 4.15 seems to have fixed it. Thanks! Matthew On 04/18/2018 05:48 PM, Nikolay Borisov wrote: On 17.04.2018 19:08, Matthew Lai wrote: Hello! I am getting ENOSPC on my root filesystem with plenty of unallocated space according to "fi usage". Any idea why that may be?

Re: [PATCH 1/3] btrfs: Add incompat flags check for btrfs_check_super_valid()

On 2018年04月19日 18:59, Nikolay Borisov wrote: > > > On 19.04.2018 12:38, Qu Wenruo wrote: >> Although we have already checked incompat flags manually before really >> mounting it, we could still enhance btrfs_check_super_valid() to check >> incompat flags for later write time super block

Re: [PATCH 1/3] btrfs: Add incompat flags check for btrfs_check_super_valid()

On 19.04.2018 12:38, Qu Wenruo wrote: > Although we have already checked incompat flags manually before really > mounting it, we could still enhance btrfs_check_super_valid() to check > incompat flags for later write time super block validation check. > > This patch adds such incompat flags

[PATCH v2 2/3] btrfs: Add csum type check for btrfs_check_super_valid()

Just like incompat flags check, although we have already done super csum type check before calling btrfs_check_super_valid(), we can still add such check for later write time check. Signed-off-by: Qu Wenruo --- v2: Move csum_type check after magic check. --- fs/btrfs/disk-io.c

[PATCH v2 3/3] btrfs: Do super block verification before writing it to disk

There are already 2 reports about strangely corrupted super blocks, where csum still matches but extra garbage gets slipped into super block. The corruption would looks like: -- superblock: bytenr=65536, device=/dev/sdc1 - csum_type

[PATCH v2 3/3] btrfs: Do super block verification before writing it to disk

There are already 2 reports about strangely corrupted super blocks, where csum still matches but extra garbage gets slipped into super block. The corruption would looks like: -- superblock: bytenr=65536, device=/dev/sdc1 - csum_type

Re: [PATCH 3/3] btrfs: Do super block verification before writing it to disk

On 2018年04月19日 18:16, David Sterba wrote: > Looks good, some minor comments below. I'm wondering how to test that. Currently I'm using the most stupid way to test it, insert code randomly modifies super blocks. > We'd have to inject either the corruption or to provide a way to > forcibly fail

Re: [PATCH 2/3] btrfs: Add csum type check for btrfs_check_super_valid()

On 2018年04月19日 18:09, David Sterba wrote: > On Thu, Apr 19, 2018 at 05:38:15PM +0800, Qu Wenruo wrote: >> Just like incompat flags check, although we have already done super csum >> type check before calling btrfs_check_super_valid(), we can still add >> such check for later write time check. >>

Re: [PATCH 3/3] btrfs: Do super block verification before writing it to disk

Looks good, some minor comments below. I'm wondering how to test that. We'd have to inject either the corruption or to provide a way to forcibly fail the test. For the latter a debugfs should do, I'll send something for comments. On Thu, Apr 19, 2018 at 05:38:16PM +0800, Qu Wenruo wrote: > @@

Re: [PATCH 2/3] btrfs: Add csum type check for btrfs_check_super_valid()

On Thu, Apr 19, 2018 at 05:38:15PM +0800, Qu Wenruo wrote: > Just like incompat flags check, although we have already done super csum > type check before calling btrfs_check_super_valid(), we can still add > such check for later write time check. > > Signed-off-by: Qu Wenruo > ---

[PATCH 3/3] btrfs: Do super block verification before writing it to disk

There are already 2 reports about strangely corrupted super blocks, where csum still matches but extra garbage gets slipped into super block. The corruption would looks like: -- superblock: bytenr=65536, device=/dev/sdc1 - csum_type

[PATCH 2/3] btrfs: Add csum type check for btrfs_check_super_valid()

Just like incompat flags check, although we have already done super csum type check before calling btrfs_check_super_valid(), we can still add such check for later write time check. Signed-off-by: Qu Wenruo --- fs/btrfs/disk-io.c | 9 + 1 file changed, 9 insertions(+)

[PATCH 1/3] btrfs: Add incompat flags check for btrfs_check_super_valid()

Although we have already checked incompat flags manually before really mounting it, we could still enhance btrfs_check_super_valid() to check incompat flags for later write time super block validation check. This patch adds such incompat flags check for btrfs_check_super_valid(), currently it

Re: [PATCH 1/3] btrfs: Drop add_delayed_ref_head fs_info parameter

On 19.04.2018 11:06, Nikolay Borisov wrote: > It's provided by the transaction handle > > Signed-off-by: Nikolay Borisov David, You said that while merging the deleayed refs cleanup/streamlining series there were some conflicts. It turns out they are due to these 3 patches

[PATCH 2/3] btrfs: Drop fs_info parameter from add_delayed_data_ref

It's provided by the transaction handle. Signed-off-by: Nikolay Borisov --- fs/btrfs/delayed-ref.c | 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index c0608cbf1354..f343ee0b143a 100644 ---

[PATCH 1/3] btrfs: Drop add_delayed_ref_head fs_info parameter

It's provided by the transaction handle Signed-off-by: Nikolay Borisov --- fs/btrfs/delayed-ref.c | 20 +--- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index e9a624196295..c0608cbf1354 100644

[PATCH 3/3] btrfs: Drop fs_info parameter from btrfs_merge_delayed_refs

It's provided by the transaction handle. Signed-off-by: Nikolay Borisov --- fs/btrfs/delayed-ref.c | 2 +- fs/btrfs/delayed-ref.h | 1 - fs/btrfs/extent-tree.c | 3 +-- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/delayed-ref.c

[PATCH 4/6] btrfs: Use list_empty instead of list_empty_careful

list_empty_careful usually is a signal of something tricky going on. Its usage in btrfs is actually not needed since both lists it's used on are local to a function and cannot be modified concurrently. So switch to plain list_empty. No functional changes. Signed-off-by: Nikolay Borisov

[PATCH 6/6] btrfs: Remove btrfs_wait_and_free_delalloc_work

This function is called from only 1 place and is effectively a wrapper over wait_completion/kfree. It doesn't really bring any value having those two calls in a separate function. Just open code it and remove it. No functional changes. Signed-off-by: Nikolay Borisov ---

[PATCH 1/6] btrfs: Sink extent_tree arguments in try_release_extent_mapping

This function already gets the page from which the two extent trees are referenced. Simplify its signature by moving the code getting the trees inside the function. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c | 6 +++--- fs/btrfs/extent_io.h

[PATCH 3/6] btrfs: Remove redundant tree argument from extent_readpages

This function is called only from btrfs_readpage and is already passed the mapping. Simplify its signature by moving the code obtaining reference to the extent tree in the function. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c | 6 +++---

[PATCH 5/6] btrfs: Remove tree argument from extent_writepages

It can be directly referenced from the passed address_space so do that. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c | 5 ++--- fs/btrfs/extent_io.h | 3 +-- fs/btrfs/inode.c | 5 + 3 files changed, 4 insertions(+), 9 deletions(-)

[PATCH 2/6] btrfs: Remove map argument from try_release_extent_state

It's not used in the function so just remove it. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index

Re: [btrfs_put_block_group] WARNING: CPU: 1 PID: 14674 at fs/btrfs/disk-io.c:3675 free_fs_root+0xc2/0xd0 [btrfs]

On 19.04.2018 08:32, Fengguang Wu wrote: > Hello, > > FYI this happens in mainline kernel and at least dates back to v4.16 . > > It's rather rare error and happens when running xfstests. Yeah, so this is something which only recently was characterised as leaking delalloc inodes. I can easily

[PATCH] btrfs: Add test that checks rmdir(2) can delete a subvolume

Add btrfs test that checks "rmdir" or "rm -r" command can delete a subvolume like an ordinary drectory. This behavior has been restricted long time but becomes allowed by following patch in the kernel: btrfs: Allow rmdir(2) to delete an empty subvolume Signed-off-by: Tomohiro Misono

Re: nvme+btrfs+compression sensibility and benchmark

On 18.04.2018 22:24, Chris Murphy wrote: > On Wed, Apr 18, 2018 at 10:38 AM, Austin S. Hemmelgarn > wrote: > >> For reference, the zstd compression in BTRFS uses level 3 by default (as >> does zlib compression IIRC), though I'm not sure about lzop (I think it >> uses the