[PATCH for 4.4] btrfs: fix rcu warning during device replace

The test btrfs/011 triggers a rcu warning === [ INFO: suspicious RCU usage. ] 4.4.0-rc1-default+ #286 Tainted: GW --- fs/btrfs/volumes.c:1977 suspicious rcu_dereference_check() usage! other info that might help us debug this:

[PULL][PATCH 0/3] Constify structs

A few more additions of const, aiming for 4.5. Thanks. The following changes since commit 8005c49d9aea74d382f474ce11afbbc7d7130bec: Linux 4.4-rc1 (2015-11-15 17:00:27 -0800) are available in the git repository at:

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

On 19 November 2015 at 06:58, Roman Mamedov wrote: > > On Wed, 18 Nov 2015 19:53:03 +0100 > linux-btrfs.tebu...@xoxy.net wrote: > > > $ uname -a > > Linux neptun 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 > > 10:21:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux [...] > >

[PATCH] btrfs: Support convert to -d dup for btrfs-convert

Since we will add support for -d dup for non-mixed filesystem, kernel need to support converting to this raid-type. This patch remove limitation of above case. Tested by following script: (combination of dup conversion with fsck): export TEST_DEV='/dev/vdc' export TEST_DIR='/var/ltf/tester/mnt'

[PATCH v3] btrfs-progs: mkfs: Enable -d dup for single device

Current code don't support dup profile in single device, except it is in mixed mode, because following reason: 1: In some ssd with deduplication function, it have no effect. 2: For a physical device, it the entire disk broken, -d dup can not help. 3: Half performance comparing with single

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

This explanation helped a lot. Got it now! Thank you! >You start with a load of unused space -- that's the "total" for > each device in btrfs fi show. That space is allocated to specific > usages as the FS needs it. The allocated space is the "used" in btrfs > fi show for each device. > >

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

>It's just freed up lots of space. You'll probably find that your > "total" value for data in btrfs fi df is close to (but not exactly) 66 > GiB now, if you've just run a full unfiltered balance. (The difference > being made up of metadata). I think i need to dive more into the details of

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

On Thu, Nov 19, 2015 at 07:45:13PM +0100, Ben Tebulin wrote: > > >It's just freed up lots of space. You'll probably find that your > > "total" value for data in btrfs fi df is close to (but not exactly) 66 > > GiB now, if you've just run a full unfiltered balance. (The difference > > being

Re: anything wrong with `balance -dusage -musage` together?

On 11/20/2015 12:59 PM, Hugo Mills wrote as excerpted: >Nothing actively wrong with that, no. It certainly won't break > anything. It's just rarely actually useful. The usual situation is > that you run out of one kind of storage before the other (data vs > metadata, that is), and you need to

[PATCH] Btrfs: fix a bug of sleeping in atomic context

while xfstesting, this bug[1] is spotted by both btrfs/061 and btrfs/063, so those sub-stripe writes are gatherred into plug callback list and hopefully we can have a full stripe writes. However, while processing these plugged callbacks, it's within an atomic context which is provided by

Re: anything wrong with `balance -dusage -musage` together?

On Fri, Nov 20, 2015 at 12:53:33PM +1300, Lukas Pirl wrote: > Hi list, > > I rarely see balance used with -dusage -musage together, esp. with > values other than zero. > > The question is, is there anything wrong with running (say) `balance > -dusage=50 -musage=30` regularly? Nothing

anything wrong with `balance -dusage -musage` together?

Hi list, I rarely see balance used with -dusage -musage together, esp. with values other than zero. The question is, is there anything wrong with running (say) `balance -dusage=50 -musage=30` regularly? Thanks and best regards, Lukas -- To unsubscribe from this list: send the line "unsubscribe

Re: [PATCH v3] btrfs-progs: mkfs: Enable -d dup for single device

On 2015-11-19 04:36, Zhao Lei wrote: Current code don't support dup profile in single device, except it is in mixed mode, because following reason: 1: In some ssd with deduplication function, it have no effect. 2: For a physical device, it the entire disk broken, -d dup can not help. 3: Half

Possibility of a BTRFS module for Ansible.

I recently started using Ansible for orchestrating updates, backups, and other similar stuff. Working with BTRFS with it has gotten annoying really fast due to some of the hoops I have to jump through to get the btrfs command to work in a way that the shell and command modules in Ansible

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

On 2015-11-19 13:28, Hugo Mills wrote: On Thu, Nov 19, 2015 at 06:35:24PM +0100, linux-btrfs.tebu...@xoxy.net wrote: Will newer kernels do the balance on their own? I think it's on the "projects" list on the wiki, so it may get done eventually. As I said above, I'm not aware of anyone

[PATCH] Btrfs: fix race between scrub and block group deletion

From: Filipe Manana Scrub can race with the cleaner kthread deleting block groups that are unused (and with relocation too) leading to a failure with error -EINVAL that gets returned to user space. The following diagram illustrates how it happens: CPU 1

[PATCH] Btrfs: fix scrub preventing unused block groups from being deleted

From: Filipe Manana Currently scrub can race with the cleaner kthread when the later attempts to delete an unused block group, and the result is preventing the cleaner kthread from ever deleting later the block group - unless the block group becomes used and unused again. The

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

On 2015-11-18 13:53, linux-btrfs.tebu...@xoxy.net wrote: P.S.: Just as user feedback: For /srv I'm using on the very same system ZFS since the very first day. With snapshots & all the fancy stuff like ZRAID-1, lz4, ... My number of Issues there: 0 Since other people have adequately answered the

[PATCH 1/3] btrfs tests: replace whole ops structure for free space tests

Preparatory work for making btrfs_free_space_op constant. In test_steal_space_from_bitmap_to_extent, we substitute use_bitmap with own version thus preventing constification. We can rework it so we replace the whole structure with the correct function pointers. Signed-off-by: David Sterba

[PATCH 2/3] btrfs: constify remaining structs with function pointers

* struct extent_io_ops * struct btrfs_free_space_op Signed-off-by: David Sterba --- fs/btrfs/free-space-cache.c | 2 +- fs/btrfs/free-space-cache.h | 2 +- fs/btrfs/inode-map.c | 4 ++-- fs/btrfs/inode.c | 4 ++--

[PATCH 3/3] btrfs: constify static arrays

There are a few statically initialized arrays that can be made const. The remaining (like file_system_type, sysfs attributes or prop handlers) do not allow that due to type mismatch when passed to the APIs or because the structures are modified through other members. Signed-off-by: David Sterba

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

Austin S Hemmelgarn posted on Thu, 19 Nov 2015 07:28:34 -0500 as excerpted: > (having all updates installed on Ubuntu doesn't really count in this > case, they're pretty bad sometimes about not properly tracking upstream > development[)] No kidding. I'm involved with an upstream that had a

[PATCH 24/25] btrfs-progs: convert: Strictly avoid meta or system chunk allocation

Before this patch, btrfs-convert only rely on large enough initial system/metadata chunk size to ensure no newer system/meta chunk will be created. But that's not safe enough. So add two new members in fs_info, avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta chunks to be

[PATCH 19/25] btrfs-progs: convert: Introduce function to migrate reserved ranges

Introduce new function, migrate_reserved_ranges() to migrate used ext2 data in btrfs reserved space. Unlike old implement, which will need to relocate all the complicated csum and reference relocation, above patches already ensure such reserved ranges won't be allocated. So here we only need copy

[PATCH 17/25] btrfs-progs: extent-tree: Enhance btrfs_record_file_extent

Btrfs_record_file_extent() has some small problems like: 1) Can't handle overlap extent 2) May create extent larger than BTRFS_MAX_EXTENT_SIZE So enhance it using previous added facilites. Signed-off-by: Qu Wenruo --- extent-tree.c | 160

[PATCH 20/25] btrfs-progs: Enhance record_file_blocks to handle reserved ranges

Enhance record_file_blocks() function to handle reserved ranges. Unlike most of file extents, ext* data in reserved ranges are not mapped on disk with 1:1 bytenr. So we can't use bytenr directly. But thanks for calling create_ext2_image_v2() before copying inodes, we have a image in convert

[PATCH 25/25] btrfs-progs: Cleanup old btrfs-convert

Cleanup all the old btrfs-convert facilities, including old init_btrfs(), create_file_image_range() and custom extent allocation function. The cleanup only focus on btrfs-convert, no make_btrfs() cleanup yet. It will be delayed until I found a nice and clean way to rework make_btrfs() to support

[PATCH 14/25] btrfs-progs: Introduce function to setup temporary extent tree

Introduce new function, setup_temporary_extent_tree() to build a temporary extent tree for make_btrfs_v2(). Signed-off-by: Qu Wenruo --- utils.c | 198 +++- 1 file changed, 197 insertions(+), 1 deletion(-)

[PATCH 21/25] btrfs-progs: convert: Introduce init_btrfs_v2 function.

Introduce new init_btrfs_v2() function for later newer do_convert(). Since we have good enough chunk allocation, a lot of wired chunk hack won't ever be used. We only need to insert data chunks and create needed subvolume/inode. Signed-off-by: Qu Wenruo ---

[PATCH 22/25] btrfs-progs: Introduce do_convert_v2 function

Introduce new function do_convert_v2() to do new convert. Signed-off-by: Qu Wenruo --- btrfs-convert.c | 187 ++-- utils.c | 3 + 2 files changed, 186 insertions(+), 4 deletions(-) diff --git

[PATCH 13/25] btrfs-progs: Introduce function to initialize csum tree

Introduce new function, setup_temp_csum_tree(), to setup temporary csum tree for make_btrfs_v2(). Signed-off-by: Qu Wenruo --- utils.c | 25 + 1 file changed, 25 insertions(+) diff --git a/utils.c b/utils.c index edd8ac1..ee2a538 100644 ---

[PATCH 08/25] btrfs-progs: Introduce function to setup temporary superblock

Introduce a new function, setup_temp_super(), to setup temporary super for make_btrfs_v2(). Unlike the old codes in make_btrfs(), it will also initialize system_chunk_array before chunk tree allocation. Signed-off-by: Qu Wenruo --- utils.c | 117

[PATCH 23/25] btrfs-progs: Convert: Add support for rollback new convert behavior

Add support to rollback new btrfs-convert. The support is quite easy unlike the new convert behavior, which only needs to check if there is block group covering the reserved ranges. Old convert behavior ensure there is always a system chunk covering reserved ranges. The new one ensure there is

[PATCH 15/25] btrfs-progs: Introduce function to create convert data chunks

Introduce new function, make_convert_data_chunks(), to build up data chunks for convert. It will call a modified verion of btrfs_alloc_data_chunk() to force data chunks to cover all known ext* data. Signed-off-by: Qu Wenruo --- btrfs-convert.c | 49

[PATCH 18/25] btrfs-progs: convert: Introduce new function to create ext2 image

Use new function, create_ext2_image_v2() to create ext2 image. Unlike old function which is called after copying all inodes, this function need to be called before copying inodes. Signed-off-by: Qu Wenruo --- btrfs-convert.c | 153

Self-destruct of btrfs RAID6 array

I have just had an apparently catastrophic collapse of a large RAID6 array. I was hoping that the dual-redundancy of a RAID6 array would compensate for having no backup media large enough to back it up! Any suggestions for repairing this array, at least to the point of mounting it read-only?

Re: anything wrong with `balance -dusage -musage` together?

Lukas Pirl posted on Fri, 20 Nov 2015 12:53:33 +1300 as excerpted: > I rarely see balance used with -dusage -musage together, esp. with > values other than zero. > > The question is, is there anything wrong with running (say) `balance > -dusage=50 -musage=30` regularly? I see the question in

Re: Kernel 3.19 and still "disk full" even though 'btrfs fi df" reports enough room left?

linux-btrfs.tebulin posted on Thu, 19 Nov 2015 18:56:45 + as excerpted: Meta-comment: Apparently that attribution should actually be to Hugo Mills. I've no idea what went wrong, but at least here as received from gmane.org, the from header really does say linux-btrfs.tebulin, so something

[PATCH 04/25] btrfs-progs: convert: Read and build up used space tree

Before we do real convert, we'd better read and build up used space cache tree for later data/meta chunk layout calculation. This patch will iterate all used blocks in ext2 filesystem and record it into mkfs_cfg.used cache tree for later used. This provides the basis for later btrfs-convert

[PATCH 06/25] btrfs-progs: utils: Introduce function to calculate the available space

Introduce a new function, calculate_available_space() to get available space for convert. Unlike old implement, this function will do the new work: 1) batch used ext* data space. To ensure data chunks will recovery them all. And restore the result into mkfs_cfg->convert_data_chunks for

[PATCH 01/25] btrfs-progs: extent-cache: Add comments for search/lookup functions

There are quite a lot search/lookup functions with different behavior, add comments for them, as it will take extra time to view source to understand the behavior difference. Signed-off-by: Qu Wenruo --- extent-cache.h | 31 +++ 1 file

[PATCH 09/25] btrfs-progs: Introduce function to setup temporary tree root

Introduce new function, setup_temp_tree_root(), to initialize temporary tree root for make_btrfs_v2(). The new function will setup tree root at metadata chunk and ensure data won't be written into metadata chunk. Also, new make_btrfs_v2() will have a much better code structure than old

[PATCH 11/25] btrfs-progs: Introduce function to initialize device tree

Introduce new function, setup_temp_device_tree(), to setup temporary device tree for make_btrfs_v2(). Signed-off-by: Qu Wenruo --- utils.c | 56 +++- 1 file changed, 55 insertions(+), 1 deletion(-) diff --git

[PATCH 16/25] btrfs-progs: extent-tree: Introduce function to find the first overlap extent.

Introduce a new function, btrfs_search_overlap_extent() to find the first overlap extent. It's useful for later btrfs-convert rework. Signed-off-by: Qu Wenruo --- ctree.c | 24 +++ ctree.h | 2 ++ extent-tree.c | 63

[PATCH 07/25] btrfs-progs: Reserve space for system/meta chunks and superblock

Now we have a free space cache tree, we can reserve space for system and metadata chunks and super blocks. With this patch, even for the temporary fs, metadata tree blocks will be prevent from being allocated into possible data chunks. This provides the basis for later btrfs-convert enhancement.

[PATCH 05/25] btrfs-progs: utils: Introduce new function to remove reserved ranges

Introduce functions to remove reserved ranges for later btrfs-convert rework. The reserved ranges includes: 1. [0,1M) 2. [btrfs_sb_offset(1), +BTRFS_STRIP_LEN) 3. [btrfs_sb_offset(2), +BTRFS_STRIP_LEN) Signed-off-by: Qu Wenruo --- utils.c | 115

[PATCH 02/25] btrfs-progs: extent-tree: Add add_merge_cache_extent function

This add_merge_cache_extent() function will try to merge adjusted cache_extent. This is used for later btrfs-convert ext2 free space cache. Signed-off-by: Qu Wenruo --- extent-cache.c | 57 + extent-cache.h | 8

[PATCH 12/25] btrfs-progs: Introduce function to initialize fs tree

Introudce new function, setup_temp_fs_tree(), to setup temporary fs tree for make_btrfs_v2(). Signed-off-by: Qu Wenruo --- utils.c | 26 ++ 1 file changed, 26 insertions(+) diff --git a/utils.c b/utils.c index 3be484a..edd8ac1 100644 ---

[PATCH 03/25] btrfs-progs: Add new init/free function and member for mkfs_config

Add new members for mkfs_config: 1. super_bytenr For convert case to restore where super block is allocated. Has no use for normal mkfs case. 2. convert_used A cache tree to record which ranges are used in original filesystem. This will gives the guide for later convert implement to

[PATCH 00/25] Btrfs-convert rework to support native separate

Here comes the 1st version of btrfs-convert rework. Any test is welcomed, and it can already pass the convert test from btrfs-progs. (Since the test doesn't test rollback function) I also did some tests like create ext4 and fill it with fsstress to test convert, at least no bug spotted yet. The

Re: Self-destruct of btrfs RAID6 array

Paul Loewenstein posted on Thu, 19 Nov 2015 20:11:14 -0800 as excerpted: > I have just had an apparently catastrophic collapse of a large RAID6 > array. I was hoping that the dual-redundancy of a RAID6 array would > compensate for having no backup media large enough to back it up! Well...

Re: [PATCH] btrfs: qgroup: fix quota disable during rescan

Hi, On Fri, Nov 06, 2015 at 10:36:42AM -0800, Justin Maggard wrote: > There's a race condition that leads to a NULL pointer dereference if you > disable quotas while a quota rescan is running. To fix this, we just need > to wait for the quota rescan worker to actually exit before tearing down >

[PATCH] btrfs: put delayed item hook into inode

From: David Sterba Inodes for delayed iput allocate a trivial helper structure, let's place the list hook directly into the inode and save a kmalloc (killing a __GFP_NOFAIL as a bonus) at the cost of increasing size of btrfs_inode. The inode can be put into the delayed_iputs

Re: [PATCH] btrfs: qgroup: fix quota disable during rescan

On Thu, Nov 19, 2015 at 1:08 PM, David Sterba wrote: > Hi, > > On Fri, Nov 06, 2015 at 10:36:42AM -0800, Justin Maggard wrote: >> There's a race condition that leads to a NULL pointer dereference if you >> disable quotas while a quota rescan is running. To fix this, we just need

Re: [PATCH] btrfs: qgroup: fix quota disable during rescan

On Thu, Nov 19, 2015 at 01:16:42PM +, Filipe Manana wrote: > On Thu, Nov 19, 2015 at 1:08 PM, David Sterba wrote: > > Hi, > > > > On Fri, Nov 06, 2015 at 10:36:42AM -0800, Justin Maggard wrote: > >> There's a race condition that leads to a NULL pointer dereference if you > >>