Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 20:06, John Ettedgui wrote: >>> That's fairly straightforward to do, though it should be quite slow so >>> I'd hope not to have to do that too often. >> >> Then it could be tried on the most frequently updated files then. > > That's an interesting idea. > More than 3/4 of

btrfs-progs: compile with musl libc

Ensure that limits.h is included in mkfs/main.c as PATH_MAX is used. The lack of this currently causes btrfs-progs to fail to compile when the systems libc is musl. From 7859e0d01778e844ae0fcbefe55581277ce7cab3 Mon Sep 17 00:00:00 2001 From: Dan Robertson Date: Tue, 13

Re: btrfs-cleaner / snapshot performance analysis

On Mon, Feb 12, 2018 at 10:37 AM, Ellis H. Wilson III wrote: > On 02/11/2018 01:24 PM, Hans van Kranenburg wrote: >> >> Why not just use `btrfs fi du ` now and then and >> update your administration with the results? .. Instead of putting the >> burden of keeping track of

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

On 13.02.2018 11:05, Qu Wenruo wrote: > > > On 2018年02月13日 11:53, Anand Jain wrote: >> For forensic investigations of issues, we would want to know >> if and when the user cancels the replace. >> >> Signed-off-by: Anand Jain > --- >> v1->v2: use btrfs_dev_name() instead

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: > > > No more. > > The dump is already good enough for me to dig for some time. > > We don't usually get such large extent tree dump from a real world use case. > > It would help us in several ways, from determine how

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 18:21, John Ettedgui wrote: > On Thu, Jul 21, 2016 at 1:19 AM, Qu Wenruo wrote: >> >> >> No more. >> >> The dump is already good enough for me to dig for some time. >> >> We don't usually get such large extent tree dump from a real world use case. >> >>

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

On 2018年02月13日 09:13, Qu Wenruo wrote: > There are reports in mail list, even with latest mainline kernel, btrfs > can't survive a power loss. > > Unlike journal based filesystem, btrfs doesn't use journal for such > work. (log tree is an optimization for fsync, not to keep fs healthy) > In

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 4:46 AM, Qu Wenruo wrote: > > > On 2018年02月13日 20:06, John Ettedgui wrote: > That's fairly straightforward to do, though it should be quite slow so I'd hope not to have to do that too often. >>> >>> Then it could be tried on the most

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

On 13.02.2018 13:07, Qu Wenruo wrote: > > > On 2018年02月13日 09:13, Qu Wenruo wrote: >> There are reports in mail list, even with latest mainline kernel, btrfs >> can't survive a power loss. >> >> Unlike journal based filesystem, btrfs doesn't use journal for such >> work. (log tree is an

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 02/13/18 12:40, Qu Wenruo wrote: >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. I have no idea why btrfs' mount even needs to touch all block groups to get going (which seems to be the root of the problem), but here's a not so crazy

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: > > > On 2018年02月13日 18:21, John Ettedgui wrote: >> Hello there, >> >> have you found anything good since then? > > Unfortunately, not really much to speed it up. Oh :/ > > This reminds me of the old (and crazy) idea to

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 19:25, John Ettedgui wrote: > On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >> >> >> On 2018年02月13日 18:21, John Ettedgui wrote: >>> Hello there, >>> >>> have you found anything good since then? >> >> Unfortunately, not really much to speed it up. > Oh

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On Tue, Feb 13, 2018 at 3:40 AM, Qu Wenruo wrote: > > > On 2018年02月13日 19:25, John Ettedgui wrote: >> On Tue, Feb 13, 2018 at 3:04 AM, Qu Wenruo wrote: >>> >>> >>> >>> The problem is not about how much space it takes, but how many extents >>> are

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月13日 20:26, Holger Hoffstätte wrote: > On 02/13/18 12:40, Qu Wenruo wrote: The problem is not about how much space it takes, but how many extents are here in the filesystem. > > I have no idea why btrfs' mount even needs to touch all block groups to > get going (which seems

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

On 2018年02月13日 15:20, Nikolay Borisov wrote: > > > On 13.02.2018 03:13, Qu Wenruo wrote: >> There are reports in mail list, even with latest mainline kernel, btrfs >> can't survive a power loss. >> >> Unlike journal based filesystem, btrfs doesn't use journal for such >> work. (log tree is an

[PATCH v2 1/2] btrfs: declare max_inline as u32

As of now btrfs_fs_info::max_line is u64, which can't be larger than btrfs_fs_info::sectorsize which is defined as u32, so make btrfs_fs_info::max_line u32, Signed-off-by: Anand Jain --- v1->v2: Born in v2. fs/btrfs/ctree.h | 2 +- fs/btrfs/super.c | 8 2 files

[PATCH] btrfs: manage metadata_ratio mount option as %u

As -o metadata_ratio mount option is unsinged so manage it as %u for token verifications, instead of %s. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 17 ++--- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/super.c

[PATCH] btrfs: fix bare unsigned declarations

Kernel style prefers "unsigned int " over "unsigned " and "signed int " over "signed ". Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index

[PATCH] btrfs: add a comment to mark the deprecated mount option

-o alloc_start and -o subvolrootid are deprecated mount options, comment them in the tokens list. And leave them as it is. No functional changes. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git

[PATCH v2 2/2] btrfs: verify max_inline mount parameter

We aren't verifying the parameter passed to the max_inline mount option, so we won't report and fail the mount if a junk value is specified for example, -o max_inline=abc. This patch converts the max_inline option to %d and checks if it's a number >= 0. Signed-off-by: Anand Jain

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

On 2018年02月13日 17:08, Nikolay Borisov wrote: > > > On 13.02.2018 11:05, Qu Wenruo wrote: >> >> >> On 2018年02月13日 11:53, Anand Jain wrote: >>> For forensic investigations of issues, we would want to know >>> if and when the user cancels the replace. >>> >>> Signed-off-by: Anand Jain

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

On 2018年02月13日 11:53, Anand Jain wrote: > For forensic investigations of issues, we would want to know > if and when the user cancels the replace. > > Signed-off-by: Anand Jain > --- > v1->v2: use btrfs_dev_name() instead of rcu_str_deref() > as btrfs_dev_name()

Re: [PATCH] btrfs: verify max_inline mount parameter

On 02/13/2018 01:13 AM, David Sterba wrote: On Mon, Feb 12, 2018 at 11:35:46PM +0800, Anand Jain wrote: We aren't verifying the parameter passed to the max_inline mount option, so we won't report and fail the mount if a junk value is specified for example, -o max_inline=abc. This patch

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

On 02/13/2018 05:01 PM, Qu Wenruo wrote: On 2018年02月13日 11:00, Anand Jain wrote: Fixes the endianness bug in the fs_info::super_copy by using its btrfs_set_super...() function to set values in the SB, as these functions manage the endianness compatibility nicely. Signed-off-by: Anand Jain

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

On 2018年02月13日 11:00, Anand Jain wrote: > Fixes the endianness bug in the fs_info::super_copy by using its > btrfs_set_super...() function to set values in the SB, as these > functions manage the endianness compatibility nicely. > > Signed-off-by: Anand Jain Also went

[PATCH] btrfs: manage subvolid mount option as %u

As -o subvolid mount option is an u64 manage it as %u for token verifications, instead of %s. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 23 ++- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c

[PATCH] btrfs: manage commit mount option as %u

As -o commit mount option is unsinged so manage it as %u for token verifications, instead of %d. Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 2 +- fs/btrfs/super.c | 26 ++ 2 files changed, 11 insertions(+), 17 deletions(-) diff --git

[PATCH] btrfs: manage thread_pool mount option as %u

-o thread_pool is alway unsigned. Manage it that way all around. Signed-off-by: Anand Jain --- fs/btrfs/ctree.h | 2 +- fs/btrfs/disk-io.c | 4 ++-- fs/btrfs/super.c | 13 ++--- 3 files changed, 9 insertions(+), 10 deletions(-) diff --git a/fs/btrfs/ctree.h

[PATCH] btrfs: manage check_int_print_mask mount option as %u

As -o check_int_print_mask mount option is unsinged so manage it as %u for token verifications, instead of %d. Signed-off-by: Anand Jain --- fs/btrfs/super.c | 16 +--- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/super.c

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

On 2018年02月13日 18:27, Anand Jain wrote: > > > On 02/13/2018 05:01 PM, Qu Wenruo wrote: >> >> >> On 2018年02月13日 11:00, Anand Jain wrote: >>> Fixes the endianness bug in the fs_info::super_copy by using its >>> btrfs_set_super...() function to set values in the SB, as these >>> functions manage

Re: [PATCH v2 1/2] btrfs: declare max_inline as u32

On Tue, Feb 13, 2018 at 05:49:49PM +0800, Anand Jain wrote: > As of now btrfs_fs_info::max_line is u64, which can't be > larger than btrfs_fs_info::sectorsize which is defined as > u32, so make btrfs_fs_info::max_line u32, > > Signed-off-by: Anand Jain Reviewed-by: David

Re: [PATCH] btrfs: manage thread_pool mount option as %u

On 13.02.2018 11:50, Anand Jain wrote: > -o thread_pool is alway unsigned. Manage it that way all around. > > Signed-off-by: Anand Jain > --- > fs/btrfs/ctree.h | 2 +- > fs/btrfs/disk-io.c | 4 ++-- > fs/btrfs/super.c | 13 ++--- > 3 files changed, 9

Re: [PATCH] btrfs: manage thread_pool mount option as %u

On Tue, Feb 13, 2018 at 05:34:56PM +0200, Nikolay Borisov wrote: > >>> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c > >>> index 02c7766e6849..8112619cac95 100644 > >>> --- a/fs/btrfs/super.c > >>> +++ b/fs/btrfs/super.c > >>> @@ -346,7 +346,7 @@ static const match_table_t tokens = { > >>>  

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 02/13/18 13:54, Qu Wenruo wrote: > On 2018年02月13日 20:26, Holger Hoffstätte wrote: >> On 02/13/18 12:40, Qu Wenruo wrote: > The problem is not about how much space it takes, but how many extents > are here in the filesystem. >> >> I have no idea why btrfs' mount even needs to touch all

Re: btrfs-progs: compile with musl libc

> Thanks, this has been fixed in > https://github.com/kdave/btrfs-progs/commit/3aa1bbdd89ee9c9c48d260a6192fae08328f1b2f Ah, my bad. Didn't check the dev branch. Thanks for the quick response! Cheers, Dan signature.asc Description: Digital signature

Re: [PATCH] btrfs: manage thread_pool mount option as %u

On 13.02.2018 17:18, Anand Jain wrote: > > > > > > >>> diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c >>> index 02c7766e6849..8112619cac95 100644 >>> --- a/fs/btrfs/super.c >>> +++ b/fs/btrfs/super.c >>> @@ -346,7 +346,7 @@ static const match_table_t tokens = { >>>   {Opt_barrier,

[PATCH 2/2] btrfs: Refactor running of delayed inode items during transaction commit

Currently calls to btrfs_run_delayed_inode items have been scattered around the transaction commit code with no real design argument when they should be execute. We have one call, after transaction writers go to 0. Then we have the delayed items running as part of creating a snapshot (once per

[PATCH 1/2] btrfs: Remove extra run_delayed_items call

Before commit 581227d0d2b8 ("Btrfs: remove the time check in btrfs_commit_transaction()") there would be a loop in the transaction commit path that will go on until all existing writers ended their transaction handles. This loop would try and flush as much pending stuff as possible before going

Re: [PATCH] btrfs: manage thread_pool mount option as %u

diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c index 02c7766e6849..8112619cac95 100644 --- a/fs/btrfs/super.c +++ b/fs/btrfs/super.c @@ -346,7 +346,7 @@ static const match_table_t tokens = { {Opt_barrier, "barrier"}, {Opt_max_inline, "max_inline=%u"},

Re: [PATCH v2 2/2] btrfs: verify max_inline mount parameter

On Tue, Feb 13, 2018 at 05:49:50PM +0800, Anand Jain wrote: > We aren't verifying the parameter passed to the max_inline mount option, > so we won't report and fail the mount if a junk value is specified for > example, -o max_inline=abc. > This patch converts the max_inline option to %d and checks

Re: [PATCH] btrfs: manage subvolid mount option as %u

On Tue, Feb 13, 2018 at 05:50:43PM +0800, Anand Jain wrote: > As -o subvolid mount option is an u64 manage it as %u for > token verifications, instead of %s. > > Signed-off-by: Anand Jain > --- > fs/btrfs/super.c | 23 ++- > 1 file changed, 10

Re: btrfs-progs: compile with musl libc

On Tue, Feb 13, 2018 at 01:25:10PM +, Dan Robertson wrote: > Ensure that limits.h is included in mkfs/main.c as PATH_MAX is used. The lack > of this currently causes btrfs-progs to fail to compile when the systems libc > is musl. > From 7859e0d01778e844ae0fcbefe55581277ce7cab3 Mon Sep 17

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

On Tue, Feb 13, 2018 at 06:27:13PM +0800, Anand Jain wrote: > On 02/13/2018 05:01 PM, Qu Wenruo wrote: > > On 2018年02月13日 11:00, Anand Jain wrote: > >> Fixes the endianness bug in the fs_info::super_copy by using its > >> btrfs_set_super...() function to set values in the SB, as these > >>

Re: [PATCH v2] btrfs: log, when replace, is canceled by the user

On Tue, Feb 13, 2018 at 11:53:43AM +0800, Anand Jain wrote: > For forensic investigations of issues, we would want to know > if and when the user cancels the replace. > > Signed-off-by: Anand Jain Reviewed-by: David Sterba I've updated the changelog a

Re: [PATCH 1/2] btrfs: Remove invalid null checks from btrfs_cleanup_dirty_bgs

On Thu, Feb 08, 2018 at 10:57:11AM -0800, Liu Bo wrote: > On Thu, Feb 08, 2018 at 06:25:17PM +0200, Nikolay Borisov wrote: > > list_first_entry is essentially a wrapper over cotnainer_of. The latter > > can never return null even if it's working on inconsistent list since it > > will either crash

Re: [PATCH] btrfs: extent_buffer_uptodate() make it static and inline

On Tue, Feb 13, 2018 at 09:18:51AM +0200, Nikolay Borisov wrote: > > > On 13.02.2018 06:35, Anand Jain wrote: > > extent_buffer_uptodate() is a trivial wrapper around test_bit() > > and nothing else. So make it static and inline, save on code > > space and call indirection. > > > > Before: > >

Re: [PATCH] btrfs: fix bare unsigned declarations

On Tue, Feb 13, 2018 at 05:50:48PM +0800, Anand Jain wrote: > Kernel style prefers "unsigned int " over "unsigned " > and "signed int " over "signed ". The changelog does not match the changes, you're switching to u32. I agree u32 looks more suitable and consistent with the other changes. Please

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

On 2018年02月14日 00:24, Holger Hoffstätte wrote: > On 02/13/18 13:54, Qu Wenruo wrote: >> On 2018年02月13日 20:26, Holger Hoffstätte wrote: >>> On 02/13/18 12:40, Qu Wenruo wrote: >> The problem is not about how much space it takes, but how many extents >> are here in the filesystem. >>> >>>

Re: btrfs-cleaner / snapshot performance analysis

On Sun, Feb 11, 2018 at 02:40:16PM +0800, Qu Wenruo wrote: > > > On 2018年02月10日 00:45, Ellis H. Wilson III wrote: > > Hi all, > > > > I am trying to better understand how the cleaner kthread (btrfs-cleaner) > > impacts foreground performance, specifically during snapshot deletion. > > My

[PATCH] btrfs-progs: Fix read beyond boundary bug in build_roots_info_cache()

This bug is exposed by fsck-test with D=asan, hit by test case 020, with the following error report: = ==10740==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62161580 at pc 0x56051f0db6cd bp 0x7ffe170f3e20 sp

[PATCH] btrfs-progs: mkfs/rootdir: Fix memory leak in traverse_directory()

The bug is exposed by mkfs test case 009, with D=asan. We are leaking memory of parent_dir_entry->path() which ,except the rootdir, is allocated by strdup(). Before fixing it, unifiy the allocation of parent_dir_entry() to heap allocation. Then fix it by adding "free(parent_dir_entry->path);"