[PATCH 3/6] btrfs: cleanup extent_op handling

2018-11-21 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order to

[PATCH 1/6] btrfs: add btrfs_delete_ref_head helper

2018-11-21 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++-

Re: [PATCH] Btrfs: fix access to available allocation bits when starting balance

2018-11-21 Thread David Sterba
On Mon, Nov 19, 2018 at 09:48:12AM +, fdman...@kernel.org wrote: > From: Filipe Manana > > The available allocation bits members from struct btrfs_fs_info are > protected by a sequence lock, and when starting balance we access them > incorrectly in two different ways: > > 1) In the read

[PATCH RESEND 1/2] btrfs-progs: fix kernel version parsing on some versions past 3.0

2018-11-21 Thread Adam Borowski
The code fails if the third section is missing (like "4.18") or is followed by anything but "." or "-". This happens for example if we're not exactly at a tag and CONFIG_LOCALVERSION_AUTO=n (which results in "4.18.5+"). Signed-off-by: Adam Borowski --- fsfeatures.c | 5 + 1 file changed, 1

[PATCH RESEND-v3 2/2] btrfs-progs: defrag: open files RO on new enough kernels

2018-11-21 Thread Adam Borowski
Defragging an executable conflicts both way with it being run, resulting in ETXTBSY. This either makes defrag fail or prevents the program from being executed. Kernels 4.19-rc1 and later allow defragging files you could have possibly opened rw, even if the passed descriptor is ro (commit

Re: [PATCH] Btrfs: fix race between enabling quotas and subvolume creation

2018-11-21 Thread David Sterba
On Mon, Nov 19, 2018 at 04:20:34PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > We have a race between enabling quotas end subvolume creation that cause > subvolume creation to fail with -EINVAL, and the following diagram shows > how it happens: > > CPU 0

[PATCHv3] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-11-21 Thread Nikolay Borisov
Running btrfs/124 in a loop hung up on me sporadically with the following call trace: btrfs D0 5760 5324 0x Call Trace: ? __schedule+0x243/0x800 schedule+0x33/0x90 btrfs_start_ordered_extent+0x10c/0x1b0 [btrfs] ?

Re: [RFC] BTRFS_DEV_REPLACE_ITEM_STATE_* doesn't match with on disk

2018-11-20 Thread Anand Jain
David, any comments on this please. Thanks, Anand On 11/13/2018 06:32 PM, Anand Jain wrote: David, Gentle ping. Thanks, Anand On 11/12/2018 03:50 PM, Nikolay Borisov wrote: On 12.11.18 г. 6:58 ч., Anand Jain wrote: The dev_replace_state defines are miss matched between the

Re: [PATCH v2] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-11-20 Thread Nikolay Borisov
On 20.11.18 г. 21:00 ч., Josef Bacik wrote: > On Fri, Oct 26, 2018 at 02:41:55PM +0300, Nikolay Borisov wrote: >> Running btrfs/124 in a loop hung up on me sporadically with the >> following call trace: >> btrfs D0 5760 5324 0x >> Call Trace: >> ?

Re: [PATCH v2] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-11-20 Thread Josef Bacik
On Fri, Oct 26, 2018 at 02:41:55PM +0300, Nikolay Borisov wrote: > Running btrfs/124 in a loop hung up on me sporadically with the > following call trace: > btrfs D0 5760 5324 0x > Call Trace: >? __schedule+0x243/0x800 >schedule+0x33/0x90 >

Re: [PATCH v6 0/3] btrfs: balance: improve kernel logs

2018-11-20 Thread David Sterba
On Tue, Nov 20, 2018 at 04:12:54PM +0800, Anand Jain wrote: > v5->v6: > Mostly the defines non functional changes. > Use goto instead of return in middle of the define. > Pls ref individual patches 1/3 and 2/3 for more info. > > v4->v5: > Mainly address David review comment [1]. > [1] >

Re: [PATCH RESEND 0/9 v2] fix warn_on for replace cancel

2018-11-20 Thread David Sterba
On Tue, Nov 20, 2018 at 07:56:14PM +0800, Anand Jain wrote: > These two patches were sent as part of >[PATCH 0/9 v2] fix replace-start and replace-cancel racing > before but as these aren't integrated so I am sending these again. > > The patch [1] which is in misc-next, calls

Re: [PATCH RESEND v2 1/2] btrfs: quieten warn if replace is canceled at finish

2018-11-20 Thread David Sterba
On Tue, Nov 20, 2018 at 07:56:15PM +0800, Anand Jain wrote: > When we successfully cancel the replace its scrub returns -ECANCELED, > which then passed to btrfs_dev_replace_finishing(), it cleans up based > on the scrub returned status and propagates the same -ECANCELED back > the parent function.

[PATCH 4/4] btrfs: dev-replace: open code trivial locking helpers

2018-11-20 Thread David Sterba
The dev-replace locking functions are now trivial wrappers around rw semaphore that can be used directly everywhere. No functional change. Signed-off-by: David Sterba --- fs/btrfs/dev-replace.c | 81 -- fs/btrfs/dev-replace.h | 4 --- fs/btrfs/reada.c

[PATCH 1/4] btrfs: reada: reorder dev-replace locks before radix tree preload

2018-11-20 Thread David Sterba
The read lock is going to use rw semaphore that might sleep, this is not possible in the radix tree preload section. The lock nesting is now: * device replace * radix tree preload * readahead spinlock Signed-off-by: David Sterba --- fs/btrfs/reada.c | 12 +++- 1 file changed, 7

[PATCH 2/4] btrfs: dev-replace: swich locking to rw semaphore

2018-11-20 Thread David Sterba
The this is first part of removing the custom locking and waiting scheme used for device replace. It was probably copied from extent buffer locking, but there's nothing that would require more than is provided by the common locking primitives. The rw spinlock protects waiting tasks counter in

[PATCH 0/4] Replace custom device-replace locking with rwsem

2018-11-20 Thread David Sterba
The first cleanup part went to 4.19, the actual switch from the custom locking to rswem was postponed as I found performance degradation. This turned out to be related to VM cache settings, so I'm resending the series again. The custom locking is based on rwlock protected reader/writer counters,

[PATCH 3/4] btrfs: dev-replace: remove custom read/write blocking scheme

2018-11-20 Thread David Sterba
After the rw semaphore has been added, the custom blocking using ::blocking_readers and ::read_lock_wq is redundant. The blocking logic in __btrfs_map_block is replaced by extending the time the semaphore is held, that has the same blocking effect on writes as the previous custom scheme that

[PATCH RESEND v2 1/2] btrfs: quieten warn if replace is canceled at finish

2018-11-20 Thread Anand Jain
When we successfully cancel the replace its scrub returns -ECANCELED, which then passed to btrfs_dev_replace_finishing(), it cleans up based on the scrub returned status and propagates the same -ECANCELED back the parent function. As of now only user can cancel the replace-scrub, so its ok to

[PATCH RESEND v2 2/2] btrfs: user requsted replace cancel is not an error

2018-11-20 Thread Anand Jain
As of now only user requested replace cancel can cancel the replace-scrub so no need to log error for it. Signed-off-by: Anand Jain --- v1->v2: none. fs/btrfs/dev-replace.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c

[PATCH RESEND 0/9 v2] fix warn_on for replace cancel

2018-11-20 Thread Anand Jain
These two patches were sent as part of [PATCH 0/9 v2] fix replace-start and replace-cancel racing before but as these aren't integrated so I am sending these again. The patch [1] which is in misc-next, calls btrfs_dev_replace_finishing() after replace is canceled, so the ret argument passed to

Re: [PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Qu Wenruo
On 2018/11/20 下午5:11, Nikolay Borisov wrote: > > > On 20.11.18 г. 11:07 ч., Qu Wenruo wrote: >> >> >> On 2018/11/20 下午4:51, Nikolay Borisov wrote: > >>> I'm beginning to wonder, should we document >>> btrfs_add_delayed_data_ref/btrfs_add_tree_ref arguments separate for >>> each function, or

Re: [PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Nikolay Borisov
On 20.11.18 г. 11:07 ч., Qu Wenruo wrote: > > > On 2018/11/20 下午4:51, Nikolay Borisov wrote: >> I'm beginning to wonder, should we document >> btrfs_add_delayed_data_ref/btrfs_add_tree_ref arguments separate for >> each function, or should only the differences be documented - in this >> case

Re: [PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Qu Wenruo
On 2018/11/20 下午4:51, Nikolay Borisov wrote: > > > On 20.11.18 г. 10:46 ч., Qu Wenruo wrote: >> Currently we only check @ref_root in btrfs_add_delayed_data_ref() to >> determine whether a data delayed ref is for reloc tree. >> >> Such check is insufficient as for relocation we could pass

Re: [PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Nikolay Borisov
On 20.11.18 г. 10:46 ч., Qu Wenruo wrote: > Currently we only check @ref_root in btrfs_add_delayed_data_ref() to > determine whether a data delayed ref is for reloc tree. > > Such check is insufficient as for relocation we could pass @ref_root > as the source file tree, causing qgroup to trace

[PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Qu Wenruo
Currently we only check @ref_root in btrfs_add_delayed_data_ref() to determine whether a data delayed ref is for reloc tree. Such check is insufficient as for relocation we could pass @ref_root as the source file tree, causing qgroup to trace unchanged data extents even we're only relocating

[PATCH] btrfs: qgroup: Skip delayed data ref for reloc trees

2018-11-20 Thread Qu Wenruo
Currently we only check @ref_root in btrfs_add_delayed_data_ref() to determine whether a data delayed ref is for reloc tree. Such check is insufficient as for relocation we could pass @ref_root as the source file tree, causing qgroup to trace unchanged data extents even we're only relocating

[PATCH v6 1/3] btrfs: add helper function describe_block_group()

2018-11-20 Thread Anand Jain
Improve on describe_relocation() add a common helper function to describe the block groups. Signed-off-by: Anand Jain Reviewed-by: David Sterba --- v5->v6: Use () in the body for the args sent in defines Use right indent to align '\' Use goto to out_overflow instead of return

[PATCH v6 0/3] btrfs: balance: improve kernel logs

2018-11-20 Thread Anand Jain
v5->v6: Mostly the defines non functional changes. Use goto instead of return in middle of the define. Pls ref individual patches 1/3 and 2/3 for more info. v4->v5: Mainly address David review comment [1]. [1] https://patchwork.kernel.org/patch/10425987/ pls ref to individual patch 2/3

[PATCH v6 3/3] btrfs: balance: add kernel log for end or paused

2018-11-20 Thread Anand Jain
Add a kernel log when the balance ends, either for cancel or completed or if it is paused. Signed-off-by: Anand Jain --- v5->v6: Quite soul. nothing. v4->v5: nothing. v3->v4: nothing. v2->v3: nothing. v1->v2: Moved from 2/3 to 3/3 fs/btrfs/volumes.c | 7 +++ 1 file changed, 7 insertions(+)

[PATCH v6 2/3] btrfs: balance: add args info during start and resume

2018-11-20 Thread Anand Jain
Balance arg info is an important information to be reviewed for the system audit. So this patch adds them to the kernel log. Example: ->btrfs bal start -f -mprofiles=raid1,convert=single,soft -dlimit=10..20,usage=50 /btrfs kernel: BTRFS info (device sdb): balance: start -f

Re: [PATCH v5 2/3] btrfs: balance: add args info during start and resume

2018-11-20 Thread Anand Jain
On 11/20/2018 01:07 AM, David Sterba wrote: On Wed, Nov 14, 2018 at 09:17:11PM +0800, Anand Jain wrote: Balance arg info is an important information to be reviewed for the system audit. So this patch adds them to the kernel log. Example: ->btrfs bal start -f

Re: [PATCH v5 1/3] btrfs: add helper function describe_block_group()

2018-11-20 Thread Anand Jain
Thanks for the review.. more below. On 11/20/2018 01:02 AM, David Sterba wrote: On Wed, Nov 14, 2018 at 09:17:10PM +0800, Anand Jain wrote: Improve on describe_relocation() add a common helper function to describe the block groups. Signed-off-by: Anand Jain Reviewed-by: David Sterba ---

Re: Broken filesystem, looking for guidance

2018-11-19 Thread Sébastien Luttringer
On Mon, 2018-11-19 at 14:28 +0800, Qu Wenruo wrote: > > On 2018/11/19 下午1:23, Sébastien Luttringer wrote: > > On Mon, 2018-11-19 at 13:00 +0800, Qu Wenruo wrote: > > From 20181029, it may be possible with extra help from btrfs-progs > developers. But at current point, it's near impossible to

Re: [PATCH] Btrfs: fix race between enabling quotas and subvolume creation

2018-11-19 Thread Qu Wenruo
On 2018/11/20 上午12:20, fdman...@kernel.org wrote: > From: Filipe Manana > > We have a race between enabling quotas end subvolume creation that cause > subvolume creation to fail with -EINVAL, and the following diagram shows > how it happens: > > CPU 0

Re: [PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午11:24, Filipe Manana wrote: > On Mon, Nov 19, 2018 at 2:48 PM Qu Wenruo wrote: >> >> >> >> On 2018/11/19 下午10:15, fdman...@kernel.org wrote: >>> From: Filipe Manana >>> >>> If the quota enable and snapshot creation ioctls are called concurrently >>> we can get into a deadlock

Re: [PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午11:36, Nikolay Borisov wrote: > > > On 19.11.18 г. 16:48 ч., Qu Wenruo wrote: >> There may be some qgroup reserved space related problem in such case, >> but I'm not 100% sure to foresee such problem. > > But why is this a problem - we always queue quota rescan following

Re: [PATCH v2] btrfs: add zstd compression level support

2018-11-19 Thread Omar Sandoval
On Tue, Nov 13, 2018 at 01:33:32AM +0100, David Sterba wrote: > On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrell wrote: > > From: Jennifer Liu > > > > Adds zstd compression level support to btrfs. Zstd requires > > different amounts of memory for each level, so the design had > > to be

Re: [PATCH v2] btrfs: add zstd compression level support

2018-11-19 Thread Omar Sandoval
On Tue, Nov 13, 2018 at 04:29:33PM +0300, Timofey Titovets wrote: > вт, 13 нояб. 2018 г. в 04:52, Nick Terrell : > > > > > > > > > On Nov 12, 2018, at 4:33 PM, David Sterba wrote: > > > > > > On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrell wrote: > > >> From: Jennifer Liu > > >> > > >>

Re: bad tree block start, want 705757184 have 82362368

2018-11-19 Thread Stephan Olbrich
Am Sonntag, 18. November 2018, 14:31:36 CET schrieb Anand Jain: > On 11/18/2018 03:56 PM, Stephan Olbrich wrote: > > Am Sonntag, 18. November 2018, 01:30:14 CET schrieb Qu Wenruo: > > Late on I got the same errors for my /home partition (on the same > > drive) > > as well. I have

Re: [PATCH v5 1/3] btrfs: add helper function describe_block_group()

2018-11-19 Thread David Sterba
On Wed, Nov 14, 2018 at 09:17:10PM +0800, Anand Jain wrote: > Improve on describe_relocation() add a common helper function to describe > the block groups. > > Signed-off-by: Anand Jain > Reviewed-by: David Sterba > --- > v4.1->v5: Initialize buf[128] to null. > v4->v4.1: Use strcpy(buf,

Re: [PATCH v5 2/3] btrfs: balance: add args info during start and resume

2018-11-19 Thread David Sterba
On Wed, Nov 14, 2018 at 09:17:11PM +0800, Anand Jain wrote: > Balance arg info is an important information to be reviewed for the > system audit. So this patch adds them to the kernel log. > > Example: > ->btrfs bal start -f -mprofiles=raid1,convert=single,soft > -dlimit=10..20,usage=50 /btrfs >

[PATCH] Btrfs: fix race between enabling quotas and subvolume creation

2018-11-19 Thread fdmanana
From: Filipe Manana We have a race between enabling quotas end subvolume creation that cause subvolume creation to fail with -EINVAL, and the following diagram shows how it happens: CPU 0 CPU 1 btrfs_ioctl() btrfs_ioctl_quota_ctl()

Re: [PATCH] btrfs: Add sysfs support for metadata_uuid feature

2018-11-19 Thread David Sterba
On Mon, Nov 19, 2018 at 05:37:45PM +0200, Nikolay Borisov wrote: > Since the metadata_uuid is a new incompat feature it requires the > respective sysfs hooks. This patch adds the 'metdata_uuid' feature to > be shown if it supported by the kernel. Additionally it adds >

[PATCH] btrfs: Add sysfs support for metadata_uuid feature

2018-11-19 Thread Nikolay Borisov
Since the metadata_uuid is a new incompat feature it requires the respective sysfs hooks. This patch adds the 'metdata_uuid' feature to be shown if it supported by the kernel. Additionally it adds /sys/fs/btrfs/UUID/metadata_uuid attribute which allows one to read the current metadata_uuid.

Re: [PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Nikolay Borisov
On 19.11.18 г. 16:48 ч., Qu Wenruo wrote: > There may be some qgroup reserved space related problem in such case, > but I'm not 100% sure to foresee such problem. But why is this a problem - we always queue quota rescan following QUOTA enable, that should take care of proper accounting, no? >

Re: [PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Filipe Manana
On Mon, Nov 19, 2018 at 2:48 PM Qu Wenruo wrote: > > > > On 2018/11/19 下午10:15, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > If the quota enable and snapshot creation ioctls are called concurrently > > we can get into a deadlock where the task enabling quotas will deadlock > > on

Re: [PATCH v6 2/2] btrfs: Add zstd support to grub btrfs

2018-11-19 Thread Daniel Kiper
On Mon, Nov 19, 2018 at 03:22:51PM +0100, Daniel Kiper wrote: > On Thu, Nov 15, 2018 at 02:36:03PM -0800, Nick Terrell wrote: > > - Adds zstd support to the btrfs module. > > - Adds a test case for btrfs zstd support. > > - Changes top_srcdir to srcdir in the btrfs module's lzo include > >

Re: [PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午10:15, fdman...@kernel.org wrote: > From: Filipe Manana > > If the quota enable and snapshot creation ioctls are called concurrently > we can get into a deadlock where the task enabling quotas will deadlock > on the fs_info->qgroup_ioctl_lock mutex because it attempts to lock

Re: [PATCH v2 0/5] btrfs: fix compiler warning with make W=1

2018-11-19 Thread David Sterba
On Mon, Nov 19, 2018 at 10:38:12AM +0100, Johannes Thumshirn wrote: > This patchset fixes most of the compiler warnings encountered when building > btrfs with make W=1. > > There are two more compiler warnings left in raid56.c: > CC [M] fs/btrfs/raid56.o > fs/btrfs/raid56.c: In function

Re: [PATCH v6 2/2] btrfs: Add zstd support to grub btrfs

2018-11-19 Thread Daniel Kiper
On Thu, Nov 15, 2018 at 02:36:03PM -0800, Nick Terrell wrote: > - Adds zstd support to the btrfs module. > - Adds a test case for btrfs zstd support. > - Changes top_srcdir to srcdir in the btrfs module's lzo include > following comments from Daniel Kiper about the zstd include. > > Tested on

Re: [PATCH v2 3/5] btrfs: remove unused function btrfs_sysfs_feature_update()

2018-11-19 Thread David Sterba
On Mon, Nov 19, 2018 at 10:38:15AM +0100, Johannes Thumshirn wrote: > btrfs_sysfs_feature_update() was introduced with commit 444e75169872 (btrfs: > sysfs: introduce helper for syncing bits with sysfs files) to provide a helper > which was used in 14e46e04958d (btrfs: synchronize incompat feature

[PATCH v2] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread fdmanana
From: Filipe Manana If the quota enable and snapshot creation ioctls are called concurrently we can get into a deadlock where the task enabling quotas will deadlock on the fs_info->qgroup_ioctl_lock mutex because it attempts to lock it twice, or the task creating a snapshot tries to commit the

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Filipe Manana
On Mon, Nov 19, 2018 at 11:52 AM Filipe Manana wrote: > > On Mon, Nov 19, 2018 at 11:35 AM Qu Wenruo wrote: > > > > > > > > On 2018/11/19 下午7:13, Filipe Manana wrote: > > > On Mon, Nov 19, 2018 at 11:09 AM Qu Wenruo wrote: > > >> > > >> > > >> > > >> On 2018/11/19 下午5:48, fdman...@kernel.org

Re: [PATCH v2 2/5] btrfs: remove set but not used variable err in btrfs_add_link

2018-11-19 Thread David Sterba
On Mon, Nov 19, 2018 at 10:38:14AM +0100, Johannes Thumshirn wrote: > err holds the return value of either btrfs_del_root_ref() or > btrfs_del_inode_ref() but it hasn't been checked since it's introduction with > commit fe66a05a0679 (Btrfs: improve error handling for btrfs_insert_dir_item >

Re: [PATCH v3 0/6] FSID change kernel support

2018-11-19 Thread David Sterba
On Tue, Oct 30, 2018 at 04:43:22PM +0200, Nikolay Borisov wrote: > Here is the 3rd submission for the kernel counterpart of the uuid change > patchset. The only difference is that I (hope) have adressed all cosmetic > feedback from David as well as have reworded some change logs to ease >

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午7:52, Filipe Manana wrote: > On Mon, Nov 19, 2018 at 11:35 AM Qu Wenruo wrote: >> >> >> >> On 2018/11/19 下午7:13, Filipe Manana wrote: >>> On Mon, Nov 19, 2018 at 11:09 AM Qu Wenruo wrote: On 2018/11/19 下午5:48, fdman...@kernel.org wrote: > From: Filipe

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Filipe Manana
On Mon, Nov 19, 2018 at 11:35 AM Qu Wenruo wrote: > > > > On 2018/11/19 下午7:13, Filipe Manana wrote: > > On Mon, Nov 19, 2018 at 11:09 AM Qu Wenruo wrote: > >> > >> > >> > >> On 2018/11/19 下午5:48, fdman...@kernel.org wrote: > >>> From: Filipe Manana > >>> > >>> If the quota enable and snapshot

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午7:13, Filipe Manana wrote: > On Mon, Nov 19, 2018 at 11:09 AM Qu Wenruo wrote: >> >> >> >> On 2018/11/19 下午5:48, fdman...@kernel.org wrote: >>> From: Filipe Manana >>> >>> If the quota enable and snapshot creation ioctls are called concurrently >>> we can get into a deadlock

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Filipe Manana
On Mon, Nov 19, 2018 at 11:09 AM Qu Wenruo wrote: > > > > On 2018/11/19 下午5:48, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > If the quota enable and snapshot creation ioctls are called concurrently > > we can get into a deadlock where the task enabling quotas will deadlock > > on

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Qu Wenruo
On 2018/11/19 下午5:48, fdman...@kernel.org wrote: > From: Filipe Manana > > If the quota enable and snapshot creation ioctls are called concurrently > we can get into a deadlock where the task enabling quotas will deadlock > on the fs_info->qgroup_ioctl_lock mutex because it attempts to lock it

Re: [PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread Nikolay Borisov
On 19.11.18 г. 11:48 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > If the quota enable and snapshot creation ioctls are called concurrently > we can get into a deadlock where the task enabling quotas will deadlock > on the fs_info->qgroup_ioctl_lock mutex because it attempts to

Re: [PATCH] Btrfs: fix access to available allocation bits when starting balance

2018-11-19 Thread Nikolay Borisov
On 19.11.18 г. 11:48 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > The available allocation bits members from struct btrfs_fs_info are > protected by a sequence lock, and when starting balance we access them > incorrectly in two different ways: > > 1) In the read sequence lock

Re: [PATCH v2 5/5] btrfs: use EXPORT_FOR_TESTS for conditionally shared functions

2018-11-19 Thread Nikolay Borisov
On 19.11.18 г. 11:38 ч., Johannes Thumshirn wrote: > Several functions in BTRFS are only used inside the source file they are > declared if CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not defined. However if > CONFIG_BTRFS_FS_RUN_SANITY_TESTS is defined these functions are shared with > the unit tests

Re: [PATCH v2 4/5] btrfs: introduce EXPORT_FOR_TESTS macro

2018-11-19 Thread Nikolay Borisov
On 19.11.18 г. 11:38 ч., Johannes Thumshirn wrote: > Depending on whether CONFIG_BTRFS_FS_RUN_SANITY_TESTS is set, some BTRFS > functions are either local to the file they are implemented in and thus > should be declared static or are called from within the test implementation > defined in a

[PATCH] Btrfs: fix deadlock when enabling quotas due to concurrent snapshot creation

2018-11-19 Thread fdmanana
From: Filipe Manana If the quota enable and snapshot creation ioctls are called concurrently we can get into a deadlock where the task enabling quotas will deadlock on the fs_info->qgroup_ioctl_lock mutex because it attempts to lock it twice. The following time diagram shows how this happens.

[PATCH] Btrfs: fix access to available allocation bits when starting balance

2018-11-19 Thread fdmanana
From: Filipe Manana The available allocation bits members from struct btrfs_fs_info are protected by a sequence lock, and when starting balance we access them incorrectly in two different ways: 1) In the read sequence lock loop at btrfs_balance() we use the values we read from

[PATCH v2 4/5] btrfs: introduce EXPORT_FOR_TESTS macro

2018-11-19 Thread Johannes Thumshirn
Depending on whether CONFIG_BTRFS_FS_RUN_SANITY_TESTS is set, some BTRFS functions are either local to the file they are implemented in and thus should be declared static or are called from within the test implementation defined in a different file. Introduce an EXPORT_FOR_TESTS macro which

[PATCH v2 0/5] btrfs: fix compiler warning with make W=1

2018-11-19 Thread Johannes Thumshirn
This patchset fixes most of the compiler warnings encountered when building btrfs with make W=1. There are two more compiler warnings left in raid56.c: CC [M] fs/btrfs/raid56.o fs/btrfs/raid56.c: In function ‘finish_rmw’: fs/btrfs/raid56.c:1185:6: warning: variable ‘p_stripe’ set but not used

[PATCH v2 3/5] btrfs: remove unused function btrfs_sysfs_feature_update()

2018-11-19 Thread Johannes Thumshirn
btrfs_sysfs_feature_update() was introduced with commit 444e75169872 (btrfs: sysfs: introduce helper for syncing bits with sysfs files) to provide a helper which was used in 14e46e04958d (btrfs: synchronize incompat feature bits with sysfs files). But commit e410e34fad91 (Revert "btrfs:

[PATCH v2 2/5] btrfs: remove set but not used variable err in btrfs_add_link

2018-11-19 Thread Johannes Thumshirn
err holds the return value of either btrfs_del_root_ref() or btrfs_del_inode_ref() but it hasn't been checked since it's introduction with commit fe66a05a0679 (Btrfs: improve error handling for btrfs_insert_dir_item callers) in 2012. As the error value hasn't been of any interest for 6 years we

[PATCH v2 5/5] btrfs: use EXPORT_FOR_TESTS for conditionally shared functions

2018-11-19 Thread Johannes Thumshirn
Several functions in BTRFS are only used inside the source file they are declared if CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not defined. However if CONFIG_BTRFS_FS_RUN_SANITY_TESTS is defined these functions are shared with the unit tests code. Before the introduction of the EXPORT_FOR_TESTS macro,

[PATCH v2 1/5] btrfs: remove unused drop_on_err in btrfs_mkdir()

2018-11-19 Thread Johannes Thumshirn
Up to commit 32955c5422a8 (btrfs: switch to discard_new_inode()) the drop_on_err variable in btrfs_mkdir() was used to check whether the inode had to be dropped via iput(). After commit 32955c5422a8 (btrfs: switch to discard_new_inode()) discard_new_inode() is called when err is set and inode is

Re: [PATCH V10 18/19] block: kill QUEUE_FLAG_NO_SG_MERGE

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:58:03PM +0100, Christoph Hellwig wrote: > On Thu, Nov 15, 2018 at 04:53:05PM +0800, Ming Lei wrote: > > Since bdced438acd83ad83a6c ("block: setup bi_phys_segments after > > splitting"), > > physical segment number is mainly figured out in blk_queue_split() for > > fast

Re: [PATCH V10 18/19] block: kill QUEUE_FLAG_NO_SG_MERGE

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 06:18:11PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:53:05PM +0800, Ming Lei wrote: > > Since bdced438acd83ad83a6c ("block: setup bi_phys_segments after > > splitting"), > > physical segment number is mainly figured out in blk_queue_split() for > > fast

Re: [PATCH V10 17/19] block: don't use bio->bi_vcnt to figure out segment number

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 06:11:40PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:53:04PM +0800, Ming Lei wrote: > > It is wrong to use bio->bi_vcnt to figure out how many segments > > there are in the bio even though CLONED flag isn't set on this bio, > > because this bio may be

Re: [PATCH V10 15/19] block: always define BIO_MAX_PAGES as 256

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 05:59:36PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:53:02PM +0800, Ming Lei wrote: > > Now multi-page bvec can cover CONFIG_THP_SWAP, so we don't need to > > increase BIO_MAX_PAGES for it. > > You mentioned to it in the cover letter, but this needs more

Re: [PATCH V10 14/19] block: enable multipage bvecs

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:53:08PM +0100, Christoph Hellwig wrote: > > - > > - if (page == bv->bv_page && off == bv->bv_offset + bv->bv_len) { > > - bv->bv_len += len; > > - bio->bi_iter.bi_size += len; > > - return true; > > -

Re: [PATCH V10 14/19] block: enable multipage bvecs

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 05:56:27PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:53:01PM +0800, Ming Lei wrote: > > This patch pulls the trigger for multi-page bvecs. > > > > Now any request queue which supports queue cluster will see multi-page > > bvecs. > > > > Cc: Dave Chinner >

Re: [PATCH V10 13/19] iomap & xfs: only account for new added page

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:49:36PM +0100, Christoph Hellwig wrote: > I'd much rather have __bio_try_merge_page only do merges in > the same page, and have a new __bio_try_merge_segment that does > multi-page merges. This will keep the accounting a lot simpler. Looks this way is clever, will do

Re: [PATCH V10 13/19] iomap & xfs: only account for new added page

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 05:46:58PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:53:00PM +0800, Ming Lei wrote: > > After multi-page is enabled, one new page may be merged to a segment > > even though it is a new added page. > > > > This patch deals with this issue by post-check in

Re: [PATCH V10 12/19] block: allow bio_for_each_segment_all() to iterate over multi-page bvec

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 05:22:45PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:59PM +0800, Ming Lei wrote: > > This patch introduces one extra iterator variable to > > bio_for_each_segment_all(), > > then we can allow bio_for_each_segment_all() to iterate over multi-page > >

Re: [PATCH V10 12/19] block: allow bio_for_each_segment_all() to iterate over multi-page bvec

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 01:42:52PM +0100, David Sterba wrote: > On Thu, Nov 15, 2018 at 04:52:59PM +0800, Ming Lei wrote: > > diff --git a/block/blk-zoned.c b/block/blk-zoned.c > > index 13ba2011a306..789b09ae402a 100644 > > --- a/block/blk-zoned.c > > +++ b/block/blk-zoned.c > > @@ -123,6 +123,7

Re: [PATCH V10 11/19] bcache: avoid to use bio_for_each_segment_all() in bch_bio_alloc_pages()

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:46:45PM +0100, Christoph Hellwig wrote: > > - bio_for_each_segment_all(bv, bio, i) { > > + for (i = 0, bv = bio->bi_io_vec; i < bio->bi_vcnt; bv++) { > > This really needs a comment. Otherwise it looks fine to me. OK, will do it in next version. Thanks, Ming

Re: [PATCH V10 11/19] bcache: avoid to use bio_for_each_segment_all() in bch_bio_alloc_pages()

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 04:44:02PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:58PM +0800, Ming Lei wrote: > > bch_bio_alloc_pages() is always called on one new bio, so it is safe > > to access the bvec table directly. Given it is the only kind of this > > case, open code the bvec

Re: [PATCH V10 10/19] block: loop: pass multi-page bvec to iov_iter

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 04:40:22PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:57PM +0800, Ming Lei wrote: > > iov_iter is implemented with bvec itererator, so it is safe to pass > > multipage bvec to it, and this way is much more efficient than > > passing one page in each bvec. >

Re: [PATCH V10 08/19] btrfs: move bio_pages_all() to btrfs

2018-11-19 Thread Christoph Hellwig
On Mon, Nov 19, 2018 at 04:19:24PM +0800, Ming Lei wrote: > On Fri, Nov 16, 2018 at 02:38:45PM +0100, Christoph Hellwig wrote: > > On Thu, Nov 15, 2018 at 04:52:55PM +0800, Ming Lei wrote: > > > BTRFS is the only user of this helper, so move this helper into > > > BTRFS, and implement it via

Re: [PATCH V10 09/19] block: introduce bio_bvecs()

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:45:41PM +0100, Christoph Hellwig wrote: > On Thu, Nov 15, 2018 at 04:52:56PM +0800, Ming Lei wrote: > > There are still cases in which we need to use bio_bvecs() for get the > > number of multi-page segment, so introduce it. > > The only user in your final tree seems to

Re: [PATCH V10 08/19] btrfs: move bio_pages_all() to btrfs

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:38:45PM +0100, Christoph Hellwig wrote: > On Thu, Nov 15, 2018 at 04:52:55PM +0800, Ming Lei wrote: > > BTRFS is the only user of this helper, so move this helper into > > BTRFS, and implement it via bio_for_each_segment_all(), since > > bio->bi_vcnt may not equal to

Re: [PATCH V10 08/19] btrfs: move bio_pages_all() to btrfs

2018-11-19 Thread Ming Lei
On Thu, Nov 15, 2018 at 04:23:56PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:55PM +0800, Ming Lei wrote: > > BTRFS is the only user of this helper, so move this helper into > > BTRFS, and implement it via bio_for_each_segment_all(), since > > bio->bi_vcnt may not equal to number

Re: [PATCH V10 07/19] btrfs: use bvec_last_segment to get bio's last page

2018-11-19 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:37:10PM +0100, Christoph Hellwig wrote: > On Thu, Nov 15, 2018 at 04:52:54PM +0800, Ming Lei wrote: > > index 2955a4ea2fa8..161e14b8b180 100644 > > --- a/fs/btrfs/compression.c > > +++ b/fs/btrfs/compression.c > > @@ -400,8 +400,11 @@ blk_status_t

Re: [PATCH V10 05/19] block: introduce bvec_last_segment()

2018-11-18 Thread Ming Lei
On Thu, Nov 15, 2018 at 03:23:56PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:52PM +0800, Ming Lei wrote: > > BTRFS and guard_bio_eod() need to get the last singlepage segment > > from one multipage bvec, so introduce this helper to make them happy. > > > > Cc: Dave Chinner > >

Re: [PATCH V10 04/19] block: use bio_for_each_bvec() to map sg

2018-11-18 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:33:14PM +0100, Christoph Hellwig wrote: > > + if (!*sg) > > + return sglist; > > + else { > > No need for an else after an early return. OK, good catch! Thanks, Ming

Re: [PATCH V10 03/19] block: use bio_for_each_bvec() to compute multi-page bvec count

2018-11-18 Thread Ming Lei
On Thu, Nov 15, 2018 at 12:20:28PM -0800, Omar Sandoval wrote: > On Thu, Nov 15, 2018 at 04:52:50PM +0800, Ming Lei wrote: > > First it is more efficient to use bio_for_each_bvec() in both > > blk_bio_segment_split() and __blk_recalc_rq_segments() to compute how > > many multi-page bvecs there are

Re: Broken filesystem, looking for guidance

2018-11-18 Thread Qu Wenruo
On 2018/11/19 下午1:23, Sébastien Luttringer wrote: > On Mon, 2018-11-19 at 13:00 +0800, Qu Wenruo wrote: >> May I ask for the oldest kernel ran on the fs? > Looks like the first kernel on the host is 3.12.2. OK, then it makes sense for such strange dir ref corruption. > >> To make it clear

Re: Broken filesystem, looking for guidance

2018-11-18 Thread Sébastien Luttringer
On Mon, 2018-11-19 at 13:00 +0800, Qu Wenruo wrote: > May I ask for the oldest kernel ran on the fs? Looks like the first kernel on the host is 3.12.2. > To make it clear again, any --init-* should only be used when you're > completely sure what is going to happen (at code level). Copy. > >

Re: Broken filesystem, looking for guidance

2018-11-18 Thread Qu Wenruo
On 2018/11/19 上午11:29, Sébastien Luttringer wrote: > On Mon, 2018-11-19 at 08:48 +0800, Qu Wenruo wrote: >> >> On 2018/11/19 上午3:37, Sébastien Luttringer wrote: >> >> You haven't post btrfs check --readonly output, thus not helpful. > > The oldest `btrfs check --readonly' output I saved is from

Re: [PATCH V10 01/19] block: introduce multi-page page bvec helpers

2018-11-18 Thread Ming Lei
On Sun, Nov 18, 2018 at 08:10:14PM -0700, Jens Axboe wrote: > On 11/18/18 7:23 PM, Ming Lei wrote: > > On Fri, Nov 16, 2018 at 02:13:05PM +0100, Christoph Hellwig wrote: > >>> -#define bvec_iter_page(bvec, iter) \ > >>> +#define mp_bvec_iter_page(bvec, iter)

Re: [PATCH V10 02/19] block: introduce bio_for_each_bvec()

2018-11-18 Thread Ming Lei
On Fri, Nov 16, 2018 at 02:30:28PM +0100, Christoph Hellwig wrote: > > +static inline void __bio_advance_iter(struct bio *bio, struct bvec_iter > > *iter, > > + unsigned bytes, bool mp) > > I think these magic 'bool np' arguments and wrappers over wrapper > don't

Re: Broken filesystem, looking for guidance

2018-11-18 Thread Sébastien Luttringer
On Mon, 2018-11-19 at 08:48 +0800, Qu Wenruo wrote: > > On 2018/11/19 上午3:37, Sébastien Luttringer wrote: > > You haven't post btrfs check --readonly output, thus not helpful. The oldest `btrfs check --readonly' output I saved is from 29th October with a vanilla linux 4.19.0 kernel. Filename:

<    2   3   4   5   6   7   8   9   10   11   >