Re: [PATCH 1/2] btrfs: track odirect bytes in flight

2019-04-12 Thread Nikolay Borisov
On 10.04.19 г. 22:56 ч., Josef Bacik wrote: > When diagnosing a slowdown of generic/224 I noticed we were wasting a > lot of time in shrink_delalloc() despite all writes being O_DIRECT > writes. O_DIRECT writes still have outstanding extents, but obviously > cannot be directly flushed, instead

Re: [PATCH RFC] fstests: generic/077: fix populate fs use _fill_fs()

2019-04-12 Thread Nikolay Borisov
On 12.04.19 г. 10:15 ч., Qu Wenruo wrote: > > > On 2019/4/12 下午1:24, Anand Jain wrote: >> Test case generic/077 uses files under /lib or /usr to fill SCRATCH_MNT. >> If /usr or /lib is below 256mb then test fails to run, or if these dirs >> are too large it takes a long time for the cp to fini

Re: [PATCH v2] btrfs: Remove bio_offset argument from submit_bio_hook

2019-04-11 Thread Nikolay Borisov
On 11.04.19 г. 16:29 ч., Johannes Thumshirn wrote: > Looks good, > Reviewed-by: Johannes Thumshirn > > Here you could also add the removal of bio_offset from async_submit_bio. > Actually no, we can't since for DIO case we do use it from: btrfs_submit_dio_bio->btrfs_wq_submit_bio

Re: [PATCH 4/6] btrfs: Pass 0 for bio_offset to btrfs_wq_submit_bio

2019-04-11 Thread Nikolay Borisov
On 11.04.19 г. 16:10 ч., Johannes Thumshirn wrote: > On 11/04/2019 15:09, Johannes Thumshirn wrote: >> I think we can also kill 'async_submit_bio::bio_offset' as no other >> value than 0 is ever used. > > Right, this is what the following patches do. Ignore this. Actually they don't particular

Re: [PATCH 2/6] btrfs: Change submit_bio_hook to taking an inode directly

2019-04-11 Thread Nikolay Borisov
On 11.04.19 г. 14:18 ч., Johannes Thumshirn wrote: > On 10/04/2019 16:24, Nikolay Borisov wrote: >> The only possible 'private_data' that is passed to this function is >> actually an inode. Make that explicit by changing the signature of the >> call back. No fu

Re: [PATCH v3] btrfs-progs: Don't BUG_ON() when write_dev_supers() fails

2019-04-11 Thread Nikolay Borisov
_ctree_super() > callers, btrfs_commit_transaction() and close_ctree_fs_info() to > handle the error correctly. > > Signed-off-by: Qu Wenruo This seems more compact than the previous version. Reviewed-by: Nikolay Borisov > --- > Changelog: > v2: > - Make write_ctree_super() cal

Re: [PATCH v2] btrfs-progs: Don't BUG_ON() when write_dev_supers() fails

2019-04-11 Thread Nikolay Borisov
On 11.04.19 г. 12:57 ч., Qu Wenruo wrote: > [BUG] > Since commit "btrfs-progs: disk-io: Flush to ensure super block write is > FUA" mkfs-tests/017 will fail like: > > == RUN MUSTFAIL /home/adam/btrfs-progs/mkfs.btrfs -K -f > /dev/mapper/btrfs-progs-thin-vol > ERROR: failed to write sup

Re: [PATCH v5 RESEND] btrfs-progs: dump-tree: add noscan option

2019-04-10 Thread Nikolay Borisov
On 11.04.19 г. 7:53 ч., Qu Wenruo wrote: > > > On 2019/4/4 下午3:29, Anand Jain wrote: >> From: Anand Jain >> >> The cli 'btrfs inspect dump-tree ' will scan for the partner devices >> if any by default. >> >> So as of now you can not inspect each mirrored device independently. >> >> This patch

Re: [PATCH] btrfs-progs: Don't BUG_ON() when write_dev_supers() fails

2019-04-10 Thread Nikolay Borisov
nores the retval (which is the retval of write_all_supers). I wonder if there is anything more that could be done in this context though? I'm in favor of removing the BUG_ON though. If there isn't anything else to do in btrfs_commit_transaction: Reviewed-by: Nikolay Borisov > > Si

[PATCH v2] btrfs: Remove bio_offset argument from submit_bio_hook

2019-04-10 Thread Nikolay Borisov
None of the implementers of the submit_bio_hook use the bio_offset parameter, simply remove it. No functional changes. Signed-off-by: Nikolay Borisov --- Change since v1: * Remove code dealing with bio internals in submit_one_bio. fs/btrfs/disk-io.c | 4 ++-- fs/btrfs/extent_io.c | 10

Re: [PATCH 6/6] btrfs: Remove bio_offset argument from submit_bio_hook

2019-04-10 Thread Nikolay Borisov
On 10.04.19 г. 17:24 ч., Nikolay Borisov wrote: > None of the implementers of the submit_bio_hook use the bio_offset > parameter, simply remove it. No functional changes. > > Signed-off-by: Nikolay Borisov > --- > fs/btrfs/disk-io.c | 4 ++-- > fs/btrfs/extent_io.c |

[PATCH 5/6] btrfs: Always pass 0 bio_offset for btree_submit_bio_start

2019-04-10 Thread Nikolay Borisov
The btree submit hook queues the async csum and forwards the bio_offset parameter passed to btree_submit_bio_hook. This is redundant since btree_submit_bio_start calls btree_csum_one_bio which doesn't use the offset at all. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs

[PATCH 2/6] btrfs: Change submit_bio_hook to taking an inode directly

2019-04-10 Thread Nikolay Borisov
The only possible 'private_data' that is passed to this function is actually an inode. Make that explicit by changing the signature of the call back. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/disk-io.c | 5 ++--- fs/btrfs/extent_io.h | 2 +- fs/btrfs/inode

[PATCH 4/6] btrfs: Pass 0 for bio_offset to btrfs_wq_submit_bio

2019-04-10 Thread Nikolay Borisov
for further simplifications down the line. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/inode.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 07cf7050c99f..6e5f1afa7407 100644 --- a/fs/btrfs/inode.c +++ b/fs

[PATCH 6/6] btrfs: Remove bio_offset argument from submit_bio_hook

2019-04-10 Thread Nikolay Borisov
None of the implementers of the submit_bio_hook use the bio_offset parameter, simply remove it. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/disk-io.c | 4 ++-- fs/btrfs/extent_io.c | 6 ++ fs/btrfs/extent_io.h | 3 +-- fs/btrfs/inode.c | 5 +++-- 4 files changed

[PATCH 3/6] btrfs: Remove 'tree' argument from read_extent_buffer_pages

2019-04-10 Thread Nikolay Borisov
This function always uses the btree inode's io_tree. Stop taking the tree as a function argument and instead access it internally from read_extent_buffer_pages. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/disk-io.c | 12 +++- fs/btrfs/extent_io.c | 4 ++-

[PATCH 0/6] Simplifications around submit_bio_hook

2019-04-10 Thread Nikolay Borisov
e_bio 313 74-239 Total: Before=1072719, After=1072448, chg -0.03% Nikolay Borisov (6): btrfs: Define submit_bio_hook's type directly btrfs: Change submit_bio_hook to taking an inode directly btrfs: Remove 'tree' argument from read_extent_buffer_pages

[PATCH 1/6] btrfs: Define submit_bio_hook's type directly

2019-04-10 Thread Nikolay Borisov
There is no need to use a typedef to define the type of the function and then use that to define the respective member in extent_io_ops. Define struct's member directly. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.h | 7 +++ 1 file changed, 3 inser

[PATCH] btrfs: Remove redundant inode argument from btrfs_add_ordered_sum

2019-04-10 Thread Nikolay Borisov
Ordered csums are keyed off of a btrfs_ordered_extent, which already has a reference to the inode. This implies that an explicit inode argument is redundant. So remove it. Signed-off-by: Nikolay Borisov --- fs/btrfs/file-item.c| 4 ++-- fs/btrfs/ordered-data.c | 5 ++--- fs/btrfs/ordered

Re: Is it a good idea to replace current eb leakage detector with bcc/ebpf based one?

2019-04-10 Thread Nikolay Borisov
On 10.04.19 г. 10:11 ч., Qu Wenruo wrote: > > > On 2019/4/10 下午3:01, Nikolay Borisov wrote: >> >> >> On 10.04.19 г. 9:58 ч., Qu Wenruo wrote: >>> >>> >>> On 2019/4/10 下午2:51, Nikolay Borisov wrote: >>>> >>>>

Re: Is it a good idea to replace current eb leakage detector with bcc/ebpf based one?

2019-04-10 Thread Nikolay Borisov
On 10.04.19 г. 9:58 ч., Qu Wenruo wrote: > > > On 2019/4/10 下午2:51, Nikolay Borisov wrote: >> >> >> On 10.04.19 г. 9:34 ч., Qu Wenruo wrote: >>> Hi, >>> >>> As we have memleak.py in bcc tools, and it provides better info >>>

Re: Is it a good idea to replace current eb leakage detector with bcc/ebpf based one?

2019-04-09 Thread Nikolay Borisov
On 10.04.19 г. 9:34 ч., Qu Wenruo wrote: > Hi, > > As we have memleak.py in bcc tools, and it provides better info > including the calling stack, I'm wondering if we should replace current > eb leakage check with bcc based one. > > Any idea on this? Why do you want to change it, given that th

Re: [PATCH 1/2] btrfs-progs: add 'btrfs inspect-internal csum-dump' command

2019-04-09 Thread Nikolay Borisov
rn Overall looks good and not nearly as ugly as I expected so the csum tree is not _THAT_ cumbersome to work with after all :). However, do you intend to submit tests with files with specific patterns to ensure we do not regress? Also I have some minor comments below but they ar

Re: [PATCH v2] fstests: btrfs/048: amend property validation cases

2019-04-07 Thread Nikolay Borisov
On 7.04.19 г. 14:54 ч., Anand Jain wrote: > On 6/4/19 8:02 pm, Eryu Guan wrote: >> On Fri, Apr 05, 2019 at 04:21:10PM +0300, Nikolay Borisov wrote: >>> >>> >>> On 3.04.19 г. 20:04 ч., Anand Jain wrote: >>>> Add more property validation ca

Re: [PATCH] btrfs: Record error in btree_csum_one_bio()

2019-04-05 Thread Nikolay Borisov
On 6.04.19 г. 6:13 ч., Qu Wenruo wrote: > Since commit 6dc4f100c175 ("block: allow bio_for_each_segment_all() to > iterate over multi-page bvec"), break will only break the inner loop of > bio_for_each_segment_all(), unable to break the outer loop. Breaking of outer loop should be implemented w

Re: [PATCH v3.1 1/9] btrfs: delayed-ref: Introduce better documented delayed ref structures

2019-04-05 Thread Nikolay Borisov
On 6.04.19 г. 2:47 ч., Qu Wenruo wrote: > > + /* For non-skinny metadata, no special member needed */ > +}; > + > +struct btrfs_ref { The structure name sounds a bit generic, but I think we can keep it short. There are no other btrfs-specific references that could

Re: [PATCH v2] fstests: btrfs/048: amend property validation cases

2019-04-05 Thread Nikolay Borisov
On 3.04.19 г. 20:04 ч., Anand Jain wrote: > Add more property validation cases which are fixed by the patches [1] > [1] > btrfs: fix vanished compression property after failed set > btrfs: fix zstd compression parameter > > Signed-off-by: Anand Jain Reviewed-b

Re: [PATCH 1/2] btrfs: tree-checker: Remove comprehensive root owner check

2019-04-03 Thread Nikolay Borisov
On 4.04.19 г. 6:47 ч., Qu Wenruo wrote: > Commit 1ba98d086fe3 ("Btrfs: detect corruption when non-root leaf has > zero item") introduced comprehensive root owner checker. > > However it's pretty expensive tree search to locate the owner root, > especially when it get reused by mandatory read an

Re: [PATCH 2/2] btrfs: tree-checker: Remove comprehensive root owner check

2019-04-03 Thread Nikolay Borisov
eck, now write time tree check > can be merged with btrfs_check_leaf_full(). > > Signed-off-by: Qu Wenruo Reviewed-by: Nikolay Borisov

Re: [PATCH 1/2] btrfs: tree-checker: Make write time tree checker verify the content of leaves

2019-04-03 Thread Nikolay Borisov
body. So how about: "Perform item body validation during write" Reviewed-by: Nikolay Borisov > --- > fs/btrfs/tree-checker.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c > index 4318e3e6

Re: Interesting btrfs csum and tree-checker performance penalty analyse

2019-04-03 Thread Nikolay Borisov
On 3.04.19 г. 11:54 ч., Qu Wenruo wrote: > Hi, > > Recently Intel LKP performance test is reporting regression of btrfs > performance. > > It points to tree-checker code, and since I'm poking around the > bcc/ebpf, I spend some time to do an interesting look into the > performance penalty abou

Re: life time of backup roots

2019-04-03 Thread Nikolay Borisov
On 3.04.19 г. 10:05 ч., Qu Wenruo wrote: > > > On 2019/4/3 上午11:58, Chris Murphy wrote: >> I'm sometimes seeing the same backup_tree_root used more than once. >> Below you'll see backup 0 and backup 2 have the same address, >> different generation. The concern is if this suggests backup 2 is >

Re: [PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > When the property fails to pass the prop_handlers::validate() check, the > thread should exit with no changes in the kernel, but as we are starting > the transaction too early, we have just updated the generation even if > there is no change. > > For

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread Nikolay Borisov
ch) but should the reservation be released after setxattr is called, even if it succeeds, since we need to hold it until the reservation is committed. E.g. if we have to inherit 2 props then in the first iteration of the loop we reserve space, we call btrfs_Setxattr with enough reserv

Re: [PATCH 2/4 RESEND] btrfs: fix vanished compression property after failed set

2019-04-02 Thread Nikolay Borisov
length in strncmp(). > > Signed-off-by: Anand Jain This mimics the code in prop_compression_apply and so if ->validate() fails we just return without changing anything. So this LGTM: Reviewed-by: Nikolay Borisov > --- > fs/btrfs/props.c | 6 +++--- > 1 file changed, 3 inser

Re: [PATCH] fstests: btrfs/048: amend property validation cases

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > Add more property validation cases which are fixed by the patches [1] > [1] > btrfs: fix property validate fail should not increment generation > btrfs: open code btrfs_set_prop in inherit_prop > btrfs: fix vanished compression property after fa

Re: [PATCH v4 13/15] btrfs: Optimize unallocated chunks discard

2019-04-02 Thread Nikolay Borisov
On 1.04.19 г. 21:44 ч., David Sterba wrote: > On Wed, Mar 27, 2019 at 02:24:16PM +0200, Nikolay Borisov wrote: >> Currently unallocated chunks are always trimmed. For example >> 2 consecutive trims on large storage would trim freespace twice >> irrespective of whether t

Re: [PATCH v4 03/15] btrfs: Handle pending/pinned chunks before blockgroup relocation during device shrink

2019-04-01 Thread Nikolay Borisov
On 1.04.19 г. 21:26 ч., David Sterba wrote: > On Wed, Mar 27, 2019 at 02:24:06PM +0200, Nikolay Borisov wrote: >> During device shrink pinned/pending chunks (i.e those which have been >> deleted/created respectively, in the current transaction and haven't >> touched

Re: [PATCH v4 08/15] btrfs: Stop using call_rcu for device freeing

2019-04-01 Thread Nikolay Borisov
On 1.04.19 г. 20:07 ч., David Sterba wrote: > On Wed, Mar 27, 2019 at 02:24:11PM +0200, Nikolay Borisov wrote: >> btrfs_device structs are freed from RCU context since device iteration >> is protected by RCU. Currently this is achieved by using call_rcu since >> no blocking

[PATCH v2] btrfs: Enable btrfs/003

2019-04-01 Thread Nikolay Borisov
not deletable. Signed-off-by: Nikolay Borisov --- Changes since v1: * Rename the function to deletable_scratch_dev_pool common/rc | 12 tests/btrfs/003 | 19 ++- 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/common/rc b/common/rc index

[PATCH 1/2] btrfs: Use kvmalloc for allocating compressed path context

2019-04-01 Thread Nikolay Borisov
it by switching to using kvmalloc. Signed-off-by: Nikolay Borisov --- fs/btrfs/inode.c | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 43ee890c715f..85f61913b92d 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -28,6

[PATCH 2/2] btrfs: Switch memory allocations in async csum calculation path to kvmalloc

2019-04-01 Thread Nikolay Borisov
viate the issue by switching the allocation from using kmalloc to using kvmalloc. For small writes this is unlikely to have any visible effect since kmalloc will still satisfy allocation requests. For larger requests the code will just fallback to vmalloc. Signed-off-by: Nikolay Borisov --- fs/bt

[PATCH 0/2] Preamture OOM fixes

2019-04-01 Thread Nikolay Borisov
icial either ways. Nikolay Borisov (2): btrfs: Use kvmalloc for allocating compressed path context btrfs: Switch memory allocations in async csum calculation path to kvmalloc fs/btrfs/file-item.c| 15 +++ fs/btrfs/inode.c| 9 +++-- fs/btrfs/ordered-data.c | 3 +

Re: [PATCH 05/15] btrfs: return whether extent is nocow or not

2019-03-31 Thread Nikolay Borisov
On 26.03.19 г. 21:02 ч., Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > We require this to set the IOMAP_F_COW flag in > iomap structure, in the later patches. > > Signed-off-by: Goldwyn Rodrigues > --- > fs/btrfs/ctree.h | 2 +- > fs/btrfs/inode.c | 9 +++-- > 2 files changed,

Re: [PATCH 7/7] btrfs: tests/fsck: Add test image for free space cache mode repair

2019-03-29 Thread Nikolay Borisov
On 25.03.19 г. 10:22 ч., Qu Wenruo wrote: > The image has one free space cache inode with invalid mode (0). > item 9 key (256 INODE_ITEM 0) itemoff 13702 itemsize 160 > generation 30 transid 30 size 65536 nbytes 1507328 > block group 0 mode 0 links 1 uid 0

Re: [PATCH 5/7] btrfs: check/lowmem: Check and repair free space cache inode mode

2019-03-29 Thread Nikolay Borisov
On 29.03.19 г. 13:02 ч., Qu Wenruo wrote: > [snip] >>> +/* >>> + * For free space inodes, we can't call check_inode_item() as free space >>> + * cache inode doesn't have INODE_REF. >>> + * We just check its inode mode. >>> + */ >>> +int check_repair_free_space_inode(struct btrfs_fs_info *fs_info

Re: [PATCH 5/7] btrfs: check/lowmem: Check and repair free space cache inode mode

2019-03-29 Thread Nikolay Borisov
On 25.03.19 г. 10:22 ч., Qu Wenruo wrote: > Unlike inodes in fs roots, we don't really check the inode items in root > tree, in fact we just skip everything other than ROOT_ITEM and ROOT_REF. > > This makes invalid inode items sneak into root tree. > For example: > item 9 key (256 INODE

Re: [PATCH 3/7] btrfs-progs: check/lowmem: Repair invalid inode mode in root tree

2019-03-29 Thread Nikolay Borisov
On 25.03.19 г. 10:22 ч., Qu Wenruo wrote: > In root tree, we only have 2 types of inodes: > - ROOT_TREE_DIR inode > Its mode is fixed to 40755 > - free space cache inodes > Its mode is fixed to 100600 > > This patch will add the ability to repair such inodes to lowmem mode. > For fs/subvolu

Re: [PATCH] btrfs: inode: Check if we get a valid inode before checking its mode.

2019-03-29 Thread Nikolay Borisov
On 29.03.19 г. 9:12 ч., Qu Wenruo wrote: > We should check the inode validation before checking the imode. > > As btrfs_iget() could return ERR_PTR(), unlike previous path, we need to > verify the inode before checking the imode. > > Signed-off-by: Qu Wenruo Reviewed-b

Re: [PATCH v4 00/15] FITRIM improvement

2019-03-28 Thread Nikolay Borisov
On 29.03.19 г. 1:18 ч., David Sterba wrote: > On Wed, Mar 27, 2019 at 02:24:03PM +0200, Nikolay Borisov wrote: >> Here is the (hopefully final) v4 of the fitrim patches. Main changes since >> v3: > > Nope, tests don't like it. It's v4 applied from mails on top

Re: [PATCH 5/7] fsstress: allow afsync on directories too

2019-03-28 Thread Nikolay Borisov
On 28.03.19 г. 23:36 ч., Nikolay Borisov wrote: > > > On 28.03.19 г. 20:54 ч., fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Currently the afsync function can only be performed against regular files. >> Allow it to operate on directories too, to

Re: [PATCH 5/7] fsstress: allow afsync on directories too

2019-03-28 Thread Nikolay Borisov
On 28.03.19 г. 20:54 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > Currently the afsync function can only be performed against regular files. > Allow it to operate on directories too, to increase test coverage and allow > for chances of finding bugs in the filesystem implementation

Re: [PATCH 1/7] btrfs: Preallocate chunks in cow_file_range_async

2019-03-28 Thread Nikolay Borisov
On 28.03.19 г. 16:11 ч., David Sterba wrote: > On Thu, Mar 28, 2019 at 02:49:30PM +0200, Nikolay Borisov wrote: >> >> >> On 27.03.19 г. 19:23 ч., David Sterba wrote: >>> On Tue, Mar 12, 2019 at 05:20:24PM +0200, Nikolay Borisov wrote: >>>> @@ -1190,45 +

Re: [PATCH 1/7] btrfs: Preallocate chunks in cow_file_range_async

2019-03-28 Thread Nikolay Borisov
On 28.03.19 г. 14:49 ч., Nikolay Borisov wrote: > > > On 27.03.19 г. 19:23 ч., David Sterba wrote: >> On Tue, Mar 12, 2019 at 05:20:24PM +0200, Nikolay Borisov wrote: >>> @@ -1190,45 +1201,71 @@ static int cow_file_range_async(struct inode >>>

Re: [PATCH 1/7] btrfs: Preallocate chunks in cow_file_range_async

2019-03-28 Thread Nikolay Borisov
On 27.03.19 г. 19:23 ч., David Sterba wrote: > On Tue, Mar 12, 2019 at 05:20:24PM +0200, Nikolay Borisov wrote: >> @@ -1190,45 +1201,71 @@ static int cow_file_range_async(struct inode *inode, >> struct page *locked_page, >> uns

[PATCH v4 08/15] btrfs: Stop using call_rcu for device freeing

2019-03-27 Thread Nikolay Borisov
is in preparation for these changes by simply switching from RCU callbacks to explicit calls of synchronize_rcu and calling btrfs_free_device directly. Signed-off-by: Nikolay Borisov --- fs/btrfs/volumes.c | 20 fs/btrfs/volumes.h | 1 - 2 files changed, 8 insertions(+), 13

[PATCH v4 11/15] btrfs: Remove 'trans' argument from find_free_dev_extent(_start)

2019-03-27 Thread Nikolay Borisov
Now that those function no longer require a handle to transaction to inspect pending/pinned chunks the argument can be removed. At the same time also remove any surrounding code which acquired the handle. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 36

[PATCH v4 14/15] btrfs: Implement find_first_clear_extent_bit

2019-03-27 Thread Nikolay Borisov
This function is very similar to find_first_extent_bit except that it locates the first contiguous span of space which does not have bits set. It's intended use is in the freespace trimming code. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c

[PATCH v4 15/15] btrfs: Switch btrfs_trim_free_extents to find_first_clear_extent_bit

2019-03-27 Thread Nikolay Borisov
code always searches in-memory data structure to figure out the space range which contains the requested which should result in speed oups. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 89 -- 1 file changed, 26 insertions(+), 63 deletions(-)

[PATCH v4 07/15] btrfs: Implement set_extent_bits_nowait

2019-03-27 Thread Nikolay Borisov
It will be used in a future patch that will require modifying an extent_io_tree struct under a spinlock. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c | 7 +++ fs/btrfs/extent_io.h | 2 ++ 2 files changed, 9 insertions(+) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c

[PATCH v4 13/15] btrfs: Optimize unallocated chunks discard

2019-03-27 Thread Nikolay Borisov
asn't been changed since the last time discard was issued. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 57 +- fs/btrfs/extent_io.h | 8 +- fs/btrfs/extent_map.c | 4 ++- 3 files changed, 66 insertions(+), 3 deletions(-) diff

[PATCH v4 09/15] btrfs: replace pending/pinned chunks lists with io tree

2019-03-27 Thread Nikolay Borisov
eration. We can also use this io tree to mark which chunks have already been trimmed so we don't repeat the operation. Signed-off-by: Nikolay Borisov --- fs/btrfs/ctree.h| 6 --- fs/btrfs/disk-io.c | 11 - fs/btrfs/extent-tree.c | 28 --- fs/btrfs/ext

[PATCH v4 00/15] FITRIM improvement

2019-03-27 Thread Nikolay Borisov
eing the mapping tree, this is prep for patch 9. This also fixes a bunch of places that weren't correctly freeing the extent mapping tree upon device close. * Fixed ASSERT condition in patch 2 Jeff Mahoney (1): btrfs: replace pending/pinned chunks lists with io tree Nikolay Borisov

[PATCH v4 03/15] btrfs: Handle pending/pinned chunks before blockgroup relocation during device shrink

2019-03-27 Thread Nikolay Borisov
ad to making another go in the body of the function. Since there is no hard requirement to perform pinned/pending chunks handling after the relocation loop, move the code before it. This leads to simplifying the code flow around - i.e no need to use 'goto again'. Signed-off-by: Nikolay B

[PATCH v4 04/15] btrfs: Rename and export clear_btree_io_tree

2019-03-27 Thread Nikolay Borisov
This function is going to be used to clear out the device extent allocation information. Give it a more generic name and export it. This is in preparation to replacing the pending/pinned chunk lists with an extent tree. No functional changes. Reviewed-by: David Sterba Signed-off-by: Nikolay

[PATCH v4 02/15] btrfs: combine device update operations during transaction commit

2019-03-27 Thread Nikolay Borisov
ays add the device to a list when we change bytes_used or disk_total_size, there's no harm in copying both values at once. Signed-off-by: Nikolay Borisov --- fs/btrfs/dev-replace.c | 2 +- fs/btrfs/disk-io.c | 7 fs/btrfs/transaction.c | 5 ++- fs/btrfs/transaction.h | 1 + fs

[PATCH v4 10/15] btrfs: Transpose btrfs_close_devices/btrfs_mapping_tree_free in close_ctree

2019-03-27 Thread Nikolay Borisov
kernel could crash due to UAF. Signed-off-by: Nikolay Borisov --- fs/btrfs/disk-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index c5900ade4094..911e2fb1f157 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -4053,8

[PATCH v4 12/15] btrfs: Factor out in_range macro

2019-03-27 Thread Nikolay Borisov
This is used in more than one places so let's factor it out in ctree.h. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/extent-tree.c | 1 - fs/btrfs/volumes.c | 1 - 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/

[PATCH v4 05/15] btrfs: Populate ->orig_block_len during read_one_chunk

2019-03-27 Thread Nikolay Borisov
ng lists. Reviewed-by: David Sterba Signed-off-by: Nikolay Borisov --- fs/btrfs/volumes.c | 41 ++--- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 256f7c5476bc..d2d37adbc6fd 100644 --- a/fs/b

[PATCH v4 06/15] btrfs: Introduce new bits for device allocation tree

2019-03-27 Thread Nikolay Borisov
d-off-by: Nikolay Borisov --- fs/btrfs/extent_io.h | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h index 7732b6a7384d..0fda249e5982 100644 --- a/fs/btrfs/extent_io.h +++ b/fs/btrfs/extent_io.h @@ -27,6 +

[PATCH v4 01/15] btrfs: Honour FITRIM range constraints during free space trim

2019-03-27 Thread Nikolay Borisov
assed range as well as completing trim after trimming fstrim_range::len bytes. Fixes: 499f377f49f0 ("btrfs: iterate over unused chunk space in FITRIM") Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 25 +++-- 1 file changed, 19 insertions(+), 6 deletions

Re: [PATCH URGENT v1.1 1/2] btrfs-progs: disk-io: Make super block write error easier to read

2019-03-27 Thread Nikolay Borisov
nt > error message for the same write error. > > This patch will change it to something more detailed: > ERROR: failed to write super block for devid 1: write error: I/O error > > This provides the basis for later super block flush error handling. > > Signed-off-by

Re: [PATCH URGENT 2/2] btrfs-progs: disk-io: Flush to ensure super block write is FUA

2019-03-27 Thread Nikolay Borisov
action but > dm-log-write only caught one super block write. > > This means all the 3 writes were merged into the last super block write. > And the super block write was the 2nd write, before all tree block > writes, completely screwing up the metadata CoW protection. > > No

Re: [PATCH URGENT 1/2] btrfs-progs: disk-io: Make super block write error more easy to read

2019-03-27 Thread Nikolay Borisov
On 27.03.19 г. 9:24 ч., Qu Wenruo wrote: > When we failed to write super blocks, we just output something like: > WARNING: failed to write sb: I/O error > Or > WARNING: failed to write all sb data > > There is no info about which device failed and there are two different > error message for

Re: WARNING at fs/btrfs/delayed-ref.c:296 btrfs_merge_delayed_refs+0x3dc/0x410 (new on 5.0.4, not in 5.0.3)

2019-03-26 Thread Nikolay Borisov
On 26.03.19 г. 17:09 ч., Zygo Blaxell wrote: > On Tue, Mar 26, 2019 at 10:42:31AM +0200, Nikolay Borisov wrote: >> >> >> On 26.03.19 г. 6:30 ч., Zygo Blaxell wrote: >>> On Mon, Mar 25, 2019 at 10:50:28PM -0400, Zygo Blaxell wrote: >>>> Running balance

Re: [PATCH] btrfs: Enable btrfs/003

2019-03-26 Thread Nikolay Borisov
On 19.03.19 г. 12:58 ч., Nikolay Borisov wrote: > For a long time this test has been failing on all kinds of VM configuration, > which are using virtio_blk devices. This is due to the fact that scsi > devices are deletable and virtio_blk are not. However, this only prevents > de

Re: [PATCH v2] fstests: Verify that removed device has its superblocks deleted

2019-03-26 Thread Nikolay Borisov
On 25.03.19 г. 23:55 ч., Anand Jain wrote: > > > On 3/25/19 10:07 PM, Nikolay Borisov wrote: >> When a device is removed from a btrfs filesystem its superblock copies >> must be deleted. > > AFAIK this bug was fixed a long time back in the kernel. Is there any >

Re: [PATCH] Btrfs: do not allow trimming when a fs is mounted with the nologreplay option

2019-03-26 Thread Nikolay Borisov
On 26.03.19 г. 14:35 ч., Filipe Manana wrote: > On Tue, Mar 26, 2019 at 12:17 PM Nikolay Borisov wrote: >> >> >> >> On 26.03.19 г. 12:49 ч., fdman...@kernel.org wrote: >>> From: Filipe Manana >>> >>> Whan a filesystem is mounted with the

Re: [PATCH] Btrfs: do not allow trimming when a fs is mounted with the nologreplay option

2019-03-26 Thread Nikolay Borisov
sense to make the check a bit more specific and only return EROFS when NOLOGREPLAY and the log tree has non-null generation? In any case: Reviewed-by: Nikolay Borisov > --- > fs/btrfs/ioctl.c | 10 ++ > 1 file changed, 10 insertions(+) > > diff --git a/fs/btrfs/ioctl.c

Re: WARNING at fs/btrfs/delayed-ref.c:296 btrfs_merge_delayed_refs+0x3dc/0x410 (new on 5.0.4, not in 5.0.3)

2019-03-26 Thread Nikolay Borisov
On 26.03.19 г. 6:30 ч., Zygo Blaxell wrote: > On Mon, Mar 25, 2019 at 10:50:28PM -0400, Zygo Blaxell wrote: >> Running balance, rsync, and dedupe, I get kernel warnings every few >> minutes on 5.0.4. No warnings on 5.0.3 under similar conditions. >> >> Mount options are: flushoncommit,space_ca

Re: [PATCH v3 00/12] FITRIM improvements

2019-03-26 Thread Nikolay Borisov
[CC'ing Filipe as he should now better ] On 25.03.19 г. 20:44 ч., Darrick J. Wong wrote: > On Mon, Mar 25, 2019 at 02:31:20PM +0200, Nikolay Borisov wrote: >> Here is v3 of the fitrim patches. Change since v2 [0]: >> >> * Replaced BUG_ON with WARN_ON in patch 2 >&

Re: [PATCH v3 07/12] btrfs: replace pending/pinned chunks lists with io tree

2019-03-25 Thread Nikolay Borisov
On 25.03.19 г. 18:26 ч., David Sterba wrote: > On Mon, Mar 25, 2019 at 02:31:27PM +0200, Nikolay Borisov wrote: >> From: Jeff Mahoney >> >> The pending chunks list contains chunks that are allocated in the >> current transaction but haven't been created yet. Th

Re: [PATCH v3 06/12] btrfs: Introduce new bits for device allocation tree

2019-03-25 Thread Nikolay Borisov
On 25.03.19 г. 18:12 ч., David Sterba wrote: > Have you squashed this change to the wrong commit? In v2 there was a > separate patch but mentioning the following one ("replace pending/pinned > chunks lists with io tree"). I'm going to drop this hunk and use the > separate patch as it makes it mo

Re: [PATCH 1/7] btrfs-progs: check/lowmem: Add inode mode check

2019-03-25 Thread Nikolay Borisov
he inodes. > > Reported-by: Thorsten Hirsch > Signed-off-by: Qu Wenruo Reviewed-by: Nikolay Borisov > --- > check/mode-common.h | 24 > check/mode-lowmem.c | 5 + > check/mode-lowmem.h | 1 + > 3 files changed, 30 insertions(+) > > d

Re: [PATCH 2/7] btrfs-progs: check/original: Add inode mode check

2019-03-25 Thread Nikolay Borisov
On 25.03.19 г. 10:22 ч., Qu Wenruo wrote: > Just like lowmem mode, check inode mode, specially for S_IFMT bits and > beyond. > > Please note that, this check only applies to inodes in fs/subvol trees. > It doesn't apply to free space cache inodes. > > Reported-by: Thorsten Hirsch > Signed-off

[PATCH] fstests: Add missing call to _scratch_dev_pool_put

2019-03-25 Thread Nikolay Borisov
Every call to _scratch_dev_pool_get must be paired with call to _scratch_dev_pool_put otherwise the SCRATCH_POOL variable will have less devices than it actually must. Signed-off-by: Nikolay Borisov --- tests/btrfs/148 | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/btrfs/148 b

Re: [PATCH] btrfs: move tree block wait and write helpers to tree-log

2019-03-25 Thread Nikolay Borisov
On 21.03.19 г. 21:21 ч., David Sterba wrote: > The wrapper names better describe what's happening so they're not > deleted though they're trivial, but at least moved closer to their place > of use. > > Signed-off-by: David Sterba Reviewed-by: Nikolay Borisov

Re: btrfs-cleaner crashes freeing delayed node (5.0.3)

2019-03-25 Thread Nikolay Borisov
On 23.03.19 г. 1:30 ч., Jan Alexander Steffens wrote: > Hi all, > > we have a build server here that accumulated lots of uncleaned subvolumes. > Snapshots containing build roots are frequently created and destroyed. > There's also a borg backup script which creates and destroys a > snapshot of

[PATCH v2] fstests: Verify that removed device has its superblocks deleted

2019-03-25 Thread Nikolay Borisov
When a device is removed from a btrfs filesystem its superblock copies must be deleted. This test ensures this is indeed the case. Signed-off-by: Nikolay Borisov --- Changes since v1: * Use _scratch_dev_pool_(get|put) to ensure the test uses exactly 2 devices. * Explicitly use -draid0

Re: [PATCH v3 02/12] btrfs: combine device update operations during transaction commit

2019-03-25 Thread Nikolay Borisov
On 25.03.19 г. 14:31 ч., Nikolay Borisov wrote: > We currently overload the pending_chunks list to handle updating > btrfs_device->commit_bytes used. We don't actually care about > the extent mapping or even the device mapping for the chunk - we > just need the devic

[PATCH v3 07/12] btrfs: replace pending/pinned chunks lists with io tree

2019-03-25 Thread Nikolay Borisov
ng and pinned chunks list and allows us to do trims on unallocated space safely without pinning the transaction for what may be a lengthy operation. We can also use this io tree to mark which chunks have already been trimmed so we don't repeat the operation. Signed-off-by: Nikolay Boris

[PATCH v3 11/12] btrfs: Implement find_first_clear_extent_bit

2019-03-25 Thread Nikolay Borisov
This function is very similar to find_first_extent_bit except that it locates the first contiguous span of space which does not have bits set. It's intended use is in the freespace trimming code. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent_io.c

[PATCH v3 08/12] btrfs: Remove 'trans' argument from find_free_dev_extent(_start)

2019-03-25 Thread Nikolay Borisov
Now that those function no longer require a handle to transaction to inspect pending/pinned chunks the argument can be removed. At the same time also remove any surrounding code which acquired the handle. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 36

[PATCH v3 04/12] btrfs: Rename and export clear_btree_io_tree

2019-03-25 Thread Nikolay Borisov
This function is going to be used to clear out the device extent allocation information. Give it a more generic name and export it. This is in preparation to replacing the pending/pinned chunk lists with an extent tree. No functional changes. Reviewed-by: David Sterba Signed-off-by: Nikolay

[PATCH v3 09/12] btrfs: Factor out in_range macro

2019-03-25 Thread Nikolay Borisov
This is used in more than one places so let's factor it out in ctree.h. No functional changes. Signed-off-by: Nikolay Borisov --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/extent-tree.c | 1 - fs/btrfs/volumes.c | 1 - 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/

[PATCH v3 12/12] btrfs: Switch btrfs_trim_free_extents to find_first_clear_extent_bit

2019-03-25 Thread Nikolay Borisov
code always searches in-memory data structure to figure out the space range which contains the requested which should result in speed oups. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 89 -- 1 file changed, 26 insertions(+), 63 deletions(-)

[PATCH v3 10/12] btrfs: Optimize unallocated chunks discard

2019-03-25 Thread Nikolay Borisov
asn't been changed since the last time discard was issued. Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 57 +- fs/btrfs/extent_io.h | 8 +- fs/btrfs/extent_map.c | 4 ++- 3 files changed, 66 insertions(+), 3 deletions(-) diff

[PATCH v3 02/12] btrfs: combine device update operations during transaction commit

2019-03-25 Thread Nikolay Borisov
ays add the device to a list when we change bytes_used or disk_total_size, there's no harm in copying both values at once. Signed-off-by: Nikolay Borisov --- fs/btrfs/dev-replace.c | 2 +- fs/btrfs/disk-io.c | 7 fs/btrfs/transaction.c | 5 ++- fs/btrfs/transaction.h | 1 + fs

[PATCH v3 01/12] btrfs: Honour FITRIM range constraints during free space trim

2019-03-25 Thread Nikolay Borisov
assed range as well as completing trim after trimming fstrim_range::len bytes. Fixes: 499f377f49f0 ("btrfs: iterate over unused chunk space in FITRIM") Signed-off-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 25 +++-- 1 file changed, 19 insertions(+), 6 deletions

[PATCH v3 03/12] btrfs: Handle pending/pinned chunks before blockgroup relocation during device shrink

2019-03-25 Thread Nikolay Borisov
ad to making another go in the body of the function. Since there is no hard requirement to perform pinned/pending chunks handling after the relocation loop, move the code before it. This leads to simplifying the code flow around - i.e no need to use 'goto again'. Signed-off-by: Nikolay B

<    2   3   4   5   6   7   8   9   10   11   >