Re: [PATCH] btrfs-progs: add a --check-bg-usage option to fsck

2019-08-02 Thread Josef Bacik
On Fri, Aug 02, 2019 at 09:54:12PM +0800, Qu Wenruo wrote: > > > On 2019/8/2 下午9:06, Josef Bacik wrote: > > Sometimes when messing with the chunk allocator code we can end up > > over-allocating chunks. Generally speaking I'll notice this when a > > random x

[PATCH] btrfs-progs: add a --check-bg-usage option to fsck

2019-08-02 Thread Josef Bacik
7;t trip up currently, so this will just keep me from regressing us. Thanks, Signed-off-by: Josef Bacik --- btrfsck.h| 1 + check/main.c | 52 ++-- 2 files changed, 51 insertions(+), 2 deletions(-) diff --git a/btrfsck.h b/btrfsck.h index ac7f5d4

[PATCH 1/5] btrfs: add a flush step for delayed iputs

2019-08-01 Thread Josef Bacik
Delayed iputs could very well free up enough space without needing to commit the transaction, so make this step it's own step. This will allow us to skip the step for evictions in a later patch. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- fs/btrfs/space-i

[PATCH 3/5] btrfs: factor out the ticket flush handling

2019-08-01 Thread Josef Bacik
We're going to make this logic a little more complicated for evict, so factor the ticket flushing/waiting code out of __reserve_metadata_bytes. This has no functional change. Signed-off-by: Josef Bacik --- fs/btrfs/space-info.c | 64 --- 1 file ch

[PATCH 4/5] btrfs: refactor priority_reclaim_metadata_space

2019-08-01 Thread Josef Bacik
ction flushing. Signed-off-by: Josef Bacik --- fs/btrfs/space-info.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/space-info.c b/fs/btrfs/space-info.c index 71749b355136..03556e411b11 100644 --- a/fs/btrfs/space-info.c +++ b/fs/btrfs/space-i

[PATCH 2/5] btrfs: unify error handling for ticket flushing

2019-08-01 Thread Josef Bacik
Currently we handle the cleanup of errored out tickets in both the priority flush path and the normal flushing path. This is the same code in both places, so just refactor so we don't duplicate the cleanup work. Signed-off-by: Josef Bacik --- fs/btrfs/space-info.c

[PATCH 5/5] btrfs: introduce an evict flushing state

2019-08-01 Thread Josef Bacik
s first. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 1 + fs/btrfs/inode.c | 81 +++ fs/btrfs/space-info.c | 27 +-- 3 files changed, 62 insertions(+), 47 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.

[PATCH 0/5] Rework eviction space flushing

2019-08-01 Thread Josef Bacik
This is a set of patches to address how we do space flushing for inode evictions. Historically we've only been allowed to do a few things to reclaim space for inode evictions, mostly because we'd deadlock with iput. But we have delayed iputs in place to make sure we're always doing iput where it'

Re: [PATCH] Btrfs: fix race leading to fs corruption after transaction abortion

2019-07-29 Thread Josef Bacik
17 @@ int btrfs_commit_transaction(struct btrfs_trans_handle > *trans) > } > } else { > spin_unlock(&fs_info->trans_lock); > + /* > + * The previous transaction was aborted and was already removed > + * from the list of transactions at fs_info->trans_list. So we > + * abort to prevent writing a new superblock that reflects a > + * corrupt state (pointing to trees with unwritten nodes/leafs). > + */ > + if (test_bit(BTRFS_FS_STATE_TRANS_ABORTED, > + &fs_info->fs_state)) { > + ret = -EROFS; > + goto cleanup_transaction; > + } > } > > extwriter_counter_dec(cur_trans, trans->type); Reviewed-by: Josef Bacik Thanks, Josef

Re: Need help with a lockdep splat, possibly perf related?

2019-07-05 Thread Josef Bacik
On Wed, Jul 03, 2019 at 11:12:10PM +0200, Peter Zijlstra wrote: > On Wed, Jul 03, 2019 at 09:54:06AM -0400, Josef Bacik wrote: > > Hello, > > > > I've been seeing a variation of the following splat recently and I have no > > earthly idea what it's trying

Need help with a lockdep splat, possibly perf related?

2019-07-03 Thread Josef Bacik
Hello, I've been seeing a variation of the following splat recently and I have no earthly idea what it's trying to tell me. I either get this one, or I get one that tells me the same thing except it's complaining about &cpuctx_mutex instead of sb_pagefaults. There is no place we take the reloc_m

Re: [PATCH 04/11] btrfs: move the space_info handling code to space-info.c

2019-06-25 Thread Josef Bacik
On Tue, Jun 25, 2019 at 01:58:43PM +0200, David Sterba wrote: > On Tue, Jun 18, 2019 at 04:09:19PM -0400, Josef Bacik wrote: > > --- /dev/null > > +++ b/fs/btrfs/space-info.c > > @@ -0,0 +1,177 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +/* > > + * C

[PATCH 14/25] btrfs: export get_alloc_profile

2019-06-20 Thread Josef Bacik
This gets used directly by a bunch of the block group code, export it to make it easier to move things around. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 1 + fs/btrfs/extent-tree.c | 16 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/fs/btrfs

[PATCH 12/25] btrfs: temporarily export inc_block_group_ro

2019-06-20 Thread Josef Bacik
This is used in a few logical parts of the block group code, temporarily export it so we can move things in pieces. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 2 ++ fs/btrfs/extent-tree.c | 15 --- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/fs

[PATCH 17/25] btrfs: migrate inc/dec_block_group_ro code

2019-06-20 Thread Josef Bacik
This can easily be moved now. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 196 + fs/btrfs/block-group.h | 2 + fs/btrfs/ctree.h | 2 - fs/btrfs/extent-tree.c | 196 - 4 files

[PATCH 09/25] btrfs: temporarily export fragment_free_space

2019-06-20 Thread Josef Bacik
This is used in caching and reading block groups, so export it while we move these chunks independently. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 1 + fs/btrfs/extent-tree.c | 8 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/block-group.h b/fs

[PATCH 23/25] btrfs: migrate the alloc_profile helpers

2019-06-20 Thread Josef Bacik
These feel more at home in block-group.c. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 100 ++ fs/btrfs/block-group.h | 16 +++ fs/btrfs/ctree.h | 4 -- fs/btrfs/extent-tree.c | 115

[PATCH 01/25] btrfs: move btrfs_add_free_space out of a header file

2019-06-20 Thread Josef Bacik
This is prep work for moving block_group_cache around. Having this in the header file makes the header file include need to be in a certain order, which is awkward, so just move it into free-space-cache.c and then we can re-arrange later. Signed-off-by: Josef Bacik --- fs/btrfs/free-space

[PATCH 00/25] btrfs: migrate the block group code

2019-06-20 Thread Josef Bacik
This is the series to migrate the block group code out of extent-tree.c. This is a much larger series than the previous two series because things were much more intertwined than block_rsv's and space_info. There is one code change patch in this series, it is btrfs: make caching_thread use btrfs_

[PATCH 19/25] btrfs: export block group accounting helpers

2019-06-20 Thread Josef Bacik
Want to move these functions into block-group.c, so export them. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 6 ++ fs/btrfs/extent-tree.c | 21 ++--- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h

[PATCH 24/25] btrfs: migrate the block group cleanup code

2019-06-20 Thread Josef Bacik
This can now be easily migrated as well. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 140 + fs/btrfs/block-group.h | 2 + fs/btrfs/ctree.h | 2 - fs/btrfs/extent-tree.c | 140

[PATCH 25/25] btrfs: unexport the temporary exported functions

2019-06-20 Thread Josef Bacik
These were renamed and exported to facilitate logical migration of different code chunks into block-group.c. Now that all the users are in one file go ahead and rename them back, move the code around, and make them static. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 140

[PATCH 15/25] btrfs: migrate the block group read/creation code

2019-06-20 Thread Josef Bacik
All of the prep work has been done so we can now cleanly move this chunk over. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 613 + fs/btrfs/block-group.h | 5 + fs/btrfs/ctree.h | 5 - fs/btrfs/extent-tree.c | 611

[PATCH 02/25] btrfs: move basic block_group definitions to their own header

2019-06-20 Thread Josef Bacik
This is prep work for moving all of the block group cache code into its own file. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 156 + fs/btrfs/ctree.h | 148 +-- fs/btrfs/disk-io.c

[PATCH 21/25] btrfs: migrate btrfs_can_relocate

2019-06-20 Thread Josef Bacik
This belongs to block-group.c, move it over. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 141 + fs/btrfs/block-group.h | 1 + fs/btrfs/ctree.h | 1 - fs/btrfs/extent-tree.c | 141

[PATCH 13/25] btrfs: migrate the block group removal code

2019-06-20 Thread Josef Bacik
This is the removal code and the unused bgs code. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 509 + fs/btrfs/block-group.h | 7 + fs/btrfs/ctree.h | 7 - fs/btrfs/extent-tree.c | 505

[PATCH 06/25] btrfs: export the block group caching helpers

2019-06-20 Thread Josef Bacik
This will make it so we can move them easily. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 14 +++ fs/btrfs/extent-tree.c | 65 ++ 2 files changed, 43 insertions(+), 36 deletions(-) diff --git a/fs/btrfs/block-group.h b/fs

[PATCH 11/25] btrfs: migrate the block group caching code

2019-06-20 Thread Josef Bacik
We can now just copy it over to block-group.c. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 463 + fs/btrfs/block-group.h | 2 + fs/btrfs/ctree.h | 2 - fs/btrfs/extent-tree.c | 461

[PATCH 22/25] btrfs: migrate the chunk allocation code

2019-06-20 Thread Josef Bacik
This feels more at home in block-group.c than in extent-tree.c. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c| 247 ++ fs/btrfs/block-group.h| 24 + fs/btrfs/ctree.h | 24 - fs/btrfs/delalloc-space.c | 1 + fs

[PATCH 20/25] btrfs: migrate the block group space accounting helpers

2019-06-20 Thread Josef Bacik
We can now easily migrate this code as well. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 174 + fs/btrfs/extent-tree.c | 174 - 2 files changed, 174 insertions(+), 174 deletions(-) diff

[PATCH 16/25] btrfs: temporarily export btrfs_get_restripe_target

2019-06-20 Thread Josef Bacik
This gets used by a few different logical chunks of the block group code, export it while we move things around. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 2 ++ fs/btrfs/extent-tree.c | 8 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/block

[PATCH 18/25] btrfs: migrate the dirty bg writeout code

2019-06-20 Thread Josef Bacik
This can be easily migrated over now. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 520 + fs/btrfs/block-group.h | 3 + fs/btrfs/ctree.h | 3 - fs/btrfs/extent-tree.c | 518 4

[PATCH 10/25] btrfs: make caching_thread use btrfs_find_next_key

2019-06-20 Thread Josef Bacik
o fix btrfs_find_next_key to deal with ->skip_locking and then it works exactly the same as the private find_next_key helper. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.c | 4 ++-- fs/btrfs/extent-tree.c | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/c

[PATCH 07/25] btrfs: export the excluded extents helpers

2019-06-20 Thread Josef Bacik
We'll need this to move the caching stuff around. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 +++ fs/btrfs/extent-tree.c | 40 +--- 2 files changed, 24 insertions(+), 19 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h

[PATCH 04/25] btrfs: migrate the block group ref counting stuff

2019-06-20 Thread Josef Bacik
Another easy set to move over to block-group.c. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 24 fs/btrfs/block-group.h | 2 ++ fs/btrfs/ctree.h | 3 --- fs/btrfs/extent-tree.c | 25 - 4 files changed, 26 insertions(+), 28

[PATCH 03/25] btrfs: migrate the block group lookup code

2019-06-20 Thread Josef Bacik
Move these bits first as they are the easiest to move. Export two of the helpers so they can be moved all at once. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile | 2 +- fs/btrfs/block-group.c | 99 ++ fs/btrfs/block-group.h | 7 fs

[PATCH 05/25] btrfs: migrate nocow and reservation helpers

2019-06-20 Thread Josef Bacik
These are relatively straightforward as well. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.c | 83 ++ fs/btrfs/block-group.h | 6 fs/btrfs/ctree.h | 6 fs/btrfs/extent-tree.c | 82

[PATCH 08/25] btrfs: export the caching control helpers

2019-06-20 Thread Josef Bacik
Man a lot of people use this stuff. Signed-off-by: Josef Bacik --- fs/btrfs/block-group.h | 3 +++ fs/btrfs/extent-tree.c | 36 ++-- 2 files changed, 21 insertions(+), 18 deletions(-) diff --git a/fs/btrfs/block-group.h b/fs/btrfs/block-group.h index

Re: [PATCH 5/8] btrfs: stop using block_rsv_release_bytes everywhere

2019-06-20 Thread Josef Bacik
On Thu, Jun 20, 2019 at 11:32:21AM +0300, Nikolay Borisov wrote: > > > On 19.06.19 г. 20:47 ч., Josef Bacik wrote: > > block_rsv_release_bytes() is the internal to the block_rsv code, and > > shouldn't be called directly by anything else. Switch all users t

[PATCH 3/4] btrfs: migrate the delalloc space stuff to it's own home

2019-06-19 Thread Josef Bacik
pace. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile | 2 +- fs/btrfs/ctree.h| 14 -- fs/btrfs/delalloc-space.c | 497 fs/btrfs/delalloc-space.h | 25 +++ fs/btrfs/extent-tree.c | 486 +

[PATCH 0/4] btrfs: move random block rsv code around

2019-06-19 Thread Josef Bacik
We have some specialized block rsvs, like the inode rsv and the delayed refs rsv, that have their own special oddities. These don't fit in extent-tree.c, so migrate them to different places. Most of them go to existing files, but the delalloc stuff gets its own file because it's a lot of code, an

[PATCH 4/4] btrfs: move the subvolume reservation stuff out of extent-tree.c

2019-06-19 Thread Josef Bacik
This is just two functions, put it in root-tree.c since it involves root items. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 54 fs/btrfs/root-tree.c | 56 ++ 2 files changed, 56

[PATCH 2/4] btrfs: migrate btrfs_trans_release_chunk_metadata

2019-06-19 Thread Josef Bacik
Move this into transaction.c with the rest of the transaction related code. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 1 - fs/btrfs/extent-tree.c | 18 -- fs/btrfs/transaction.c | 18 ++ fs/btrfs/transaction.h | 1 + 4 files changed, 19 insertions

[PATCH 1/4] btrfs: migrate the delayed refs rsv code

2019-06-19 Thread Josef Bacik
These belong with the delayed refs related code, not in extent-tree.c. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 9 --- fs/btrfs/delayed-ref.c | 177 + fs/btrfs/delayed-ref.h | 10 +++ fs/btrfs/extent-tree.c | 175

[PATCH 6/8] btrfs: migrate the block-rsv code to block-rsv.c

2019-06-19 Thread Josef Bacik
This moves everything out of extent-tree.c to block-rsv.c. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile | 3 +- fs/btrfs/block-rsv.c | 257 + fs/btrfs/extent-tree.c | 249 --- 3 files

[PATCH 5/8] btrfs: stop using block_rsv_release_bytes everywhere

2019-06-19 Thread Josef Bacik
block_rsv_release_bytes() is the internal to the block_rsv code, and shouldn't be called directly by anything else. Switch all users to the exported helpers. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 14 ++ 1 file changed, 6 insertions(+), 8 deletions(-) diff --

[PATCH 8/8] btrfs: export and migrate use_block_rsv/unuse_block_rsv

2019-06-19 Thread Josef Bacik
Move these into block_rsv.c/h respectively, and export them for use by alloc_tree_block. Signed-off-by: Josef Bacik --- fs/btrfs/block-rsv.c | 82 fs/btrfs/block-rsv.h | 13 ++- fs/btrfs/extent-tree.c | 92

[PATCH 4/8] btrfs: cleanup the target logic in __btrfs_block_rsv_release

2019-06-19 Thread Josef Bacik
This works for all callers already, but if we wanted to use the helper for the global_block_rsv it would end up trying to refill itself. Fix the logic to be able to be used no matter which block rsv is passed in to this helper. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 14

[PATCH 7/8] btrfs: migrate the global_block_rsv helpers to block-rsv.c

2019-06-19 Thread Josef Bacik
These helpers belong in block-rsv.c Signed-off-by: Josef Bacik --- fs/btrfs/block-rsv.c | 90 + fs/btrfs/block-rsv.h | 3 ++ fs/btrfs/extent-tree.c | 99 +++--- 3 files changed, 98 insertions(+), 94

[PATCH 3/8] btrfs: export __btrfs_block_rsv_release

2019-06-19 Thread Josef Bacik
The delalloc reserve stuff calls this directly because it cares about the qgroup accounting stuff, so export it so we can move it around. Fix btrfs_block_rsv_release() to just be a static inline since it just calls __btrfs_block_rsv_release() with NULL for the qgroup stuff. Signed-off-by: Josef

[PATCH 0/8] btrfs: move the block_rsv code out of extent-tree.c

2019-06-19 Thread Josef Bacik
This patchset depends on the space_info migration patchset. This is the next logical chunk of things to move out of extent-tree.c With these sets of patches we're down below 10k loc in extent-tree.c. This chunk was much easier to move as we had exported a lot of these functions already. There i

[PATCH 1/8] btrfs: move btrfs_block_rsv definitions into it's own header

2019-06-19 Thread Josef Bacik
Prep work for separating out all of the block_rsv related code into its own file. Signed-off-by: Josef Bacik --- fs/btrfs/block-rsv.h | 81 ++ fs/btrfs/ctree.h | 70 +-- fs/btrfs/extent-tree.c | 1

[PATCH 2/8] btrfs: export btrfs_block_rsv_add_bytes

2019-06-19 Thread Josef Bacik
This is used in a few places, we need to make sure it's exported so we can move it around. Signed-off-by: Josef Bacik --- fs/btrfs/block-rsv.h | 2 ++ fs/btrfs/extent-tree.c | 18 +- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/block-rsv.h

[PATCH 08/11] btrfs: export block_rsv_use_bytes

2019-06-18 Thread Josef Bacik
We are going to need this to move the metadata reservation stuff to space_info.c. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/extent-tree.c | 14 ++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index

[PATCH 10/11] btrfs: move reserve_metadata_bytes and supporting code to space-info.c

2019-06-18 Thread Josef Bacik
This moves all of the metadata reservation code into space-info.c. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 710 + fs/btrfs/space-info.c | 698 fs/btrfs/space-info.h | 4 + 3

[PATCH 09/11] btrfs: move dump_space_info to space-info.c

2019-06-18 Thread Josef Bacik
We'll need this exported so we can use it in all the various was we need to use it. This is prep work to move reserve_metadata_bytes. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 70 +- fs/btrfs/space-info.c

[PATCH 11/11] btrfs: unexport can_overcommit

2019-06-18 Thread Josef Bacik
Now that we've moved all of the users to space-info.c, unexport it and name it back to can_overcommit. Signed-off-by: Josef Bacik --- fs/btrfs/space-info.c | 23 +++ fs/btrfs/space-info.h | 4 2 files changed, 11 insertions(+), 16 deletions(-) diff --git a/fs/

[PATCH 07/11] btrfs: move btrfs_space_info_add_*_bytes to space-info.c

2019-06-18 Thread Josef Bacik
Now that we've moved all the pre-requisite stuff, move these two functions. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 114 - fs/btrfs/space-info.c | 106 + fs/btrfs/space-info.h

[PATCH 05/11] btrfs: move and export can_overcommit

2019-06-18 Thread Josef Bacik
This is the first piece of moving the space reservation code to space-info.c Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 83 +- fs/btrfs/space-info.c | 68 + fs/btrfs/space-info.h | 4 +++ 3

[PATCH 04/11] btrfs: move the space_info handling code to space-info.c

2019-06-18 Thread Josef Bacik
These are the basic init and lookup functions and some helper functions, fairly straightforward before the bad stuff starts. Signed-off-by: Josef Bacik --- fs/btrfs/Makefile | 2 +- fs/btrfs/extent-tree.c | 205 + fs/btrfs/space-info.c

[PATCH 01/11] btrfs: move space_info to space_info.h

2019-06-18 Thread Josef Bacik
Migrate the struct definition and the one helper that's in ctree.h into space_info.h Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h| 73 +--- fs/btrfs/extent-tree.c | 1 + fs/btrfs/free-space-cache.c | 1 + fs/btrfs/ioctl.c

[PATCH 00/11] btrfs: move the space_info code out of extent-tree.c

2019-06-18 Thread Josef Bacik
This is the first pass at making extent-tree.c much smaller. I've purposefully done no other cleanups or changes. The places where I needed to modify callers were done in separate patches. The only time I moved and changed callers in large chunks was the moving of reserve_metadata_bytes out of e

[PATCH 02/11] btrfs: rename do_chunk_alloc to btrfs_chunk_alloc

2019-06-18 Thread Josef Bacik
Really we just need the enum, but as we break more things up it'll help to have this external to extent-tree.c. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 22 ++ fs/btrfs/extent-tree.c | 50 ++ 2 files change

[PATCH 06/11] btrfs: move the space info update macro to space-info.h

2019-06-18 Thread Josef Bacik
Also rename it to btrfs_space_info_update_* so it's clear what we're updating. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 68 +++--- fs/btrfs/space-info.h | 22 2 files changed, 53 insertions(+), 37 deletion

[PATCH 03/11] btrfs: export space_info_add_*_bytes

2019-06-18 Thread Josef Bacik
Prep work for consolidating all of the space_info code into one file. We need to export these so multiple files can use them. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 42 -- fs/btrfs/space-info.h | 6 ++ 2 files changed, 26 insertions

[PATCH][v2] btrfs: run delayed iput at unlink time

2019-06-18 Thread Josef Bacik
o the caller and get throttled appropriately without affecting the main workload. Signed-off-by: Josef Bacik --- v1->v2: - consolidate the delayed iput run into a helper. fs/btrfs/inode.c | 40 ++-- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/

Re: [PATCH 07/19] btrfs: do sequential extent allocation in HMZONED mode

2019-06-18 Thread Josef Bacik
On Tue, Jun 18, 2019 at 08:28:07AM +, Naohiro Aota wrote: > On 2019/06/13 23:07, Josef Bacik wrote: > > On Fri, Jun 07, 2019 at 10:10:13PM +0900, Naohiro Aota wrote: > >> @@ -9616,7 +9701,8 @@ static int inc_block_group_ro(struct > >> btrfs_block_gr

Re: [PATCH 08/19] btrfs: make unmirroed BGs readonly only if we have at least one writable BG

2019-06-18 Thread Josef Bacik
On Tue, Jun 18, 2019 at 07:42:46AM +, Naohiro Aota wrote: > On 2019/06/13 23:09, Josef Bacik wrote: > > On Fri, Jun 07, 2019 at 10:10:14PM +0900, Naohiro Aota wrote: > >> If the btrfs volume has mirrored block groups, it unconditionally makes > >> un-mirrored bloc

Re: [PATCH 11/19] btrfs: introduce submit buffer

2019-06-18 Thread Josef Bacik
On Mon, Jun 17, 2019 at 03:16:05AM +, Damien Le Moal wrote: > Josef, > > On 2019/06/13 23:15, Josef Bacik wrote: > > On Fri, Jun 07, 2019 at 10:10:17PM +0900, Naohiro Aota wrote: > >> Sequential allocation is not enough to maintain sequential delivery of > >>

Re: [PATCH 03/19] btrfs: Check and enable HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:09PM +0900, Naohiro Aota wrote: > HMZONED mode cannot be used together with the RAID5/6 profile for now. > Introduce the function btrfs_check_hmzoned_mode() to check this. This > function will also check if HMZONED flag is enabled on the file system and > if the file s

Re: [PATCH 02/19] btrfs: Get zone information of zoned block devices

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:08PM +0900, Naohiro Aota wrote: > If a zoned block device is found, get its zone information (number of zones > and zone size) using the new helper function btrfs_get_dev_zonetypes(). To > avoid costly run-time zone report commands to test the device zones type > duri

Re: [PATCH 02/19] btrfs: Get zone information of zoned block devices

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:08PM +0900, Naohiro Aota wrote: > If a zoned block device is found, get its zone information (number of zones > and zone size) using the new helper function btrfs_get_dev_zonetypes(). To > avoid costly run-time zone report commands to test the device zones type > duri

Re: [PATCH 05/19] btrfs: disable direct IO in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:11PM +0900, Naohiro Aota wrote: > Direct write I/Os can be directed at existing extents that have already > been written. Such write requests are prohibited on host-managed zoned > block devices. So disable direct IO support for a volume with HMZONED mode > enabled. >

Re: [PATCH 07/19] btrfs: do sequential extent allocation in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:13PM +0900, Naohiro Aota wrote: > On HMZONED drives, writes must always be sequential and directed at a block > group zone write pointer position. Thus, block allocation in a block group > must also be done sequentially using an allocation pointer equal to the > block

Re: [PATCH 08/19] btrfs: make unmirroed BGs readonly only if we have at least one writable BG

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:14PM +0900, Naohiro Aota wrote: > If the btrfs volume has mirrored block groups, it unconditionally makes > un-mirrored block groups read only. When we have mirrored block groups, but > don't have writable block groups, this will drop all writable block groups. > So, c

Re: [PATCH 11/19] btrfs: introduce submit buffer

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:17PM +0900, Naohiro Aota wrote: > Sequential allocation is not enough to maintain sequential delivery of > write IOs to the device. Various features (async compress, async checksum, > ...) of btrfs affect ordering of the IOs. This patch introduces submit > buffer to so

Re: [PATCH 09/19] btrfs: limit super block locations in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:15PM +0900, Naohiro Aota wrote: > When in HMZONED mode, make sure that device super blocks are located in > randomly writable zones of zoned block devices. That is, do not write super > blocks in sequential write required zones of host-managed zoned block > devices as

Re: [PATCH 13/19] btrfs: avoid sync IO prioritization on checksum in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:19PM +0900, Naohiro Aota wrote: > Btrfs prioritize sync I/Os to be handled by async checksum worker earlier. > As a result, checksumming sync I/Os to larger logical extent address can > finish faster than checksumming non-sync I/Os to smaller logical extent > address.

Re: [PATCH 12/19] btrfs: expire submit buffer on timeout

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:18PM +0900, Naohiro Aota wrote: > It is possible to have bios stalled in the submit buffer due to some bug or > device problem. In such situation, btrfs stops working waiting for buffered > bios completions. To avoid such hang, add a worker that will cancel the > stall

Re: [PATCH 14/19] btrfs: redirty released extent buffers in sequential BGs

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:20PM +0900, Naohiro Aota wrote: > Tree manipulating operations like merging nodes often release > once-allocated tree nodes. Btrfs cleans such nodes so that pages in the > node are not uselessly written out. On HMZONED drives, however, such > optimization blocks the fo

Re: [PATCH 16/19] btrfs: wait existing extents before truncating

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:22PM +0900, Naohiro Aota wrote: > When truncating a file, file buffers which have already been allocated but > not yet written may be truncated. Truncating these buffers could cause > breakage of a sequential write pattern in a block group if the truncated > blocks ar

Re: [PATCH 18/19] btrfs: support dev-replace in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:24PM +0900, Naohiro Aota wrote: > Currently, dev-replace copy all the device extents on source device to the > target device, and it also clones new incoming write I/Os from users to the > source device into the target device. > > Cloning incoming IOs can break the se

Re: [PATCH 17/19] btrfs: shrink delayed allocation size in HMZONED mode

2019-06-13 Thread Josef Bacik
On Fri, Jun 07, 2019 at 10:10:23PM +0900, Naohiro Aota wrote: > In a write heavy workload, the following scenario can occur: > > 1. mark page #0 to page #2 (and their corresponding extent region) as dirty >and candidate for delayed allocation > > pages0 1 2 3 4 > dirtyo o o - - > towr

Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle

2019-06-04 Thread Josef Bacik
On Tue, Jun 04, 2019 at 08:31:23AM +0800, Qu Wenruo wrote: > > > On 2019/6/4 上午1:36, Josef Bacik wrote: > > On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote: > >> > >> > >> On 2019/2/13 上午12:03, David Sterba wrote: > >>> On Th

Re: [PATCH] btrfs: don't end the transaction for delayed refs in throttle

2019-06-03 Thread Josef Bacik
On Mon, Jun 03, 2019 at 02:53:00PM +0800, Qu Wenruo wrote: > > > On 2019/2/13 上午12:03, David Sterba wrote: > > On Thu, Jan 24, 2019 at 09:31:43AM -0500, Josef Bacik wrote: > >> Previously callers to btrfs_end_transaction_throttle() would commit the > >> transacti

Re: [PATCH 1/2] Btrfs: fix race between ranged fsync and writeback of adjacent ranges

2019-05-07 Thread Josef Bacik
On Tue, May 07, 2019 at 08:09:02PM +0100, Filipe Manana wrote: > On Tue, May 7, 2019 at 6:44 PM Josef Bacik wrote: > > > > On Mon, May 06, 2019 at 04:44:02PM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > When we do a full fsy

Re: [PATCH] Btrfs: avoid fallback to transaction commit during fsync of files with holes

2019-05-07 Thread Josef Bacik
nserted a file extent item > representing a hole for this offset (68K) in the previous call to > copy_items(), when processing leaf N. > > The -EEXIST error gets propagated to the fsync callback, btrfs_sync_file(), > which falls back to a full transaction commit. > > Fix this by adjusting *last_extent after inserting a hole when we had to > look at the next leaf. > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 2/2] Btrfs: teach tree-checker to detect file extent items with overlapping ranges

2019-05-07 Thread Josef Bacik
_extents() when it traims file extent > items). Therefore teach the tree checker to detect such cases. This is > motivated by a recently fixed bug (race between ranged full fsync and > writeback or adjacent ranges). > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 1/2] Btrfs: fix race between ranged fsync and writeback of adjacent ranges

2019-05-07 Thread Josef Bacik
nt item to represent that hole, having > a key offset of 68K. After that we now have a log tree with 2 different > extent items that have overlapping ranges: > Well this kind of sucks. I wonder if we should be locking the extent range while we're doing this in order to keep this problem from happening. A fix for another day though Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH RFC] btrfs: reflink: Flush before reflink any extent to prevent NOCOW write falling back to CoW without data reservation

2019-05-07 Thread Josef Bacik
On Fri, May 03, 2019 at 09:08:52AM +0800, Qu Wenruo wrote: > [BUG] > The following command can lead to unexpected data COW: > > #!/bin/bash > > dev=/dev/test/test > mnt=/mnt/btrfs > > mkfs.btrfs -f $dev -b 1G > /dev/null > mount $dev $mnt -o nospace_cache > > xfs_io -f -c "falloc 8k

Re: [PATCH v2 0/3] Ordered extent flushing refactor

2019-05-07 Thread Josef Bacik
; Nikolay Borisov (3): > btrfs: Implement btrfs_lock_and_flush_ordered_range > btrfs: Use newly introduced btrfs_lock_and_flush_ordered_range > btrfs: Always use a cached extent_state in > btrfs_lock_and_flush_ordered_range > You can add Reviewed-by: Josef Bacik for the whole series. Thanks, Josef

[PATCH] btrfs: run delayed iput at unlink time

2019-05-07 Thread Josef Bacik
o the caller and get throttled appropriately without affecting the main workload. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index b6d549c993f6..e58685b5d398 100644 --- a/fs/btrfs/in

[PATCH] btrfs: don't double unlock on error in btrfs_punch_hole

2019-05-03 Thread Josef Bacik
If we have an error writing out a delalloc range in btrfs_punch_hole_lock_range we'll unlock the inode and then goto out_only_mutex, where we will again unlock the inode. This is bad, don't do this. Signed-off-by: Josef Bacik --- fs/btrfs/file.c | 4 +--- 1 file changed, 1 inser

[PATCH][v2] btrfs: track odirect bytes in flight

2019-04-25 Thread Josef Bacik
writes than delalloc bytes we'll go ahead and wait on any ordered extents irregardless of our flush state as flushing delalloc is likely to not gain us anything. Signed-off-by: Josef Bacik --- v1->v2: - updated the changelog fs/btrfs/ctree.h| 1 + fs/btrfs/disk-io.c

Re: fallocate does not prevent ENOSPC on write

2019-04-25 Thread Josef Bacik
On Thu, Apr 25, 2019 at 10:11:41PM +0800, Qu Wenruo wrote: > > > On 2019/4/25 下午10:09, Josef Bacik wrote: > > On Thu, Apr 25, 2019 at 09:50:25PM +0800, Qu Wenruo wrote: > >> > >> > >> On 2019/4/25 下午9:25, Josef Bacik wrote: > >> [snip] >

Re: fallocate does not prevent ENOSPC on write

2019-04-25 Thread Josef Bacik
On Thu, Apr 25, 2019 at 09:50:25PM +0800, Qu Wenruo wrote: > > > On 2019/4/25 下午9:25, Josef Bacik wrote: > [snip] > >>> > >>> What if the commit is reverted, if the problem is otherwise hard to fix? > >>> This seems to break the semantics of falloc

Re: fallocate does not prevent ENOSPC on write

2019-04-25 Thread Josef Bacik
On Thu, Apr 25, 2019 at 01:49:23PM +0800, Qu Wenruo wrote: > > > On 2019/4/23 下午7:33, David Sterba wrote: > > On Tue, Apr 23, 2019 at 10:16:32AM +0800, Qu Wenruo wrote: > >> On 2019/4/23 上午5:09, Jakob Unterwurzacher wrote: > >>> I have a user who is reporting ENOSPC errors when running gocryptfs

Re: [PATCH] btrfs: extent_io: Handle memory allocation failure in __clear_extent_bit()

2019-04-18 Thread Josef Bacik
On Thu, Apr 18, 2019 at 10:24:10AM +0300, Nikolay Borisov wrote: > > > On 18.04.19 г. 10:21 ч., Qu Wenruo wrote: > > There is a BUG_ON() in __clear_extent_bit() for memory allocation > > failure. > > > > While comment of __clear_extent_bit() says it can return error, but we > > always return 0.

Re: [PATCH 2/2] btrfs: reserve delalloc metadata differently

2019-04-12 Thread Josef Bacik
On Fri, Apr 12, 2019 at 04:35:20PM +0300, Nikolay Borisov wrote: > > > On 12.04.19 г. 16:26 ч., Josef Bacik wrote: > > On Fri, Apr 12, 2019 at 04:06:25PM +0300, Nikolay Borisov wrote: > >> > >> > >> On 10.04.19 г. 22:56 ч., Josef Bacik wrote: >

Re: [PATCH 1/2] btrfs: track odirect bytes in flight

2019-04-12 Thread Josef Bacik
On Fri, Apr 12, 2019 at 01:17:40PM +0300, Nikolay Borisov wrote: > > > On 10.04.19 г. 22:56 ч., Josef Bacik wrote: > > When diagnosing a slowdown of generic/224 I noticed we were wasting a > > lot of time in shrink_delalloc() despite all writes being O_DIRECT > > wri

<    3   4   5   6   7   8   9   10   11   12   >