[PATCH][v2] btrfs: run delayed items before dropping the snapshot

2018-12-05 Thread Josef Bacik
From: Josef Bacik With my delayed refs patches in place we started seeing a large amount of aborts in __btrfs_free_extent BTRFS error (device sdb1): unable to find ref byte nr 91947008 parent 0 root 35964 owner 1 offset 0 Call Trace: ? btrfs_merge_delayed_refs+0xaf/0x340

Re: [RFC PATCH] btrfs: Remove __extent_readpages

2018-12-05 Thread Josef Bacik
On Mon, Dec 03, 2018 at 12:25:32PM +0200, Nikolay Borisov wrote: > When extent_readpages is called from the generic readahead code it first > builds a batch of 16 pages (which might or might not be consecutive, > depending on whether add_to_page_cache_lru failed) and submits them to >

Re: [PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-12-04 Thread Josef Bacik
On Tue, Dec 04, 2018 at 01:46:58PM +0200, Nikolay Borisov wrote: > > > On 3.12.18 г. 18:06 ч., Josef Bacik wrote: > > The throttle path doesn't take cleaner_delayed_iput_mutex, which means > > we could think we're done flushing iputs in the data space reservation > &

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-12-04 Thread Josef Bacik
On Tue, Dec 04, 2018 at 11:21:14AM +0200, Nikolay Borisov wrote: > > > On 3.12.18 г. 18:06 ч., Josef Bacik wrote: > > The cleaner thread usually takes care of delayed iputs, with the > > exception of the btrfs_end_transaction_throttle path. The cleaner > > thread

[PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-12-03 Thread Josef Bacik
Manana Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 +++ fs/btrfs/disk-io.c | 3 +++ fs/btrfs/inode.c | 2 ++ 3 files changed, 8 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index c8ddbacb6748..dc56a4d940c3 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@ -769,6

[PATCH 1/3] btrfs: run delayed iputs before committing

2018-12-03 Thread Josef Bacik
. If there is and we freed enough we can then commit the transaction and potentially be able to make our reservation. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 9 + 1 file changed, 9 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[PATCH 0/3][V2] Delayed iput fixes

2018-12-03 Thread Josef Bacik
v1->v2: - only wakeup if the cleaner isn't currently doing work. - re-arranged some stuff for running delayed iputs during flushint. - removed the open code wakeup in the waitqueue patch. -- Original message -- Here are some delayed iput fixes. Delayed iputs can hold reservations for a while

[PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-12-03 Thread Josef Bacik
the cleaner_delayed_iput_mutex whenever we flush the delayed iputs just replace it with an atomic counter and a waitqueue. This removes the short (or long depending on how big the inode is) window where we think there are no more pending iputs when there really are some. Signed-off-by: Josef Bacik --- fs/btrfs

[PATCH 8/8] btrfs: reserve extra space during evict()

2018-12-03 Thread Josef Bacik
We could generate a lot of delayed refs in evict but never have any left over space from our block rsv to make up for that fact. So reserve some extra space and give it to the transaction so it can be used to refill the delayed refs rsv every loop through the truncate path. Signed-off-by: Josef

[PATCH 1/8] btrfs: check if free bgs for commit

2018-12-03 Thread Josef Bacik
reservation. So instead of just returning ENOSPC, check if we have free block groups pending, and if so commit the transaction to allow us to use that free space. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: Nikolay Borisov --- fs/btrfs/extent-tree.c | 34

[PATCH 4/8] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-12-03 Thread Josef Bacik
tests in xfstests with my previous patch. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- fs/btrfs/extent-tree.c | 18 +- include/trace/events/btrfs.h | 1 + 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/fs

[PATCH 0/8][V2] Enospc cleanups and fixeS

2018-12-03 Thread Josef Bacik
v1->v2: - addressed comments from reviewers. - fixed a bug in patch 6 that was introduced because of changes to upstream. -- Original message -- The delayed refs rsv patches exposed a bunch of issues in our enospc infrastructure that needed to be addressed. These aren't really one coherent

[PATCH 2/8] btrfs: dump block_rsv whe dumping space info

2018-12-03 Thread Josef Bacik
For enospc_debug having the block rsvs is super helpful to see if we've done something wrong. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: David Sterba --- fs/btrfs/extent-tree.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/fs/btrfs/extent-tree.c

[PATCH 6/8] btrfs: loop in inode_rsv_refill

2018-12-03 Thread Josef Bacik
it re-calculate our new reservation size and try again. If our reservation size doesn't change between tries then we know we are actually out of space and can error out. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 58 +- 1 file changed

[PATCH 3/8] btrfs: don't use global rsv for chunk allocation

2018-12-03 Thread Josef Bacik
no longer require assuming the global reserve is used space in our calculations. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 9 - 1 file changed, 9 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 204b35434056..667b992d322d 100644 --- a/fs/bt

[PATCH 7/8] btrfs: be more explicit about allowed flush states

2018-12-03 Thread Josef Bacik
and to allocate chunks, everything else has the potential to deadlock. Future proof this by explicitly specifying the states that FLUSH_LIMIT is allowed to use. This will keep us from introducing bugs later on when adding new flush states. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 21

[PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-12-03 Thread Josef Bacik
of returning what reservation they did receive in hopes that it could satisfy reservations down the line. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 45 + 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs

[PATCH 08/10] btrfs: rework btrfs_check_space_for_delayed_refs

2018-12-03 Thread Josef Bacik
refs rsv. If our total size is beyond that amount then we know it's time to commit the transaction and stop any more delayed refs from being generated. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 2 +- fs/btrfs/extent-tree.c | 48 ++-- fs

[PATCH 10/10] btrfs: fix truncate throttling

2018-12-03 Thread Josef Bacik
We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/inode.c

[PATCH 09/10] btrfs: don't run delayed refs in the end transaction logic

2018-12-03 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 38 -- 1 file changed, 38 deletions(-) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 2d8401bf8df9..01f39401619a 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -798,22 +798,12

[PATCH 02/10] btrfs: add cleanup_ref_head_accounting helper

2018-12-03 Thread Josef Bacik
From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we

[PATCH 00/10][V2] Delayed refs rsv

2018-12-03 Thread Josef Bacik
v1->v2: - addressed the comments from the various reviewers. - split "introduce delayed_refs_rsv" into 5 patches. The patches are the same together as they were, just split out more logically. They can't really be bisected across in that you will likely have fun enospc failures, but they

[PATCH 06/10] btrfs: update may_commit_transaction to use the delayed refs rsv

2018-12-03 Thread Josef Bacik
have enough bytes to satisfy our reservation ticket then we are good to go, otherwise subtract out what space we would gain back by committing the transaction and compare that against the pinned space to make our decision. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 24

[PATCH 05/10] btrfs: introduce delayed_refs_rsv

2018-12-03 Thread Josef Bacik
From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted

[PATCH 04/10] btrfs: only track ref_heads in delayed_ref_updates

2018-12-03 Thread Josef Bacik
From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only

[PATCH 03/10] btrfs: cleanup extent_op handling

2018-12-03 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order

[PATCH 01/10] btrfs: add btrfs_delete_ref_head helper

2018-12-03 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent

[PATCH 07/10] btrfs: add new flushing states for the delayed refs rsv

2018-12-03 Thread Josef Bacik
delayed refs. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 10 ++ fs/btrfs/extent-tree.c | 14 ++ include/trace/events/btrfs.h | 2 ++ 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 52a87d446945

Re: [PATCH 1/2] btrfs: catch cow on deleting snapshots

2018-11-30 Thread Josef Bacik
On Fri, Nov 30, 2018 at 05:14:54PM +, Filipe Manana wrote: > On Fri, Nov 30, 2018 at 4:53 PM Josef Bacik wrote: > > > > From: Josef Bacik > > > > When debugging some weird extent reference bug I suspected that we were > > changing a snapshot while we were de

[PATCH 0/2] Fix aborts when dropping snapshots

2018-11-30 Thread Josef Bacik
With more machines running my recent delayed ref rsv patches we started seeing a spike in boxes aborting in __btrfs_free_extent with being unable to find the extent ref. The full details are in 2/2, but the summary is there's been a bug ever since the original delayed inode item stuff was

[PATCH 1/2] btrfs: catch cow on deleting snapshots

2018-11-30 Thread Josef Bacik
From: Josef Bacik When debugging some weird extent reference bug I suspected that we were changing a snapshot while we were deleting it, which could explain my bug. This was indeed what was happening, and this patch helped me verify my theory. It is never correct to modify the snapshot once

[PATCH 2/2] btrfs: run delayed items before dropping the snapshot

2018-11-30 Thread Josef Bacik
From: Josef Bacik With my delayed refs patches in place we started seeing a large amount of aborts in __btrfs_free_extent BTRFS error (device sdb1): unable to find ref byte nr 91947008 parent 0 root 35964 owner 1 offset 0 Call Trace: ? btrfs_merge_delayed_refs+0xaf/0x340

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 07:59:42PM +, Chris Mason wrote: > On 27 Nov 2018, at 14:54, Josef Bacik wrote: > > > On Tue, Nov 27, 2018 at 10:26:15AM +0200, Nikolay Borisov wrote: > >> > >> > >> On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > >>&g

Re: [PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 10:29:57AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > > The throttle path doesn't take cleaner_delayed_iput_mutex, which means > > Which one is the throttle path? btrfs_end_transaction_throttle is only > cal

Re: [PATCH] btrfs: only run delayed refs if we're committing

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 07:43:39PM +, Filipe Manana wrote: > On Tue, Nov 27, 2018 at 7:22 PM Josef Bacik wrote: > > > > On Fri, Nov 23, 2018 at 04:59:32PM +, Filipe Manana wrote: > > > On Thu, Nov 22, 2018 at 12:35 AM Josef Bacik wrote: > > > > >

Re: [PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-27 Thread Josef Bacik
On Tue, Nov 27, 2018 at 10:26:15AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:09 ч., Josef Bacik wrote: > > The cleaner thread usually takes care of delayed iputs, with the > > exception of the btrfs_end_transaction_throttle path. The cleaner > > thread

Re: [PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-11-27 Thread Josef Bacik
On Mon, Nov 26, 2018 at 02:25:52PM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 21:03 ч., Josef Bacik wrote: > > With the introduction of the per-inode block_rsv it became possible to > > have really really large reservation requests made because of data >

Re: [PATCH] btrfs: only run delayed refs if we're committing

2018-11-27 Thread Josef Bacik
On Fri, Nov 23, 2018 at 04:59:32PM +, Filipe Manana wrote: > On Thu, Nov 22, 2018 at 12:35 AM Josef Bacik wrote: > > > > I noticed in a giant dbench run that we spent a lot of time on lock > > contention while running transaction commit. This is because dbench > >

Re: [PATCH 5/6] btrfs: introduce delayed_refs_rsv

2018-11-27 Thread Josef Bacik
On Mon, Nov 26, 2018 at 11:14:12AM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 20:59 ч., Josef Bacik wrote: > > From: Josef Bacik > > > > Traditionally we've had voodoo in btrfs to account for the space that > > delayed refs may take up by having a g

Re: [PATCH 3/6] btrfs: cleanup extent_op handling

2018-11-27 Thread Josef Bacik
On Thu, Nov 22, 2018 at 12:09:34PM +0200, Nikolay Borisov wrote: > > > On 21.11.18 г. 20:59 ч., Josef Bacik wrote: > > From: Josef Bacik > > > > The cleanup_extent_op function actually would run the extent_op if it > > needed running, which made the

Re: [PATCH] btrfs: Remove unnecessary code from __btrfs_rebalance

2018-11-26 Thread Josef Bacik
quot; > > Clearly, this piece of code has lost its original intent throughout > the years. It doesn't really bring any real practical benefits to the > relocation process. No functional changes. > > Signed-off-by: Nikolay Borisov > Suggested-by: Josef Bacik Reviewed-by: Josef Bacik Thanks, Josef

[PATCH 1/3] btrfs: run delayed iputs before committing

2018-11-21 Thread Josef Bacik
. If there is and we freed enough we can then commit the transaction and potentially be able to make our reservation. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 9 + 1 file changed, 9 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[PATCH 2/3] btrfs: wakeup cleaner thread when adding delayed iput

2018-11-21 Thread Josef Bacik
Manana Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 3da9ac463344..3c42d8887183 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -3264,6 +3264,7 @@ void btrfs_add_delayed_iput(struct inode *inode

[PATCH] btrfs: only run delayed refs if we're committing

2018-11-21 Thread Josef Bacik
. This leads to seconds of 0 throughput. Change this to only run the delayed refs if we're the ones committing the transaction. This makes the latency go away and we get no more lock contention. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/transaction.c | 24

[PATCH 0/3] Delayed iput fixes

2018-11-21 Thread Josef Bacik
Here are some delayed iput fixes. Delayed iputs can hold reservations for a while and there's no real good way to make sure they were gone for good, which means we could early enospc when in reality if we had just waited for the iput we would have had plenty of space. So fix this up by making us

[PATCH 3/3] btrfs: replace cleaner_delayed_iput_mutex with a waitqueue

2018-11-21 Thread Josef Bacik
the cleaner_delayed_iput_mutex whenever we flush the delayed iputs just replace it with an atomic counter and a waitqueue. This removes the short (or long depending on how big the inode is) window where we think there are no more pending iputs when there really are some. Signed-off-by: Josef Bacik --- fs/btrfs

[PATCH 4/7] btrfs: call btrfs_create_pending_block_groups unconditionally

2018-11-21 Thread Josef Bacik
The first thing we do is loop through the list, this if (!list_empty()) btrfs_create_pending_block_groups(); thing is just wasted space. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 3 +-- fs/btrfs/transaction.c | 6 ++ 2 files changed, 3

[PATCH 6/7] btrfs: cleanup pending bgs on transaction abort

2018-11-21 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 826a15a07fce..3c1be9db897c 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -2269,6 +2269,10 @@ int btrfs_commit_transaction

[PATCH 5/7] btrfs: just delete pending bgs if we are aborted

2018-11-21 Thread Josef Bacik
We still need to do all of the accounting cleanup for pending block groups if we abort. So set the ret to trans->aborted so if we aborted the cleanup happens and everybody is happy. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 8 +++- 1 file chan

[PATCH 7/7] btrfs: wait on ordered extents on abort cleanup

2018-11-21 Thread Josef Bacik
transaction cleanup stuff. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 8 1 file changed, 8 insertions(+) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 8e7926c91e35..c5918ff8241b 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c

[PATCH 0/7] Abort cleanup fixes

2018-11-21 Thread Josef Bacik
A new xfstests that really hammers on transaction aborts (generic/495 I think?) uncovered a lot of random issues. Some of these were introduced with the new delayed refs rsv patches, others were just exposed by them, such as the pending bg stuff. With these patches in place I stopped getting all

[PATCH 3/7] btrfs: handle delayed ref head accounting cleanup in abort

2018-11-21 Thread Josef Bacik
We weren't doing any of the accounting cleanup when we aborted transactions. Fix this by making cleanup_ref_head_accounting global and calling it from the abort code, this fixes the issue where our accounting was all wrong after the fs aborts. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h

[PATCH 2/7] btrfs: make btrfs_destroy_delayed_refs use btrfs_delete_ref_head

2018-11-21 Thread Josef Bacik
Instead of open coding this stuff use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 7 +-- 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index f062fb0487cd..7d02748cf3f6 100644

[PATCH 1/7] btrfs: make btrfs_destroy_delayed_refs use btrfs_delayed_ref_lock

2018-11-21 Thread Josef Bacik
We have this open coded in btrfs_destroy_delayed_refs, use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 11 ++- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index

[PATCH 3/8] btrfs: don't use global rsv for chunk allocation

2018-11-21 Thread Josef Bacik
readily. Instead use the actual used amount when determining if we need to allocate a chunk or not. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 9 - 1 file changed, 9 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 7a30fbc05e5e..a91b3183dcae

[PATCH 4/8] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-11-21 Thread Josef Bacik
tests in xfstests with my previous patch. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- fs/btrfs/extent-tree.c | 18 +- include/trace/events/btrfs.h | 1 + 3 files changed, 20 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs

[PATCH 5/8] btrfs: don't enospc all tickets on flush failure

2018-11-21 Thread Josef Bacik
of returning what reservation they did receive in hopes that it could satisfy reservations down the line. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 45 + 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs

[PATCH 7/8] btrfs: be more explicit about allowed flush states

2018-11-21 Thread Josef Bacik
for flushing with FLUSH_LIMIT and use that for our state machine. Then as we add new things that are safe we can just add them to this list. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 21 ++--- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/fs/btrfs

[PATCH 0/8] Enospc cleanups and fixes

2018-11-21 Thread Josef Bacik
The delayed refs rsv patches exposed a bunch of issues in our enospc infrastructure that needed to be addressed. These aren't really one coherent group, but they are all around flushing and reservations. may_commit_transaction() needed to be updated a little bit, and we needed to add a new state

[PATCH 6/8] btrfs: loop in inode_rsv_refill

2018-11-21 Thread Josef Bacik
it re-calculate our new reservation size and try again. If our reservation size doesn't change between tries then we know we are actually out of space and can error out. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 56 -- 1 file changed

[PATCH 2/8] btrfs: dump block_rsv whe dumping space info

2018-11-21 Thread Josef Bacik
For enospc_debug having the block rsvs is super helpful to see if we've done something wrong. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval Reviewed-by: David Sterba --- fs/btrfs/extent-tree.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/fs/btrfs/extent-tree.c

[PATCH 8/8] btrfs: reserve extra space during evict()

2018-11-21 Thread Josef Bacik
We could generate a lot of delayed refs in evict but never have any left over space from our block rsv to make up for that fact. So reserve some extra space and give it to the transaction so it can be used to refill the delayed refs rsv every loop through the truncate path. Signed-off-by: Josef

[PATCH 1/8] btrfs: check if free bgs for commit

2018-11-21 Thread Josef Bacik
reservation. So instead of just returning ENOSPC, check if we have free block groups pending, and if so commit the transaction to allow us to use that free space. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/extent-tree.c | 33 +++-- 1 file changed, 19

[PATCH 6/6] btrfs: fix truncate throttling

2018-11-21 Thread Josef Bacik
We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 79

[PATCH 5/6] btrfs: introduce delayed_refs_rsv

2018-11-21 Thread Josef Bacik
From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted

[PATCH 0/6] Delayed refs rsv

2018-11-21 Thread Josef Bacik
This patchset changes how we do space reservations for delayed refs. We were hitting probably 20-40 enospc abort's per day in production while running delayed refs at transaction commit time. This means we ran out of space in the global reserve and couldn't easily get more space in

[PATCH 4/6] btrfs: only track ref_heads in delayed_ref_updates

2018-11-21 Thread Josef Bacik
From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only

[PATCH 2/6] btrfs: add cleanup_ref_head_accounting helper

2018-11-21 Thread Josef Bacik
From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we

[PATCH 3/6] btrfs: cleanup extent_op handling

2018-11-21 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order

[PATCH 1/6] btrfs: add btrfs_delete_ref_head helper

2018-11-21 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik Reviewed-by: Omar Sandoval --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent

Re: [PATCH v2] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-11-20 Thread Josef Bacik
> extent hang") Can we just remove the endio cleanup in __extent_writepage() and make this do the proper cleanup? I'm not sure if that is feasible or not, but seems like it would make the cleanup stuff less confusing and more straightforward. If not you can add Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: remove no longer used io_err from btrfs_log_ctx

2018-11-12 Thread Josef Bacik
e inode. We did this in > commit b5e6c3e170b7 ("btrfs: always wait on ordered extents at fsync > time") and commit a2120a473a80 ("btrfs: clean up the left over > logged_list usage") removed its last use. > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix rare chances for data loss when doing a fast fsync

2018-11-12 Thread Josef Bacik
; Fixes: b5e6c3e170b7 ("btrfs: always wait on ordered extents at fsync time") > CC: sta...@vger.kernel.org # 4.19+ > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [Regression bisected] btrfs: always wait on ordered extents at fsync time

2018-11-12 Thread Josef Bacik
On Fri, Nov 09, 2018 at 10:56:42PM +, Mel Gorman wrote: > On Fri, Nov 09, 2018 at 09:51:48PM +, Mel Gorman wrote: > > Unfortunately, as > > I'm about to travel, I didn't attempt a revert and a test comparing 4.18, > > 4.19 and 4.20-rc1 is a few hours away so this could potentially be fixed

Re: [PATCH v15.1 04/13] btrfs: dedupe: Introduce function to remove hash from in-memory tree

2018-11-09 Thread Josef Bacik
On Tue, Nov 06, 2018 at 02:41:13PM +0800, Lu Fengqi wrote: > From: Wang Xiaoguang > > Introduce static function inmem_del() to remove hash from in-memory > dedupe tree. > And implement btrfs_dedupe_del() and btrfs_dedup_disable() interfaces. > > Also for btrfs_dedupe_disable(), add new

Re: [PATCH v15.1 01/13] btrfs: dedupe: Introduce dedupe framework and its header

2018-11-09 Thread Josef Bacik
On Tue, Nov 06, 2018 at 02:41:10PM +0800, Lu Fengqi wrote: > From: Wang Xiaoguang > > Introduce the header for btrfs in-band(write time) de-duplication > framework and needed header. > > The new de-duplication framework is going to support 2 different dedupe > methods and 1 dedupe hash. > >

Re: [PATCH] btrfs: Check for missing device before bio submission in btrfs_map_bio

2018-11-09 Thread Josef Bacik
count. No functional changes. > > Signed-off-by: Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: simpler and more efficient cleanup of a log tree's extent io tree

2018-11-09 Thread Josef Bacik
gt; Signed-off-by: Filipe Manana Sheesh Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix data corruption due to cloning of eof block

2018-11-07 Thread Josef Bacik
07d19dc9fbe9 ("vfs: avoid problematic remapping requests into > partial EOF block"). So this change is more geared towards stable kernels, > as it's unlikely the new VFS checks get removed intentionally. > > A test case for fstests follows soon, as well as an update to filter > existing tests that expect -EOPNOTSUPP to accept -EINVAL as well. > > CC: # 4.4+ > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 8/8] btrfs: Remove extent_io_ops::split_extent_hook callback

2018-11-01 Thread Josef Bacik
Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 7/8] btrfs: Remove extent_io_ops::merge_extent_hook callback

2018-11-01 Thread Josef Bacik
ve it a more descriptive name. No functional changes. > > Signed-off-by: Nikolay Borisov > --- Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 6/8] btrfs: Remove extent_io_ops::clear_bit_hook callback

2018-11-01 Thread Josef Bacik
isov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 5/8] btrfs: Remove extent_io_ops::set_bit_hook extent_io callback

2018-11-01 Thread Josef Bacik
ops weren't set > for the inode. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 4/8] btrfs: Remove extent_io_ops::check_extent_io_range callback

2018-11-01 Thread Josef Bacik
gt;private_data set i.e. relate to a data node and > not the btree one. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 3/8] btrfs: Remove extent_io_ops::writepage_end_io_hook

2018-11-01 Thread Josef Bacik
patch to remove the struct extent_state *state from the arg list as well. Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 2/8] btrfs: Remove extent_io_ops::writepage_start_hook

2018-11-01 Thread Josef Bacik
gt; callback definition, exports the callback function and calls it > directly at the only call site. Also give the function a more descriptive > name. No functional changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH 1/8] btrfs: Remove extent_io_ops::fill_delalloc

2018-11-01 Thread Josef Bacik
ted > via the extent_io_ops structure. This patch removes the callback > definition, exports the function and calls it directly. No functional > changes. > > Signed-off-by: Nikolay Borisov Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] fstests: fix fssum to actually ignore file holes when supposed to

2018-10-31 Thread Josef Bacik
port holes and fallocate, we can just change > the test and pass the '-s' option to all fssum calls. > > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

Re: [PATCH] Btrfs: fix missing data checksums after a ranged fsync (msync)

2018-10-31 Thread Josef Bacik
t ranged fsync may collect them if needed. > Also, if we find a hole extent outside of the range still log it, just > to prevent having gaps between extent items after replaying the log, > otherwise fsck will complain when we are not using the NO_HOLES feature > (fstest btrfs/056 triggers such case). > > Fixes: e7175a692765 ("btrfs: remove the wait ordered logic in the > log_one_extent path") > CC: sta...@vger.kernel.org # 4.19+ > Signed-off-by: Filipe Manana Nice catch, Reviewed-by: Josef Bacik Josef

Re: [PATCH 0/3] fix pinned underflow in generic/475

2018-10-25 Thread Josef Bacik
rflow earlier > You can add Reviewed-by: Josef Bacik to the series, thanks, Josef

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
On Wed, Oct 24, 2018 at 01:48:40PM +0100, Filipe Manana wrote: > On Wed, Oct 24, 2018 at 1:40 PM Josef Bacik wrote: > > > > On Wed, Oct 24, 2018 at 12:53:59PM +0100, Filipe Manana wrote: > > > On Wed, Oct 24, 2018 at 12:37 PM Josef Bacik wrote: > > > > > &

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
On Wed, Oct 24, 2018 at 12:53:59PM +0100, Filipe Manana wrote: > On Wed, Oct 24, 2018 at 12:37 PM Josef Bacik wrote: > > > > On Wed, Oct 24, 2018 at 10:13:03AM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > When we

Re: [PATCH v4] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-24 Thread Josef Bacik
On Wed, Oct 24, 2018 at 10:13:03AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > When we are writing out a free space cache, during the transaction commit > phase, we can end up in a deadlock which results in a stack trace like the > following: > > schedule+0x28/0x80 >

Re: [PATCH v3] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-23 Thread Josef Bacik
On Mon, Oct 22, 2018 at 11:05:08PM +0100, Filipe Manana wrote: > On Mon, Oct 22, 2018 at 8:18 PM Josef Bacik wrote: > > > > On Mon, Oct 22, 2018 at 08:10:37PM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > When we

Re: [PATCH v3] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
ached block > groups have no free extents. > > Reported-by: Andrew Nelson > Link: > https://lore.kernel.org/linux-btrfs/captelenq9x5kowuq+fa7h1r3nsjg8vyith8+ifjurc_duhh...@mail.gmail.com/ > Fixes: 9d66e233c704 ("Btrfs: load free space cache if it exists") > Tested-by: Andrew Nelson > Signed-off-by: Filipe Manana Great, thanks, Reviewed-by: Josef Bacik Josef

Re: [PATCH v2] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
On Mon, Oct 22, 2018 at 07:48:30PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > When we are writing out a free space cache, during the transaction commit > phase, we can end up in a deadlock which results in a stack trace like the > following: > > schedule+0x28/0x80 >

Re: [PATCH] Btrfs: fix deadlock on tree root leaf when finding free extent

2018-10-22 Thread Josef Bacik
On Mon, Oct 22, 2018 at 10:09:46AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > When we are writing out a free space cache, during the transaction commit > phase, we can end up in a deadlock which results in a stack trace like the > following: > > schedule+0x28/0x80 >

Re: [PATCH] Btrfs: fix use-after-free when dumping free space

2018-10-22 Thread Josef Bacik
R0: 80050033 > [ 9520.416848] CR2: 0011 CR3: 000106b52000 CR4: > 06a0 > [ 9520.418477] DR0: DR1: DR2: > > [ 9520.418846] DR3: DR6: fffe0ff0 DR7: > 0400 > [ 9520.419204] Kernel panic - not syncing: Fatal exception > [ 9520.419666] Dumping ftrace buffer: > [ 9520.419930](ftrace buffer empty) > [ 9520.420168] Kernel Offset: disabled > [ 9520.420406] ---[ end Kernel panic - not syncing: Fatal exception ]--- > > Fix this by acquiring the respective lock before iterating the rbtree. > > Reported-by: Nikolay Borisov > Signed-off-by: Filipe Manana Reviewed-by: Josef Bacik Thanks, Josef

[PATCH 33/42] btrfs: fix insert_reserved error handling

2018-10-12 Thread Josef Bacik
are sure that both cases are properly cleaned up in case of a transaction abort. Reviewed-by: David Sterba Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 12 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/fs/btrfs/extent-tree.c b

[PATCH 31/42] btrfs: cleanup pending bgs on transaction abort

2018-10-12 Thread Josef Bacik
-by: Josef Bacik --- fs/btrfs/transaction.c | 4 1 file changed, 4 insertions(+) diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 46ca775a709e..9168efaca37e 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -2280,6 +2280,10 @@ int btrfs_commit_transaction

  1   2   3   4   5   6   7   8   9   10   >