Re: List of known BTRFS Raid 5/6 Bugs?

2018-09-11 Thread Duncan
Stefan K posted on Tue, 11 Sep 2018 13:29:38 +0200 as excerpted: > wow, holy shit, thanks for this extended answer! > >> The first thing to point out here again is that it's not >> btrfs-specific. > so that mean that every RAID implemantation (with parity) has such Bug? > I'm looking a bit, it

Re: [PATCH 0/3] btrfs-progs: Detect compressed extent without csum

2018-09-11 Thread Qu Wenruo
On 2018/9/11 下午11:58, David Sterba wrote: > On Mon, May 14, 2018 at 02:54:41PM +0800, Qu Wenruo wrote: >> Patches can be fetch from github: >> https://github.com/adam900710/btrfs-progs/tree/compress_nodatasum >> >> It's based on v4.16 stable branch. >> >> James Harvey from mail list reports a

Re: [PATCH 0/6] btrfs-progs: Variant fixes for fuzz-tests

2018-09-11 Thread Qu Wenruo
On 2018/9/12 上午12:38, David Sterba wrote: > On Wed, Aug 29, 2018 at 01:27:39PM +0800, Qu Wenruo wrote: >> Gentle ping. >> >> These fixes are pretty small, I'd like to see them merged before I need >> to rebase them again and again. > > I've merged them now, thanks. I had to edit all changelogs

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-09-11 Thread Qu Wenruo
On 2018/9/11 下午11:52, David Sterba wrote: > On Tue, Sep 04, 2018 at 08:39:55PM +0800, Qu Wenruo wrote: >> >> >> On 2018/8/23 下午3:45, Qu Wenruo wrote: >>> >>> >>> On 2018/8/23 下午3:36, Nikolay Borisov wrote: On 23.08.2018 10:31, Qu Wenruo wrote: > Introduce --breadth-first

Re: [PATCH] btrfs-progs: Continue checking even we found something wrong in free space cache

2018-09-11 Thread Qu Wenruo
On 2018/9/11 下午10:48, David Sterba wrote: > On Tue, Sep 04, 2018 at 08:41:28PM +0800, Qu Wenruo wrote: >> No need to abort checking, especially for RO check free space cache is >> meaningless, the errors in fs/extent tree is more interesting for >> developers. >> >> So continue checking even

Re: [PATCH 03/36] btrfs: cleanup extent_op handling

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 01:57:34PM -0400, Josef Bacik wrote: > From: Josef Bacik > > The cleanup_extent_op function actually would run the extent_op if it > needed running, which made the name sort of a misnomer. Change it to > run_and_cleanup_extent_op, and move the actual cleanup work to >

Re: [PATCH 08/36] btrfs: dump block_rsv whe dumping space info

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 01:57:39PM -0400, Josef Bacik wrote: > For enospc_debug having the block rsvs is super helpful to see if we've > done something wrong. > > Signed-off-by: Josef Bacik > --- > fs/btrfs/extent-tree.c | 16 > 1 file changed, 16 insertions(+) > > diff --git

Re: [PATCH 07/36] btrfs: check if free bgs for commit

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 01:57:38PM -0400, Josef Bacik wrote: > may_commit_transaction will skip committing the transaction if we don't > have enough pinned space or if we're trying to find space for a SYSTEM > chunk. However if we have pending free block groups in this transaction > we still want

Re: [PATCH 05/36] btrfs: only count ref heads run in __btrfs_run_delayed_refs

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 01:57:36PM -0400, Josef Bacik wrote: > We pick the number of ref's to run based on the number of ref heads, and > only make the decision to stop once we've processed entire ref heads, so > only count the ref heads we've run and bail once we've hit the number of > ref heads

Re: [PATCH 01/36] btrfs: add btrfs_delete_ref_head helper

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 01:57:32PM -0400, Josef Bacik wrote: > From: Josef Bacik > > We do this dance in cleanup_ref_head and check_ref_cleanup, unify it > into a helper and cleanup the calling functions. Reviewed-by: Omar Sandoval > Signed-off-by: Josef Bacik > --- > fs/btrfs/delayed-ref.c

[PATCH v7 1/6] mm: split SWP_FILE into SWP_ACTIVATED and SWP_FS

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval The SWP_FILE flag serves two purposes: to make swap_{read,write}page() go through the filesystem, and to make swapoff() call ->swap_deactivate(). For Btrfs, we want the latter but not the former, so split this flag into two. This makes us always call ->swap_deactivate() if

[PATCH v7 0/6] Btrfs: implement swap file support

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval Hi, This series implements swap file support for Btrfs. Changes from v6 [1]: - Moved btrfs_get_chunk_map() comment to function body - Added more comments about pinned block group/device rbtree - Fixed bug in patch 4 which broke resize Based on v4.19-rc3. Thanks! 1:

[PATCH v7 3/6] vfs: update swap_{,de}activate documentation

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval The documentation for these functions is wrong in several ways: - swap_activate() is called with the inode locked - swap_activate() takes a swap_info_struct * and a sector_t * - swap_activate() can also return a positive number of extents it added itself -

[PATCH v7 2/6] mm: export add_swap_extent()

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval Btrfs will need this for swap file support. Signed-off-by: Omar Sandoval --- mm/swapfile.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/swapfile.c b/mm/swapfile.c index d3f95833d12e..51cb30de17bc 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -2365,6 +2365,7

[PATCH v7 4/6] Btrfs: prevent ioctls from interfering with a swap file

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval When a swap file is active, we must make sure that the extents of the file are not moved and that they don't become shared. That means that the following are not safe: - chattr +c (enable compression) - reflink - dedupe - snapshot - defrag Don't allow those to happen on an

[PATCH v7 6/6] Btrfs: support swap files

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval Implement the swap file a_ops on Btrfs. Activation needs to make sure that the file can be used as a swap file, which currently means it must be fully allocated as nocow with no compression on one device. It must also do the proper tracking so that ioctls will not interfere

[PATCH v7 5/6] Btrfs: rename get_chunk_map() and make it non-static

2018-09-11 Thread Omar Sandoval
From: Omar Sandoval The Btrfs swap code is going to need it, so give it a btrfs_ prefix and make it non-static. Reviewed-by: Nikolay Borisov Signed-off-by: Omar Sandoval --- fs/btrfs/volumes.c | 29 ++--- fs/btrfs/volumes.h | 2 ++ 2 files changed, 20 insertions(+),

[PATCH] Btrfs: assert page dirty bit

2018-09-11 Thread Liu Bo
Just in case that someone breaks the rule that pages are dirty as long as eb is dirty. Signed-off-by: Liu Bo --- fs/btrfs/extent_io.c | 5 + 1 file changed, 5 insertions(+) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index fb2bf50134a1..99895f196ecb 100644 ---

[PATCH] Btrfs: remove level==0 check in balance_level

2018-09-11 Thread Liu Bo
btrfs_search_slot() if (level != 0) setup_nodes_for_search() balance_level() It is just impossible to have level=0 in balance_level. Signed-off-by: Liu Bo --- fs/btrfs/ctree.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c index

[PATCH] Btrfs: unify error handling of btrfs_lookup_dir_item

2018-09-11 Thread Liu Bo
Unify the error handling part with IS_ERR_OR_NULL. No functional changes. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 21 + fs/btrfs/send.c | 8 ++-- 2 files changed, 7 insertions(+), 22 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index

[PATCH] Btrfs: remove wait_ordered_range in btrfs_evict_inode

2018-09-11 Thread Liu Bo
As VFS has called ->invalidatepage() to get all ordered extents done and truncated all page cache pages, wait_ordered_range() is just a noop. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index

[PATCH] Btrfs: remove unused code in __btrfs_unlink_inode

2018-09-11 Thread Liu Bo
It might get @leaf and @key in order to do some sanity checks on key's fields, but since we don't check it any more, it's fine to remove the code. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 4 1 file changed, 4 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index

[PATCH] Btrfs: do not wait after queue async work for delaye refs

2018-09-11 Thread Liu Bo
If metadata space is hungry, how fast flush_space() can run determines the latency we spend in reserve_metadata_space(). flush_space() case FLUSH_DELAYED_ITEMS: ... btrfs_end_transaction() case ALLOC_CHUNK: ... btrfs_end_transaction() btrfs_end_transaction()

[PATCH] Btrfs: skip set_page_dirty if eb is dirty

2018-09-11 Thread Liu Bo
As long as @eb is marked with EXTENT_BUFFER_DIRTY, all of its pages are dirty, so no need to set pages dirty again. Signed-off-by: Liu Bo --- fs/btrfs/extent_io.c | 11 +++ fs/btrfs/extent_io.h | 2 +- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/extent_io.c

[PATCH] Btrfs: skip setting path to blocking mode if balance is not needed

2018-09-11 Thread Liu Bo
balance_level() may return early in some cases, but these checks don't have to be done with blocking write lock. This puts together these checks into a helper and the benefit is to avoid switching spinning locks to blocking locks (in these paticular cases) which slows down btrfs overall.

Re: [PATCH 4/4] btrfs: tests: polish ifdefs around testing helper

2018-09-11 Thread Omar Sandoval
On Tue, Sep 11, 2018 at 11:22:51AM +0200, David Sterba wrote: > On Mon, Sep 10, 2018 at 04:43:29PM -0700, Omar Sandoval wrote: > > On Mon, Sep 10, 2018 at 07:22:31PM +0200, David Sterba wrote: > > > Avoid the inline ifdefs and use two sections for self-tests enabled and > > > disabled. > > > > >

Re: [PATCH 0/5] rb_first to rb_first_cached conversion

2018-09-11 Thread Liu Bo
On Tue, Sep 11, 2018 at 05:34:03PM +0200, David Sterba wrote: > On Thu, Aug 23, 2018 at 03:51:48AM +0800, Liu Bo wrote: > > Several structs in btrfs are using rb_first() in a while loop, it'd be > > more efficient to do this with rb_first_cached() which has the O(1) > > complexity. > > We'd like

[PATCH 33/36] btrfs: only free reserved extent if we didn't insert it

2018-09-11 Thread Josef Bacik
When we insert the file extent once the ordered extent completes we free the reserved extent reservation as it'll have been migrated to the bytes_used counter. However if we error out after this step we'll still clear the reserved extent reservation, resulting in a negative accounting of the

[PATCH 34/36] btrfs: fix insert_reserved error handling

2018-09-11 Thread Josef Bacik
We were not handling the reserved byte accounting properly for data references. Metadata was fine, if it errored out the error paths would free the bytes_reserved count and pin the extent, but it even missed one of the error cases. So instead move this handling up into run_one_delayed_ref so we

[PATCH 36/36] MAINTAINERS: update my email address for btrfs

2018-09-11 Thread Josef Bacik
My work email is completely useless, switch it to my personal address so I get emails on a account I actually pay attention to. Signed-off-by: Josef Bacik --- MAINTAINERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index

[PATCH 35/36] btrfs: wait on ordered extents on abort cleanup

2018-09-11 Thread Josef Bacik
If we flip read-only before we initiate writeback on all dirty pages for ordered extents we've created then we'll have ordered extents left over on umount, which results in all sorts of bad things happening. Fix this by making sure we wait on ordered extents if we have to do the aborted

[PATCH 31/36] btrfs: cleanup pending bgs on transaction abort

2018-09-11 Thread Josef Bacik
We may abort the transaction during a commit and not have a chance to run the pending bgs stuff, which will leave block groups on our list and cause us accounting issues and leaked memory. Fix this by running the pending bgs when we cleanup a transaction. Reviewed-by: Omar Sandoval

[PATCH 32/36] btrfs: clear delayed_refs_rsv for dirty bg cleanup

2018-09-11 Thread Josef Bacik
We keep track of dirty bg's as a reservation in the delayed_refs_rsv, so when we abort and we cleanup those dirty bgs we need to drop their reservation so we don't have accounting issues and lots of scary messages on umount. Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 1 + 1 file

[PATCH 19/36] btrfs: set max_extent_size properly

2018-09-11 Thread Josef Bacik
From: Josef Bacik We can't use entry->bytes if our entry is a bitmap entry, we need to use entry->max_extent_size in that case. Fix up all the logic to make this consistent. Signed-off-by: Josef Bacik --- fs/btrfs/free-space-cache.c | 29 +++-- 1 file changed, 19

[PATCH 06/36] btrfs: introduce delayed_refs_rsv

2018-09-11 Thread Josef Bacik
From: Josef Bacik Traditionally we've had voodoo in btrfs to account for the space that delayed refs may take up by having a global_block_rsv. This works most of the time, except when it doesn't. We've had issues reported and seen in production where sometimes the global reserve is exhausted

[PATCH 23/36] btrfs: make sure we create all new bgs

2018-09-11 Thread Josef Bacik
Allocating new chunks modifies both the extent and chunk tree, which can trigger new chunk allocations. So instead of doing list_for_each_safe, just do while (!list_empty()) so we make sure we don't exit with other pending bg's still on our list. Reviewed-by: Omar Sandoval Reviewed-by: Liu Bo

[PATCH 21/36] btrfs: reset max_extent_size on clear in a bitmap

2018-09-11 Thread Josef Bacik
From: Josef Bacik We need to clear the max_extent_size when we clear bits from a bitmap since it could have been from the range that contains the max_extent_size. Reviewed-by: Liu Bo Signed-off-by: Josef Bacik --- fs/btrfs/free-space-cache.c | 2 ++ 1 file changed, 2 insertions(+) diff

[PATCH 18/36] btrfs: move the dio_sem higher up the callchain

2018-09-11 Thread Josef Bacik
We're getting a lockdep splat because we take the dio_sem under the log_mutex. What we really need is to protect fsync() from logging an extent map for an extent we never waited on higher up, so just guard the whole thing with dio_sem. Signed-off-by: Josef Bacik --- fs/btrfs/file.c | 12

[PATCH 28/36] btrfs: handle delayed ref head accounting cleanup in abort

2018-09-11 Thread Josef Bacik
We weren't doing any of the accounting cleanup when we aborted transactions. Fix this by making cleanup_ref_head_accounting global and calling it from the abort code, this fixes the issue where our accounting was all wrong after the fs aborts. Signed-off-by: Josef Bacik --- fs/btrfs/ctree.h

[PATCH 17/36] btrfs: loop in inode_rsv_refill

2018-09-11 Thread Josef Bacik
With severe fragmentation we can end up with our inode rsv size being huge during writeout, which would cause us to need to make very large metadata reservations. However we may not actually need that much once writeout is complete. So instead try to make our reservation, and if we couldn't make

[PATCH 04/36] btrfs: only track ref_heads in delayed_ref_updates

2018-09-11 Thread Josef Bacik
From: Josef Bacik We use this number to figure out how many delayed refs to run, but __btrfs_run_delayed_refs really only checks every time we need a new delayed ref head, so we always run at least one ref head completely no matter what the number of items on it. Fix the accounting to only be

[PATCH 16/36] btrfs: run delayed iputs before committing

2018-09-11 Thread Josef Bacik
Delayed iputs means we can have final iputs of deleted inodes in the queue, which could potentially generate a lot of pinned space that could be free'd. So before we decide to commit the transaction for ENOPSC reasons, run the delayed iputs so that any potential space is free'd up. If there is

[PATCH 27/36] btrfs: make btrfs_destroy_delayed_refs use btrfs_delete_ref_head

2018-09-11 Thread Josef Bacik
Instead of open coding this stuff use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 7 +-- 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index c72ab2ca7627..1d3f5731d616 100644 ---

[PATCH 10/36] btrfs: protect space cache inode alloc with nofs

2018-09-11 Thread Josef Bacik
If we're allocating a new space cache inode it's likely going to be under a transaction handle, so we need to use memalloc_nofs_save() in order to avoid deadlocks, and more importantly lockdep messages that make xfstests fail. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik ---

[PATCH 05/36] btrfs: only count ref heads run in __btrfs_run_delayed_refs

2018-09-11 Thread Josef Bacik
We pick the number of ref's to run based on the number of ref heads, and only make the decision to stop once we've processed entire ref heads, so only count the ref heads we've run and bail once we've hit the number of ref heads we wanted to process. Signed-off-by: Josef Bacik ---

[PATCH 26/36] btrfs: make btrfs_destroy_delayed_refs use btrfs_delayed_ref_lock

2018-09-11 Thread Josef Bacik
We have this open coded in btrfs_destroy_delayed_refs, use the helper instead. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 11 ++- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index

[PATCH 30/36] btrfs: just delete pending bgs if we are aborted

2018-09-11 Thread Josef Bacik
We still need to do all of the accounting cleanup for pending block groups if we abort. So set the ret to trans->aborted so if we aborted the cleanup happens and everybody is happy. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 8 +++- 1 file changed,

[PATCH 15/36] btrfs: don't enospc all tickets on flush failure

2018-09-11 Thread Josef Bacik
With the introduction of the per-inode block_rsv it became possible to have really really large reservation requests made because of data fragmentation. Since the ticket stuff assumed that we'd always have relatively small reservation requests it just killed all tickets if we were unable to

[PATCH 07/36] btrfs: check if free bgs for commit

2018-09-11 Thread Josef Bacik
may_commit_transaction will skip committing the transaction if we don't have enough pinned space or if we're trying to find space for a SYSTEM chunk. However if we have pending free block groups in this transaction we still want to commit as we may be able to allocate a chunk to make our

[PATCH 29/36] btrfs: call btrfs_create_pending_block_groups unconditionally

2018-09-11 Thread Josef Bacik
The first thing we do is loop through the list, this if (!list_empty()) btrfs_create_pending_block_groups(); thing is just wasted space. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 3 +-- fs/btrfs/transaction.c | 6 ++ 2 files changed, 3

[PATCH 13/36] btrfs: add ALLOC_CHUNK_FORCE to the flushing code

2018-09-11 Thread Josef Bacik
With my change to no longer take into account the global reserve for metadata allocation chunks we have this side-effect for mixed block group fs'es where we are no longer allocating enough chunks for the data/metadata requirements. To deal with this add a ALLOC_CHUNK_FORCE step to the flushing

[PATCH 22/36] btrfs: only run delayed refs if we're committing

2018-09-11 Thread Josef Bacik
I noticed in a giant dbench run that we spent a lot of time on lock contention while running transaction commit. This is because dbench results in a lot of fsync()'s that do a btrfs_transaction_commit(), and they all run the delayed refs first thing, so they all contend with each other. This

[PATCH 11/36] btrfs: fix truncate throttling

2018-09-11 Thread Josef Bacik
We have a bunch of magic to make sure we're throttling delayed refs when truncating a file. Now that we have a delayed refs rsv and a mechanism for refilling that reserve simply use that instead of all of this magic. Signed-off-by: Josef Bacik --- fs/btrfs/inode.c | 78

[PATCH 12/36] btrfs: don't use global rsv for chunk allocation

2018-09-11 Thread Josef Bacik
We've done this forever because of the voodoo around knowing how much space we have. However we have better ways of doing this now, and on normal file systems we'll easily have a global reserve of 512MiB, and since metadata chunks are usually 1GiB that means we'll allocate metadata chunks more

[PATCH 14/36] btrfs: reset max_extent_size properly

2018-09-11 Thread Josef Bacik
If we use up our block group before allocating a new one we'll easily get a max_extent_size that's set really really low, which will result in a lot of fragmentation. We need to make sure we're resetting the max_extent_size when we add a new chunk or add new space. Signed-off-by: Josef Bacik

[PATCH 24/36] btrfs: assert on non-empty delayed iputs

2018-09-11 Thread Josef Bacik
I ran into an issue where there was some reference being held on an inode that I couldn't track. This assert wasn't triggered, but it at least rules out we're doing something stupid. Reviewed-by: Omar Sandoval Signed-off-by: Josef Bacik --- fs/btrfs/disk-io.c | 1 + 1 file changed, 1

[PATCH 03/36] btrfs: cleanup extent_op handling

2018-09-11 Thread Josef Bacik
From: Josef Bacik The cleanup_extent_op function actually would run the extent_op if it needed running, which made the name sort of a misnomer. Change it to run_and_cleanup_extent_op, and move the actual cleanup work to cleanup_extent_op so it can be used by check_ref_cleanup() in order to

[PATCH 08/36] btrfs: dump block_rsv whe dumping space info

2018-09-11 Thread Josef Bacik
For enospc_debug having the block rsvs is super helpful to see if we've done something wrong. Signed-off-by: Josef Bacik --- fs/btrfs/extent-tree.c | 16 1 file changed, 16 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index

[PATCH 09/36] btrfs: release metadata before running delayed refs

2018-09-11 Thread Josef Bacik
We want to release the unused reservation we have since it refills the delayed refs reserve, which will make everything go smoother when running the delayed refs if we're short on our reservation. Reviewed-by: Omar Sandoval Reviewed-by: Liu Bo Reviewed-by: Nikolay Borisov Signed-off-by: Josef

[PATCH 20/36] btrfs: don't use ctl->free_space for max_extent_size

2018-09-11 Thread Josef Bacik
From: Josef Bacik max_extent_size is supposed to be the largest contiguous range for the space info, and ctl->free_space is the total free space in the block group. We need to keep track of these separately and _only_ use the max_free_space if we don't have a max_extent_size, as that means our

[PATCH 25/36] btrfs: pass delayed_refs_root to btrfs_delayed_ref_lock

2018-09-11 Thread Josef Bacik
We don't need the trans except to get the delayed_refs_root, so just pass the delayed_refs_root into btrfs_delayed_ref_lock and call it a day. Reviewed-by: Nikolay Borisov Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 5 + fs/btrfs/delayed-ref.h | 2 +- fs/btrfs/extent-tree.c | 2

[PATCH 01/36] btrfs: add btrfs_delete_ref_head helper

2018-09-11 Thread Josef Bacik
From: Josef Bacik We do this dance in cleanup_ref_head and check_ref_cleanup, unify it into a helper and cleanup the calling functions. Signed-off-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 14 ++ fs/btrfs/delayed-ref.h | 3 ++- fs/btrfs/extent-tree.c | 22

[PATCH 02/36] btrfs: add cleanup_ref_head_accounting helper

2018-09-11 Thread Josef Bacik
From: Josef Bacik We were missing some quota cleanups in check_ref_cleanup, so break the ref head accounting cleanup into a helper and call that from both check_ref_cleanup and cleanup_ref_head. This will hopefully ensure that we don't screw up accounting in the future for other things that we

[PATCH 00/35][v2] My current patch queue

2018-09-11 Thread Josef Bacik
v1->v2: - addressed all of the issues brought up. - added more comments. - split up some patches. original message: This is the current queue of things that I've been working on. The main thing these patches are doing is separating out the delayed refs reservations from the global reserve into

Re: btrfs-progs: btrfs-convert: unable to find block group for 0

2018-09-11 Thread David Sterba
On Tue, Sep 11, 2018 at 08:55:30PM +0800, Qu Wenruo wrote: > > > On 2018/9/11 下午10:52, David Sterba wrote: > > On Tue, Aug 28, 2018 at 08:07:52AM +0800, Qu Wenruo wrote: > >> > >> > >> On 2018/8/28 上午6:33, Tucker Boniface wrote: > >>> Hello, I am trying to convert an ext4 partition to btrfs

Re: [PATCH] btrfs-progs: qgroup: Don't return 1 if qgroup is marked inconsistent during relationship assignment

2018-09-11 Thread David Sterba
On Fri, Aug 10, 2018 at 10:34:06AM +0800, Qu Wenruo wrote: > BTRFS_IOC_QGROUP_ASSIGN ioctl could return >0 if qgroup is marked > inconsistent after successful relationship assignment/removal. > > We leak the return value as the final return value of btrfs command. > But according to the man page,

Re: [PATCH 0/6] btrfs-progs: Variant fixes for fuzz-tests

2018-09-11 Thread David Sterba
On Wed, Aug 29, 2018 at 01:27:39PM +0800, Qu Wenruo wrote: > Gentle ping. > > These fixes are pretty small, I'd like to see them merged before I need > to rebase them again and again. I've merged them now, thanks. I had to edit all changelogs and remove the - lines, shortened the paths in

Editing 4

2018-09-11 Thread Aaron
Hi, If you have photos for editing, please send email to: hansre...@outlook.com We have 12 in house image editors and we can help you for cutting out your photos, or path the photos. Includes retouching if needed. Used for products photos or portrait photos, catalog photos. You may drop us

Editing 4

2018-09-11 Thread Aaron
Hi, If you have photos for editing, please send email to: hansre...@outlook.com We have 12 in house image editors and we can help you for cutting out your photos, or path the photos. Includes retouching if needed. Used for products photos or portrait photos, catalog photos. You may drop us

Re: [PATCH 0/3] btrfs-progs: Detect compressed extent without csum

2018-09-11 Thread David Sterba
On Mon, May 14, 2018 at 02:54:41PM +0800, Qu Wenruo wrote: > Patches can be fetch from github: > https://github.com/adam900710/btrfs-progs/tree/compress_nodatasum > > It's based on v4.16 stable branch. > > James Harvey from mail list reports a strange kernel panic, whichs show > obviously kernel

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-09-11 Thread David Sterba
On Tue, Sep 04, 2018 at 08:39:55PM +0800, Qu Wenruo wrote: > > > On 2018/8/23 下午3:45, Qu Wenruo wrote: > > > > > > On 2018/8/23 下午3:36, Nikolay Borisov wrote: > >> > >> > >> On 23.08.2018 10:31, Qu Wenruo wrote: > >>> Introduce --breadth-first option to do breadth-first tree dump. > >>> This

Re: [PATCH 1/2] Install uncompressed man pages

2018-09-11 Thread David Sterba
On Tue, Aug 07, 2018 at 04:09:57PM +0200, David Sterba wrote: > On Mon, Aug 06, 2018 at 04:54:31PM -0400, Mike Gilbert wrote: > > Build systems do not typically compress man pages when installing them. > > This is generally left to distro packaging mechanisms, which may end up > > recompressing

Re: Scrub aborts due to corrupt leaf

2018-09-11 Thread Larkin Lowrey
On 8/29/2018 1:32 AM, Qu Wenruo wrote: On 2018/8/28 下午9:56, Chris Murphy wrote: On Tue, Aug 28, 2018 at 7:42 AM, Qu Wenruo wrote: On 2018/8/28 下午9:29, Larkin Lowrey wrote: On 8/27/2018 10:12 PM, Larkin Lowrey wrote: On 8/27/2018 12:46 AM, Qu Wenruo wrote: The system uses ECC memory and

Re: [PATCH] btrfs-progs: Continue checking even we found something wrong in free space cache

2018-09-11 Thread David Sterba
On Tue, Sep 04, 2018 at 08:41:28PM +0800, Qu Wenruo wrote: > No need to abort checking, especially for RO check free space cache is > meaningless, the errors in fs/extent tree is more interesting for > developers. > > So continue checking even something in free space cache is wrong. > >

Re: [PATCH] btrfs-progs: calibrate extent_end when found a gap

2018-09-11 Thread David Sterba
On Tue, Sep 04, 2018 at 08:42:01PM +0800, Lu Fengqi wrote: > The extent_end will be used to check whether there is gap between this > extent and next extent. If it is not calibrated, check_file_extent will Do you mean 'synchronized' or 'matching'. > mistake that there are gaps between the

Re: [PATCH 0/5] rb_first to rb_first_cached conversion

2018-09-11 Thread David Sterba
On Thu, Aug 23, 2018 at 03:51:48AM +0800, Liu Bo wrote: > Several structs in btrfs are using rb_first() in a while loop, it'd be > more efficient to do this with rb_first_cached() which has the O(1) > complexity. We'd like to see some numbers, though just by reading the code I think there's going

Re: I HAVE NOT HEARD FROM YOUR

2018-09-11 Thread Reem Al-Hashimy
Hello, My name is ms. Reem Al-Hashimi. The UAE minister of state for international cooparation. I got your contact from a certain email database from your country while i was looking for someone to handle a huge financial transaction for me in confidence. Can you receive and invest on behalf

Re: btrfs-progs: btrfs-convert: unable to find block group for 0

2018-09-11 Thread Qu Wenruo
On 2018/9/11 下午10:52, David Sterba wrote: > On Tue, Aug 28, 2018 at 08:07:52AM +0800, Qu Wenruo wrote: >> >> >> On 2018/8/28 上午6:33, Tucker Boniface wrote: >>> Hello, I am trying to convert an ext4 partition to btrfs using >>> btrfs-convert. I am running Arch Linux with kernel 4.18.5 and >>>

Re: btrfs-progs: btrfs-convert: unable to find block group for 0

2018-09-11 Thread David Sterba
On Tue, Aug 28, 2018 at 08:07:52AM +0800, Qu Wenruo wrote: > > > On 2018/8/28 上午6:33, Tucker Boniface wrote: > > Hello, I am trying to convert an ext4 partition to btrfs using > > btrfs-convert. I am running Arch Linux with kernel 4.18.5 and > > btrfs-progs 4.17.1. The full error is inline

Re: [PATCH] btrfs-progs: print-tree: Skip deprecated blockptr / nodesize output

2018-09-11 Thread David Sterba
On Thu, Aug 23, 2018 at 02:11:29PM +0800, Qu Wenruo wrote: > When printing tree nodes, we output slots like: > key (EXTENT_TREE ROOT_ITEM 0) block 73625600 (17975) gen 16 > > The number in the parentheses is blockptr / nodesize. > > However this number doesn't really do any thing useful. > And

Re: [PATCH] btrfs-progs: dump-tree: print invalid argument and strerror

2018-09-11 Thread David Sterba
On Thu, Aug 30, 2018 at 05:08:04PM +0800, Su Yue wrote: > Before this patch: > $ ls nothingness > ls: cannot access 'nothingness': No such file or directory > $ btrfs inspect-internal dump-tree nothingness > ERROR: not a block device or regular file: nothingness > > The confusing error message

Re: [PATCH] btrfs-progs: tests: Add test for missing device delete error value

2018-09-11 Thread David Sterba
On Mon, Sep 10, 2018 at 07:57:20PM +0200, David Sterba wrote: > On Mon, Sep 03, 2018 at 01:02:57PM +0300, Nikolay Borisov wrote: > > Add a test which ensures the kernel returns the correct error value > > when missing device removal is requested. This test verifies that kernel > > refactoring

Re: [PATCH] btrfs-progs: docs: update document about option -R of btrfs-scrub

2018-09-11 Thread David Sterba
On Mon, Sep 03, 2018 at 11:07:07AM +0800, Su Yue wrote: > The option '-R' of btrfs-scrub was documented by mistake as > 'print raw statistics per-device instead of a summary'. > > Here change it to 'raw print mode, print full data instead of > summary' which it works actually. > > Fixes:

Re: [RFC PATCH v2 1/4] btrfs: factor out btrfs_link_subvol from create_subvol

2018-09-11 Thread Lu Fengqi
On Tue, Sep 11, 2018 at 07:57:03PM +0800, Qu Wenruo wrote: > > >On 2018/9/11 下午7:29, Lu Fengqi wrote: >> The function btrfs_link_subvol is responsible to link the subvolume to >> the specified directory, which is the opposite of what >> btrfs_unlink_subvol does. >> >> No functional change. >> >>

Re: [RFC PATCH v2 1/4] btrfs: factor out btrfs_link_subvol from create_subvol

2018-09-11 Thread Qu Wenruo
On 2018/9/11 下午7:29, Lu Fengqi wrote: > The function btrfs_link_subvol is responsible to link the subvolume to > the specified directory, which is the opposite of what > btrfs_unlink_subvol does. > > No functional change. > > Signed-off-by: Lu Fengqi The patch itself is OK. Just small

Re: [PATCH v3 00/10] undelete subvolume offline version

2018-09-11 Thread Lu Fengqi
On Mon, May 07, 2018 at 11:10:23AM +0800, Lu Fengqi wrote: >This patchset will add undelete-subvol subcommand for btrfs rescue. > Hi David Although there are some disagreements about undeleting subvolumes online implementation, the offline version is considered more acceptable. Would you like to

[RFC PATCH v2 2/2] btrfs-progs: subvolume: undelete: add btrfs subvolume undelete subcommand

2018-09-11 Thread Lu Fengqi
Add the undelete subcommand, this is depend on the BTRFS_IOC_SUBVOL_UNDELETE ioctl. Signed-off-by: Lu Fengqi --- btrfs-completion | 2 +- cmds-subvolume.c | 70 2 files changed, 71 insertions(+), 1 deletion(-) diff --git a/btrfs-completion

[RFC PATCH v2 1/2] btrfs-progs: ioctl: add BTRFS_IOC_SUBVOL_UNDELETE to ioctl.h

2018-09-11 Thread Lu Fengqi
Copied from uapi/linux/btrfs.h. Signed-off-by: Lu Fengqi --- ioctl.h | 7 +++ 1 file changed, 7 insertions(+) diff --git a/ioctl.h b/ioctl.h index 709e996f401c..75978a4e8265 100644 --- a/ioctl.h +++ b/ioctl.h @@ -670,6 +670,11 @@ struct btrfs_ioctl_send_args_64 { }

Re: List of known BTRFS Raid 5/6 Bugs?

2018-09-11 Thread Stefan K
wow, holy shit, thanks for this extended answer! > The first thing to point out here again is that it's not btrfs-specific. so that mean that every RAID implemantation (with parity) has such Bug? I'm looking a bit, it looks like that ZFS doesn't have a write hole. And it _only_ happens when

[RFC PATCH v2 4/4] btrfs: undelete: Add BTRFS_IOCTL_SUBVOL_UNDELETE ioctl

2018-09-11 Thread Lu Fengqi
This ioctl will provide user the ability to recover the subvolume of the given id to the given directory. Note: It will lock fs_info->cleaner_mutex to keep the cleaner kthread from deleting the subvolume which we want to recover. Signed-off-by: Lu Fengqi --- fs/btrfs/ioctl.c | 64

[RFC PATCH v2 3/4] btrfs: undelete: introduce btrfs_undelete_subvolume

2018-09-11 Thread Lu Fengqi
The function will do the following things which are almost the opposite of what btrfs_delete_subvolume() does: 1. link the subvolume to the parent specified; 2. clear root flag and set root_refs to 1; 3. add the subvol to the uuid_tree; 4. delete the orphan_item. Signed-off-by: Lu Fengqi ---

[RFC PATCH v2 0/4] undelete subvolume online version

2018-09-11 Thread Lu Fengqi
This patchset will add the BTRFS_IOC_SUBVOL_UNDELETE ioctl for online btrfs subvolume undelete. And using the online_undelete version of btrfs-progs, user can recover the subvolume given by to the directory given by . The optional parameter [-n ] can be used to set the name of the recovered

[RFC PATCH v2 1/4] btrfs: factor out btrfs_link_subvol from create_subvol

2018-09-11 Thread Lu Fengqi
The function btrfs_link_subvol is responsible to link the subvolume to the specified directory, which is the opposite of what btrfs_unlink_subvol does. No functional change. Signed-off-by: Lu Fengqi --- fs/btrfs/ioctl.c | 64 +++- 1 file changed, 41

[RFC PATCH v2 2/4] btrfs: don't BUG_ON() in btrfs_link_subvol()

2018-09-11 Thread Lu Fengqi
Both of btrfs_update_inode() and btrfs_add_root_ref() may fail because of ENOMEM. So there's no reason to panic here, we can replace BUG_ON() with btrfs_abort_transaction() here. Signed-off-by: Lu Fengqi --- fs/btrfs/ioctl.c | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff

Re: compiling btrfs-progs 4.17.1 gives error "reiserfs/misc.h: No such file or directory"

2018-09-11 Thread Jürgen Herrmann
Am 7.9.2018 17:46, schrieb Chris Murphy: On Fri, Sep 7, 2018 at 3:56 AM, Jürgen Herrmann wrote: Hello! I'm having a problem with btrfs send which stops after several seconds. The process hangs with 100% cpu time on one cpu. The system is still responsive to input but no io is happening

[PATCH v3] fstests: btrfs/149 make it sectorsize independent

2018-09-11 Thread Anand Jain
Originally this test case was designed to work with 4K sectorsize. Now enhance it to work with any sector sizes and makes the following changes: .out file not to contain any traces of sector size. Use max_inline=0 mount option so that it meets the requisite of non inline regular extent. Don't log

Re: [PATCH 4/4] btrfs: tests: polish ifdefs around testing helper

2018-09-11 Thread David Sterba
On Mon, Sep 10, 2018 at 04:43:29PM -0700, Omar Sandoval wrote: > On Mon, Sep 10, 2018 at 07:22:31PM +0200, David Sterba wrote: > > Avoid the inline ifdefs and use two sections for self-tests enabled and > > disabled. > > > > Signed-off-by: David Sterba > > --- > > fs/btrfs/ctree.h | 9 ++---

Re: [PATCH 1/4] btrfs: tests: add separate stub for find_lock_delalloc_range

2018-09-11 Thread David Sterba
On Mon, Sep 10, 2018 at 04:40:55PM -0700, Omar Sandoval wrote: > On Mon, Sep 10, 2018 at 07:22:24PM +0200, David Sterba wrote: > > The helper find_lock_delalloc_range is now conditionally built static, > > dpending on whether the self-tests are enabled or not. There's a macro > > that is suppsed

Re: [PATCH v2 0/5] btrfs: qgroup: Skip unrelated tree blocks for balance

2018-09-11 Thread David Sterba
On Tue, Sep 11, 2018 at 10:43:32AM +0800, Qu Wenruo wrote: > > > On 2018/9/7 下午5:32, Qu Wenruo wrote: > > This patchset can be fetched from github: > > https://github.com/adam900710/linux/tree/qgroup_balance_skip_trees > > The base commit is v4.19-rc1 tag. > > > > There are a lot of reports of