Re: [PATCH] Btrfs: make plug in writing meta blocks really work

2017-08-21 Thread Liu Bo
On Mon, Aug 21, 2017 at 03:23:30PM -0400, Josef Bacik wrote: > On Mon, Aug 21, 2017 at 12:14:16PM -0700, Liu Bo wrote: > > On Mon, Aug 21, 2017 at 01:48:01PM -0400, Josef Bacik wrote: > > > On Fri, Aug 18, 2017 at 11:42:07AM -0600, Liu Bo wrote: > > >

Re: [PATCH] Btrfs: make plug in writing meta blocks really work

2017-08-21 Thread Liu Bo
On Mon, Aug 21, 2017 at 01:48:01PM -0400, Josef Bacik wrote: > On Fri, Aug 18, 2017 at 11:42:07AM -0600, Liu Bo wrote: > > We have started plug in btrfs_write_and_wait_marked_extents() but the > > generated IOs actually go to device's schedule IO list where the work > >

Re: [PATCH v2] btrfs: submit superblock io with REQ_META and REQ_PRIO

2017-08-21 Thread Liu Bo
would effectively block the whole transaction, also blocking any other > operation holding the device_list_mutex. > Reviewed-by: Liu Bo -liubo > Signed-off-by: David Sterba > --- > fs/btrfs/disk-io.c | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > > d

[PATCH v3 3/7] Btrfs: remove BUG() in btrfs_extent_inline_ref_size

2017-08-18 Thread Liu Bo
Now that btrfs_get_extent_inline_ref_type() can report if type is a valid one and all callers can gracefully deal with that, we don't need to crash here. Signed-off-by: Liu Bo --- fs/btrfs/ctree.h | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h

[PATCH v3 7/7] Btrfs: add one more sanity check for shared ref type

2017-08-18 Thread Liu Bo
e has been misused. This adds the above mentioned check and also updates print_extent_item() called by btrfs_print_leaf() to point out the invalid ref while printing the tree structure. Signed-off-by: Liu Bo --- fs/btrfs/extent-tree.c | 29 + fs/btrfs/print-tr

[PATCH v3 0/7] add sanity check for extent inline ref type

2017-08-18 Thread Liu Bo
RN(1) to btrfs_err so that we know which btrfs has that error. v2: - add enum type and return BTRFS_REF_TYPE_INVALID instead of -EINVAL. - remove one more BUG_ON which is in __add_tree_block. - add validation check for shared refs. - improve btrfs_print_leaf to show which refs has something

[PATCH v3 2/7] Btrfs: convert to use btrfs_get_extent_inline_ref_type

2017-08-18 Thread Liu Bo
Since we have a helper which can do sanity check, this converts all btrfs_extent_inline_ref_type to it. Signed-off-by: Liu Bo --- fs/btrfs/backref.c | 11 +-- fs/btrfs/extent-tree.c | 36 ++-- fs/btrfs/relocation.c | 13 +++-- 3 files changed

[PATCH v3 6/7] Btrfs: remove BUG_ON in __add_tree_block

2017-08-18 Thread Liu Bo
WARN() followed by calling btrfs_print_leaf() to show more details about what's going on and returning -EINVAL to upper callers. Signed-off-by: Liu Bo --- fs/btrfs/relocation.c | 12 +++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/relocation.c b/fs/

[PATCH v3 4/7] Btrfs: remove BUG() in print_extent_item

2017-08-18 Thread Liu Bo
btrfs_print_leaf() is used in btrfs_get_extent_inline_ref_type, so here we really want to print the invalid value of ref type instead of causing a kernel panic. Signed-off-by: Liu Bo --- fs/btrfs/print-tree.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/print

[PATCH v3 5/7] Btrfs: remove BUG() in add_data_reference

2017-08-18 Thread Liu Bo
Now that we have a helper to report invalid value of extent inline ref type, we need to quit gracefully instead of throwing out a kernel panic. Signed-off-by: Liu Bo --- fs/btrfs/relocation.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/relocation.c b/fs

[PATCH v3 1/7] Btrfs: add a helper to retrive extent inline ref type

2017-08-18 Thread Liu Bo
An invalid value of extent inline ref type may be read from a malicious image which may force btrfs to crash. This adds a helper which does sanity check for the ref type, so we can know if it's sane, return type if so, otherwise return an error. Signed-off-by: Liu Bo --- fs/btrfs/ct

Re: [PATCH] btrfs: submit superblock io with REQ_META

2017-08-18 Thread Liu Bo
On Fri, Aug 18, 2017 at 06:21:53PM +0200, David Sterba wrote: > The superblock is also metadata of the filesystem so the relevant IO > should be tagged as such. > Reviewed-by: Liu Bo -liubo > Signed-off-by: David Sterba > --- > fs/btrfs/disk-io.c | 6 -- > 1 file

[PATCH] Btrfs: do not async submit for nodatacsum inodes

2017-08-18 Thread Liu Bo
IO list instead of DIO's plug list, so that IOs get much less merges in general. Lets use sync submit for nodatasum inodes. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 95c2120..f4a

[PATCH] Btrfs: remove unnecessary skip_sum

2017-08-18 Thread Liu Bo
Now that we can get inode from dip->inode, it's not necessary to pass skip_sum from callers. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 18 ++ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index f4a48d8..91e701c 1006

[PATCH] Btrfs: make plug in writing meta blocks really work

2017-08-18 Thread Liu Bo
s the same case as writing log tree, doing sync submit can merge more IOs. Signed-off-by: Liu Bo --- fs/btrfs/disk-io.c | 6 -- fs/btrfs/transaction.c | 2 ++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 080e2eb..8d097ba 100644

Re: qcow2 images make scrub believe the filesystem is corrupted.

2017-08-18 Thread Liu Bo
On Fri, Aug 18, 2017 at 06:23:18PM +0200, Goffredo Baroncelli wrote: > On 08/18/2017 01:39 AM, Josef Bacik wrote: > [...] > > This is happening because the app (the guest OS in this case, we saw this a > > lot > > with windows guests) is changing the pages while they are in flight. We > > calcula

Re: 8 disk metadata radi10 + data raid1

2017-08-17 Thread Liu Bo
On Thu, Aug 17, 2017 at 10:41:59PM +0200, waxhead wrote: > Hi, > > On one of my machines I run a BTRFS filesystem with the following > configuration > > Kernel: 4.11.0-1-amd64 #1 SMP Debian 4.11.6-1 (2017-06-19) x86_64 GNU/Linux > Disks: 8 > Metadata: Raid 10 > Data: Raid1 > > One of the disks i

Re: [PATCH v2 0/7] add sanity check for extent inline ref type

2017-08-16 Thread Liu Bo
On Wed, Aug 16, 2017 at 04:53:15PM +0200, David Sterba wrote: > On Mon, Aug 07, 2017 at 03:55:24PM -0600, Liu Bo wrote: > > An invalid extent inline ref type could be read from a btrfs image and > > it ends up with a panic[1], this set is to deal with the insane value > > gr

Re: [PATCH] Btrfs: incremental send, fix emission of invalid clone operations

2017-08-15 Thread Liu Bo
nting the destination filesystem with compression enabled. > > So fix this by issuing regular write operations in the send stream > instead of clone operations when the source offset is zero and the > range has a length matching the sector size. Reviewed-by: Liu Bo Thanks, -liubo

[PATCH v3] Btrfs: fix out of bounds array access while reading extent buffer

2017-08-09 Thread Liu Bo
be the first item in the eb, then offset/length will get over eb->len which ends up an invalid memory access. This is adding proper checks in order to avoid invalid memory access, ie. 'general protection fault', before it's too late. Reviewed-by: Filipe Manana Signed-off-by:

Re: [PATCH v2] Btrfs: fix out of bounds array access while reading extent buffer

2017-08-09 Thread Liu Bo
On Wed, Aug 09, 2017 at 06:40:04PM +0100, Filipe Manana wrote: > On Wed, Aug 9, 2017 at 5:31 PM, Liu Bo wrote: > > There is a cornel case that slip through the checkers in functions > > reading extent buffer, ie. > > > > if (start < eb->len) and (start + l

[PATCH v2] Btrfs: fix out of bounds array access while reading extent buffer

2017-08-09 Thread Liu Bo
be the first item in the eb, then offset/length will get over eb->len which ends up an invalid memory access. This is adding proper checks in order to avoid invalid memory access, ie. 'general protection fault', before it's too late. Signed-off-by: Liu Bo --- v2: Improve the

Re: [PATCH] Btrfs: fix out of bounds array access while reading extent buffer

2017-08-08 Thread Liu Bo
Hi Filipe, On Tue, Aug 08, 2017 at 09:47:21AM +0100, Filipe Manana wrote: > On Mon, Aug 7, 2017 at 8:39 PM, Liu Bo wrote: > > There is a cornel case that slip through the checkers in functions > > reading extent buffer, ie. > > > > if (start < eb->len) and

[PATCH v2 5/7] Btrfs: remove BUG() in add_data_reference

2017-08-07 Thread Liu Bo
Now that we have a helper to report invalid value of extent inline ref type, we need to quit gracefully instead of throwing out a kernel panic. Signed-off-by: Liu Bo --- fs/btrfs/relocation.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/relocation.c b/fs

[PATCH v2 7/7] Btrfs: add one more sanity check for shared ref type

2017-08-07 Thread Liu Bo
e has been misused. This adds the above mentioned check and also updates print_extent_item() called by btrfs_print_leaf() to point out the invalid ref while printing the tree structure. Signed-off-by: Liu Bo --- fs/btrfs/extent-tree.c | 29 + fs/btrfs/print-tr

[PATCH v2 1/7] Btrfs: add a helper to retrive extent inline ref type

2017-08-07 Thread Liu Bo
eturn BTRFS_REF_TYPE_INVALID instead of -EINVAL Signed-off-by: Liu Bo --- fs/btrfs/ctree.h | 11 +++ fs/btrfs/extent-tree.c | 36 2 files changed, 47 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 3f3eb7b..9866420 100644 --- a/fs/btrfs/ctree.h

[PATCH v2 2/7] Btrfs: convert to use btrfs_get_extent_inline_ref_type

2017-08-07 Thread Liu Bo
Since we have a helper which can do sanity check, this converts all btrfs_extent_inline_ref_type to it. Signed-off-by: Liu Bo --- fs/btrfs/backref.c | 11 +-- fs/btrfs/extent-tree.c | 36 ++-- fs/btrfs/relocation.c | 13 +++-- 3 files changed

[PATCH v2 6/7] Btrfs: remove BUG_ON in __add_tree_block

2017-08-07 Thread Liu Bo
WARN() followed by calling btrfs_print_leaf() to show more details about what's going on and returning -EINVAL to upper callers. Signed-off-by: Liu Bo --- fs/btrfs/relocation.c | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/relocation.c b/fs/

[PATCH v2 4/7] Btrfs: remove BUG() in print_extent_item

2017-08-07 Thread Liu Bo
btrfs_print_leaf() is used in btrfs_get_extent_inline_ref_type, so here we really want to print the invalid value of ref type instead of causing a kernel panic. Signed-off-by: Liu Bo --- fs/btrfs/print-tree.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/print

[PATCH v2 3/7] Btrfs: remove BUG() in btrfs_extent_inline_ref_size

2017-08-07 Thread Liu Bo
Now that btrfs_get_extent_inline_ref_type() can report if type is a valid one and all callers can gracefully deal with that, we don't need to crash here. Signed-off-by: Liu Bo --- fs/btrfs/ctree.h | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h

[PATCH v2 0/7] add sanity check for extent inline ref type

2017-08-07 Thread Liu Bo
INVAL. - remove one more BUG_ON which is in __add_tree_block. - add validation check for shared refs. - improve btrfs_print_leaf to show which refs has something wrong. Liu Bo (7): Btrfs: add a helper to retrive extent inline ref type Btrfs: convert to use btrfs_get_extent_inline_ref_type Bt

[PATCH] Btrfs: fix out of bounds array access while reading extent buffer

2017-08-07 Thread Liu Bo
ne ref type from shared data ref to non-shared data ref. This is adding proper checks in order to avoid invalid memory access, ie. 'general protection', before it's too late. Signed-off-by: Liu Bo --- fs/btrfs/extent_io.c | 22 ++ 1 file changed, 14 insertions(+),

[PATCH v2] Btrfs: search parity device wisely

2017-08-03 Thread Liu Bo
After mapping block with BTRFS_MAP_WRITE, parities have been sorted to the end position, so this search can start from the first parity stripe. Signed-off-by: Liu Bo --- v2: fix typo (data_stripes -> nr_data). fs/btrfs/raid56.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) d

[PATCH] Btrfs: search parity device wisely

2017-08-02 Thread Liu Bo
After mapping block with BTRFS_MAP_WRITE, parities have been sorted to the end position, so this search can start from the first parity stripe. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-02 Thread Liu Bo
On Wed, Aug 02, 2017 at 10:41:30PM +0200, Goffredo Baroncelli wrote: > Hi Liu, > > thanks for your reply, below my comments > On 2017-08-02 19:57, Liu Bo wrote: > > On Wed, Aug 02, 2017 at 12:14:27AM +0200, Goffredo Baroncelli wrote: > >> On 2017-08-01 19:24, Liu Bo

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-02 Thread Liu Bo
On Wed, Aug 02, 2017 at 12:14:27AM +0200, Goffredo Baroncelli wrote: > On 2017-08-01 19:24, Liu Bo wrote: > > On Tue, Aug 01, 2017 at 07:42:14PM +0200, Goffredo Baroncelli wrote: > >> Hi Liu, > >> > >> On 2017-08-01 18:14, Liu Bo wrote: > >>> Th

Re: BTRFS error: bad tree block start 0 623771648

2017-08-01 Thread Liu Bo
On Tue, Aug 01, 2017 at 11:04:10AM +0500, Roman Mamedov wrote: > On Mon, 31 Jul 2017 11:12:01 -0700 > Liu Bo wrote: > > > Superblock and chunk tree root is OK, looks like the header part of > > the tree root is now all-zero, but I'm unable to think of a btrfs bug >

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-01 Thread Liu Bo
On Tue, Aug 01, 2017 at 07:42:14PM +0200, Goffredo Baroncelli wrote: > Hi Liu, > > On 2017-08-01 18:14, Liu Bo wrote: > > This aims to fix write hole issue on btrfs raid5/6 setup by adding a > > separate disk as a journal (aka raid5/6 log), so that after unclean > > shu

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-01 Thread Liu Bo
On Tue, Aug 01, 2017 at 01:39:59PM -0400, Austin S. Hemmelgarn wrote: > On 2017-08-01 13:25, Roman Mamedov wrote: > > On Tue, 1 Aug 2017 10:14:23 -0600 > > Liu Bo wrote: > > > > > This aims to fix write hole issue on btrfs raid5/6 setup by adding a > > >

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-01 Thread Liu Bo
On Tue, Aug 01, 2017 at 10:25:47PM +0500, Roman Mamedov wrote: > On Tue, 1 Aug 2017 10:14:23 -0600 > Liu Bo wrote: > > > This aims to fix write hole issue on btrfs raid5/6 setup by adding a > > separate disk as a journal (aka raid5/6 log), so that after unclean > >

Re: [PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-01 Thread Liu Bo
On Tue, Aug 01, 2017 at 05:28:57PM +, Hugo Mills wrote: >Hi, > >Great to see something addressing the write hole at last. > > On Tue, Aug 01, 2017 at 10:14:23AM -0600, Liu Bo wrote: > > This aims to fix write hole issue on btrfs raid5/6 setup by adding a &

[PATCH 12/14] Btrfs: raid56: fix error handling while adding a log device

2017-08-01 Thread Liu Bo
, and step #2 is to assign the pointer in %fs_info. And by running step #1 ahead of starting transaction, we can gracefully bail out on errors now. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 48 +--- fs/btrfs/raid56.h | 5 + fs/btrfs/volumes.c

[PATCH 10/14] Btrfs: raid56: use the readahead helper to get page

2017-08-01 Thread Liu Bo
This updates recovery code to use the readahead helper. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 24f7cbb..8f47e56 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c

[PATCH 06/14] Btrfs: raid56: add reclaim support

2017-08-01 Thread Liu Bo
in the log and raid5/6 array. Also we need to take care of the case when IOs get reordered. A list is used to keep the order right. Signed-off-by: Liu Bo --- fs/btrfs/ctree.h | 10 +++- fs/btrfs/raid56.c | 63 -- fs/btrfs/transact

[PATCH 01/14] Btrfs: raid56: add raid56 log via add_dev v2 ioctl

2017-08-01 Thread Liu Bo
This introduces add_dev_v2 ioctl to add a device as raid56 journal device. With the help of a journal device, raid56 is able to to get rid of potential write holes. Signed-off-by: Liu Bo --- fs/btrfs/ctree.h| 6 ++ fs/btrfs/ioctl.c| 48

[PATCH 09/14] Btrfs: raid56: add readahead for recovery

2017-08-01 Thread Liu Bo
While doing recovery, blocks are read from the raid5/6 disk one by one, so this is adding readahead so that we can read at most 256 contiguous blocks in one read IO. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 114 +++--- 1 file changed, 109

[PATCH 14/14] Btrfs: raid56: maintain IO order on raid5/6 log

2017-08-01 Thread Liu Bo
int offset where the highest successful writeback is, we cannot allow IO to be reordered. This is adding a list in which IO order is maintained properly. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 42 ++ fs/btrfs/raid56.h | 5 + 2 files changed,

[PATCH 1/2] Btrfs-progs: add option to add raid5/6 log device

2017-08-01 Thread Liu Bo
This introduces an option for 'btrfs device add' to add a device as raid5/6 log at run time. Signed-off-by: Liu Bo --- cmds-device.c | 30 +- ioctl.h | 3 +++ 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/cmds-device.c b/cmds-devi

[PATCH 07/14] Btrfs: raid56: load r5log

2017-08-01 Thread Liu Bo
check if anything needs to be replayed, otherwise it creates a new empty block at the beginning of the disk and new writes will append to it. Signed-off-by: Liu Bo --- fs/btrfs/disk-io.c | 16 +++ fs/btrfs/raid56.c | 128 + fs/btrfs/rai

[PATCH 05/14] Btrfs: raid56: add stripe log for raid5/6

2017-08-01 Thread Liu Bo
ta block is written to the raid5/6 stripe log with data/parity in a single bio(could be two bios, doesn't support more than two bios). Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 512 +++--- fs/btrfs/raid56.h | 65 +++ 2 files changed, 51

[PATCH 08/14] Btrfs: raid56: log recovery

2017-08-01 Thread Liu Bo
/data/parity. If a power loss happens, the new recovery starts again from block A, and since block B is now valid, it may replay block C as well which has become stale. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 151 ++ 1 file changed, 151 inser

[PATCH 04/14] Btrfs: raid56: add verbose debug

2017-08-01 Thread Liu Bo
Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 2 ++ fs/btrfs/volumes.c | 7 ++- fs/btrfs/volumes.h | 4 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 2b91b95..c75766f 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c

[PATCH 2/2] Btrfs-progs: introduce super_journal_tail to inspect-dump-super

2017-08-01 Thread Liu Bo
We've record journal_tail of raid5/6 log in super_block so that recovery of raid5/6 log can scan from this position. This teaches inspect-dump-super to acknowledge %journal_tail. Signed-off-by: Liu Bo --- cmds-inspect-dump-super.c | 2 ++ ctree.h | 6 +- 2 files ch

[PATCH 13/14] Btrfs: raid56: initialize raid5/6 log after adding it

2017-08-01 Thread Liu Bo
We need to initialize the raid5/6 log after adding it, but we don't want to race with concurrent writes. So we initialize it before assigning the log pointer in %fs_info. Signed-off-by: Liu Bo --- fs/btrfs/disk-io.c | 2 +- fs/btrfs/raid56.c | 18 -- fs/btrfs/raid56.h

[PATCH 03/14] Btrfs: raid56: detect raid56 log on mount

2017-08-01 Thread Liu Bo
We've put the flag BTRFS_DEV_RAID56_LOG in device->type, so we can recognize the journal device of raid56 while reading the chunk tree. Signed-off-by: Liu Bo --- fs/btrfs/volumes.c | 12 1 file changed, 12 insertions(+) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes

[PATCH 02/14] Btrfs: raid56: do not allocate chunk on raid56 log

2017-08-01 Thread Liu Bo
The journal device (aka raid56 log) is not for chunk allocation, lets skip it. Signed-off-by: Liu Bo --- fs/btrfs/volumes.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index dafc541..5c50df7 100644 --- a/fs/btrfs/volumes.c +++ b

[PATCH 00/14 RFC] Btrfs: Add journal for raid5/6 writes

2017-08-01 Thread Liu Bo
s currently based on 4.12-rc3. The patch set is tagged with RFC, and comments are always welcome, thanks. Known limitations: - Deleting a log device is not implemented yet. Liu Bo (14): Btrfs: raid56: add raid56 log via add_dev v2 ioctl Btrfs: raid56: do not allocate chunk on raid56 log

[PATCH 11/14] Btrfs: raid56: add csum support

2017-08-01 Thread Liu Bo
block, we just skip this this meta/data/parity pair and move onto the next one. Signed-off-by: Liu Bo --- fs/btrfs/raid56.c | 235 -- fs/btrfs/raid56.h | 4 + 2 files changed, 197 insertions(+), 42 deletions(-) diff --git a/fs/btrfs/raid56.c

Re: BTRFS error: bad tree block start 0 623771648

2017-07-31 Thread Liu Bo
On Sun, Jul 30, 2017 at 06:14:35PM +0200, marcel.cochem wrote: > Hello, > > I had to shut down my PC because it hang up, after the next reboot i > got the following error: > BTRFS error on /dev/nvme0n1p4 bad tree block start 0 623771648 > open_ctree failed > mount wrong fs type > b

Re: [PATCH v3 12/13] btrfs: allow backref search checks for shared extents

2017-07-28 Thread Liu Bo
On Wed, Jul 12, 2017 at 04:20:10PM -0600, Edmund Nadolski wrote: > When called with a struct share_check, find_parent_nodes() > will detect a shared extent and immediately return with > BACKREF_SHARED_FOUND. > Reviewed-by: Liu Bo Thanks, -liubo > Signed-off-by: Edmund Nadolsk

Re: [PATCH v3 08/13] btrfs: convert prelimary reference tracking to use rbtrees

2017-07-28 Thread Liu Bo
veral hours now takes about half of that. This > runtime still isn't acceptable and a future patch will address that > by moving the rbtrees higher in the stack so the lookups can be > shared across multiple calls to find_parent_nodes. > Reviewed-by: Liu Bo Thanks, -liubo > S

Re: [PATCH 2/2] btrfs: increase ctx->pos for delayed dir index

2017-07-28 Thread Liu Bo
e we don't increase ctx->pos > after doing the dir_emit. > Looks good. Reviewed-by: Liu Bo Thanks, -liubo > Signed-off-by: Josef Bacik > --- > fs/btrfs/delayed-inode.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/fs/btrfs/delayed-inode.c b/f

Re: [PATCH 1/2][v2] btrfs: fix readdir deadlock with pagefault

2017-07-28 Thread Liu Bo
On Mon, Jul 24, 2017 at 03:14:25PM -0400, jo...@toxicpanda.com wrote: > From: Josef Bacik > > Readdir does dir_emit while under the btree lock. dir_emit can trigger > the page fault which means we can deadlock. Fix this by allocating a > buffer on opening a directory and copying the readdir int

Re: write corruption due to bio cloning on raid5/6

2017-07-26 Thread Liu Bo
On Mon, Jul 24, 2017 at 10:22:53PM +0200, Janos Toth F. wrote: > I accidentally ran into this problem (it's pretty silly because I > almost never run RC kernels or do dio writes but somehow I just > happened to do both at once, exactly before I read your patch notes). > I didn't initially catch any

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-14 Thread Liu Bo
On Fri, Jul 14, 2017 at 10:04:34AM +0100, Filipe Manana wrote: > On Thu, Jul 13, 2017 at 11:38 PM, Liu Bo wrote: > > On Thu, Jul 13, 2017 at 09:36:19AM -0700, Liu Bo wrote: > >> On Thu, Jul 13, 2017 at 03:09:54PM +0100, fdman...@kernel.org wrote: > >> > From: Fi

Re: [PATCH] Btrfs: fix unexpected return value of bio_readpage_error

2017-07-14 Thread Liu Bo
On Fri, Jul 14, 2017 at 02:32:09PM +0200, David Sterba wrote: > On Thu, Jul 13, 2017 at 03:00:50PM -0700, Liu Bo wrote: > > With blk_status_t, in bio_readpage_error() may return 1 as now > > ->submit_bio_hook() may not set %ret if it runs without problem. > > Huh, tha

[PATCH] Btrfs: fix unexpected return value of bio_readpage_error

2017-07-13 Thread Liu Bo
With blk_status_t, in bio_readpage_error() may return 1 as now ->submit_bio_hook() may not set %ret if it runs without problem. This fixes that unexpected return value by changing btrfs_check_repairable() to return a bool instead of updating %ret. Signed-off-by: Liu Bo --- fs/btrfs/extent_i

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 09:36:19AM -0700, Liu Bo wrote: > On Thu, Jul 13, 2017 at 03:09:54PM +0100, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > The recent changes to make bio cloning faster (added in the 4.13 merge > > window) by using the bio_cl

Re: Containers, Btrfs vs Btrfs + overlayfs

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 02:49:27PM -0600, Chris Murphy wrote: > Has anyone been working with Docker and Btrfs + overlayfs? It seems > superfluous or unnecessary to use overlayfs, but the shared page cache > aspect and avoiding some of the problems with large numbers of Btrfs > snapshots, might make

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 07:11:24PM +0100, Filipe Manana wrote: > On Thu, Jul 13, 2017 at 7:00 PM, Liu Bo wrote: > > On Thu, Jul 13, 2017 at 10:06:32AM -0700, Liu Bo wrote: > >> On Thu, Jul 13, 2017 at 05:46:13PM +0100, Filipe Manana wrote: > >> > On Thu, Jul 13,

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 10:06:32AM -0700, Liu Bo wrote: > On Thu, Jul 13, 2017 at 05:46:13PM +0100, Filipe Manana wrote: > > On Thu, Jul 13, 2017 at 5:36 PM, Liu Bo wrote: > > > On Thu, Jul 13, 2017 at 03:09:54PM +0100, fdman...@kernel.org wrote: > >

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 05:46:13PM +0100, Filipe Manana wrote: > On Thu, Jul 13, 2017 at 5:36 PM, Liu Bo wrote: > > On Thu, Jul 13, 2017 at 03:09:54PM +0100, fdman...@kernel.org wrote: > >> From: Filipe Manana > >> [...] > > > > > >> Signed-off-by:

Re: [PATCH] btrfs: test direct IO write against raid5/6 filesystems

2017-07-13 Thread Liu Bo
" > Thanks for the testcase. Reviewed-by: Liu Bo -liubo > Signed-off-by: Filipe Manana > --- > tests/btrfs/148 | 85 > + > tests/btrfs/148.out | 23 +++ > tests/btrfs/group | 1 + > 3 files ch

Re: BTRFS file I/O with O_DIRECT

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 04:16:44PM +0200, Aleksandar Čekrlić wrote: > Hi all, > > I am currently working on a file system related code where user can > opt to use direct I/O for file reads / writes. Code seems to be > working and operational, and on majority of the file systems I/O is > not using

Re: [PATCH] btrfs: btrfs_create_repair_bio never fails, skip error handling

2017-07-13 Thread Liu Bo
On Thu, Jul 13, 2017 at 05:42:15PM +0200, David Sterba wrote: > As it uses the non-failinig bio allocation, we can remove error handling > from the callers as well. > Looks good. Reviewed-by: Liu Bo -liubo > Signed-off-by: David Sterba > --- > fs/btrfs/extent_io.c |

Re: [PATCH] btrfs: cloned bios must not be iterated by bio_for_each_segment_all

2017-07-13 Thread Liu Bo
cloned bios have wrong bi_vcnt and this could lead to silent > corruptions. This patch adds assertions to all remaining > bio_for_each_segment_all cases. > > [1] https://patchwork.kernel.org/patch/9838535/ > Reviewed-by: Liu Bo -liubo > CC: Liu Bo > Signed-off-by: David S

Re: [PATCH] Btrfs: fix write corruption due to bio cloning on raid5/6

2017-07-13 Thread Liu Bo
_index + i] = bvec->bv_page; > + if (bio_flagged(bio, BIO_CLONED)) > + bio->bi_iter = btrfs_io_bio(bio)->iter; > + I think we can use use bio->bi_iter directly as the bio is not submitted yet, i.e. bi_iter is not advanced yet. &g

Re: [PATCH] Btrfs: report errors when checksum is not found

2017-07-12 Thread Liu Bo
On Wed, Jul 12, 2017 at 11:46:29AM -0600, Liu Bo wrote: > On Wed, Jul 12, 2017 at 04:40:36PM +0200, David Sterba wrote: > > On Tue, Jul 11, 2017 at 02:43:16PM -0600, Liu Bo wrote: > > > When btrfs fails the checksum check, it'll fill the whole page with > > > &qu

Re: [PATCH] Btrfs: report errors when checksum is not found

2017-07-12 Thread Liu Bo
On Wed, Jul 12, 2017 at 04:40:36PM +0200, David Sterba wrote: > On Tue, Jul 11, 2017 at 02:43:16PM -0600, Liu Bo wrote: > > When btrfs fails the checksum check, it'll fill the whole page with > > "1". > > One could ask, why is the page filled with 1s. Brough

[PATCH] Btrfs: report errors when checksum is not found

2017-07-11 Thread Liu Bo
ful but getting a page with all content being "1". This can happen due to a bug in btrfs-convert. This fixes it by always returning errors if checksum doesn't match. Signed-off-by: Liu Bo --- fs/btrfs/inode.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/fs/btrfs/in

Re: [PATCH v2] btrfs: Correct assignment of pos

2017-07-10 Thread Liu Bo
iter(struct kiocb > *iocb, > inode_unlock(inode); > return -EAGAIN; > } > - } else > - inode_lock(inode); > - > - err = generic_write_checks(iocb, from); > - if (err <= 0) { > -

[PATCH] Btrfs-progs: convert: do not clear nodatasum flag in inode item

2017-06-28 Thread Liu Bo
With the current btrfs-convert, if we convert a ext4 without data checksum, it'd not set nodatasum flag in inode item, nor create csum item, reading file ends up with checksum errors. Signed-off-by: Liu Bo --- convert/source-ext2.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

[PATCH] Btrfs-progs: man: nodesize must be power of 2 now

2017-06-27 Thread Liu Bo
This updates mkfs.btrfs's man page with the new limitation that nodesize must be a power of 2 as well. Signed-off-by: Liu Bo --- Documentation/mkfs.btrfs.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Documentation/mkfs.btrfs.asciidoc b/Document

Re: [PATCH] Btrfs-progs: convert: do not clear header rev

2017-06-27 Thread Liu Bo
On Mon, Jun 26, 2017 at 05:55:02PM -0600, Liu Bo wrote: > So btrfs_set_header_flags() vs btrfs_set_header_flag, the difference is sort > of > similar to "=" vs "|=", when creating and initialising a new extent buffer, > convert uses the former one which clears he

[PATCH] Btrfs-progs: convert: do not clear header rev

2017-06-26 Thread Liu Bo
So btrfs_set_header_flags() vs btrfs_set_header_flag, the difference is sort of similar to "=" vs "|=", when creating and initialising a new extent buffer, convert uses the former one which clears header_rev by accident. Signed-off-by: Liu Bo --- convert/common.c | 2 +

[Patch v2] Btrfs-progs: fix infinite loop in find_free_extent

2017-06-26 Thread Liu Bo
nd up with looking at the same block group forever. Cc: David Sterba Signed-off-by: Liu Bo --- v2: - enhance commit log with more details. - fix typo on bg_cache->key.objectid. extent-tree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/extent-tree.c b/extent

Re: [PATCH v2 14/51] btrfs: avoid to access bvec table directly for a cloned bio

2017-06-26 Thread Liu Bo
vec() to retrieve the 1st bvec. > Looks good to me. Reviewed-by: Liu Bo -liubo > Cc: Chris Mason > Cc: Josef Bacik > Cc: David Sterba > Cc: linux-btrfs@vger.kernel.org > Cc: Liu Bo > Signed-off-by: Ming Lei > --- > fs/btrfs/inode.c | 4 +++- > 1 file changed, 3

Re: [PATCH] Btrfs-progs: fix infinite loop in find_free_extent

2017-06-26 Thread Liu Bo
On Mon, Jun 26, 2017 at 04:09:53PM +0200, David Sterba wrote: > On Fri, Jun 23, 2017 at 10:28:31PM -0600, Liu Bo wrote: > > From: Liu Bo Ah, my From was broken again. > > > > %search_start is calculated in a wrong way, and if %ins is a cross-stripe > > one, it

[PATCH] Btrfs-progs: fix infinite loop in find_free_extent

2017-06-23 Thread Liu Bo
From: Liu Bo %search_start is calculated in a wrong way, and if %ins is a cross-stripe one, it'll search the same block group forever. Signed-off-by: Liu Bo --- extent-tree.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/extent-tree.c b/extent-tree.c index b1

[PATCH v3] Btrfs: change how we iterate bios in endio

2017-06-12 Thread Liu Bo
segments to access each bvec in dio read's endio. Note that it doesn't affect other calls of btrfs_bio_clone() because they don't need to use this iterator. Cc: David Sterba Signed-off-by: Liu Bo --- v2: Fix null pointer crash in the case of non-split dio reads. fs/btrfs/extent_i

Re: [PATCH 3/3] btrfs: sink gfp parameter to btrfs_io_bio_alloc

2017-06-12 Thread Liu Bo
contexts and the bio allocating API now looks more consistent. > Reviewed-by: Liu Bo -liubo > Signed-off-by: David Sterba > --- > fs/btrfs/check-integrity.c | 2 +- > fs/btrfs/disk-io.c | 2 +- > fs/btrfs/extent_io.c | 8 > fs/btrfs/extent_i

Re: [PATCH 3/7] Btrfs: update total_bytes_pinned when pinning down extents

2017-06-12 Thread Liu Bo
wing it. > Also adjust btrfs_free_tree_block() to avoid accounting for the same > extent twice. > Reviewed-by: Liu Bo -liubo > Signed-off-by: Omar Sandoval > --- > fs/btrfs/extent-tree.c | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/fs/bt

Re: [PATCH 1/7] Btrfs: make add_pinned_bytes() take an s64 num_bytes instead of u64

2017-06-12 Thread Liu Bo
On Tue, Jun 06, 2017 at 04:45:26PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > There are a few places where we pass in a negative num_bytes, so make it > signed for clarity. Also move it up in the file since later patches will > need it there. > Reviewed-by: Liu Bo

Re: [PATCH 2/3] btrfs: add helper to initialize the non-bio part of btrfs_io_bio

2017-06-12 Thread Liu Bo
tialized. Although we initialize some of the members in our > helpers, we should initialize the whole structures. > Reviewed-by: Liu Bo -liubo > Signed-off-by: David Sterba > --- > fs/btrfs/extent_io.c | 30 ++ > 1 file changed, 14 inserti

Re: mount gets stuck - BUG: soft lockup

2017-06-07 Thread Liu Bo
On Wed, Jun 07, 2017 at 09:44:41AM +, Thomas Mischke wrote: > Hello, > > i tried to convert a JBOD BTRFS consisting of 5 disks (6TB each) to raid10 > (converting from an earlier configuration). > All disk were backed by bcache. > > Because a rebalance takes very long I had to pause the balan

Re: [PATCH 7/7] Btrfs: warn if total_bytes_pinned is non-zero on unmount

2017-06-07 Thread Liu Bo
On Tue, Jun 06, 2017 at 04:45:32PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > Catch any future/remaining leaks or underflows of total_bytes_pinned. > This might be a little bit late, what about checking it after btrfs_finish_extetn_commit()? -liubo > Signed-off-by: Omar Sandoval >

Re: [PATCH 4/7] Btrfs: always account pinned bytes when dropping a tree block ref

2017-06-07 Thread Liu Bo
l_bytes_pinned an overestimate when references to shared extents are > dropped, but in the worst case this will just make us try to commit the > transaction to try to free up space and find we didn't free enough. > I recognize the same problem, the patch looks good. Reviewed-by: Li

Re: [PATCH 6/7] Btrfs: rework delayed ref total_bytes_pinned accounting

2017-06-07 Thread Liu Bo
t goes back to non-negative or after we've run all of the > delayed refs for that extent. > The patch could be cleaner if we inc/dec %pinned inside delayed_ref.c. The idea looks good to me. Reviewed-by: Liu Bo -liubo > Signed-off-by: Omar Sandoval > --- > fs/btrfs/extent-tree

Re: [PATCH 5/7] Btrfs: return old and new total ref mods when adding delayed refs

2017-06-07 Thread Liu Bo
On Tue, Jun 06, 2017 at 04:45:30PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > We need this to decide when to account pinned bytes. > > Signed-off-by: Omar Sandoval > --- > fs/btrfs/delayed-ref.c | 29 > fs/btrfs/delayed-ref.h | 6 -- > fs/btrfs/exte

[PATCH] Btrfs: fix Null pointer dereference in dio read endio

2017-06-06 Thread Liu Bo
ned-off-by: Liu Bo --- Based on David's for-next. Fixes: commit "Btrfs: change how we iterate bios in endio" Have run through fstests without introducing new problems. fs/btrfs/extent_io.c | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io

<    1   2   3   4   5   6   7   8   9   10   >