Re: bio linked list corruption.

> Dave, can you hit the warnings with this? Totally untested... Can we just kill off the unhelpful blk_map_ctx structure, e.g.: diff --git a/block/blk-mq.c b/block/blk-mq.c index ddc2eed..d74a74a 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1190,21 +1190,15 @@ static inline bool

[PATCH] btrfs: don't abuse REQ_OP_* flags for btrfs_map_block

btrfs_map_block supports different types of mappings, which to a large extent resemble block layer operations. But they don't always do, and currently btrfs dangerously overlays it's own flag over the block layer flags. This is just asking for a conflict, so introduce a different map flags enum

[PATCH v3] btrfs: change btrfs_csum_final result param type to u8

csum member of struct btrfs_super_block has array type of u8. It makes sense that function btrfs_csum_final should be also declared to accept u8 *. I changed the declaration of method void btrfs_csum_final(u32 crc, char *result); to void btrfs_csum_final(u32 crc, u8 *result); Signed-off-by:

Re: Compression and device replace on raid10 kernel panic on 4.4.6 and 4.6.x

Hi, Le 27/10/2016 à 02:50, Lionel Bouton a écrit : > [...] > I'll stop for tonight and see what happens during the day. I'd like to > try a device add / delete next but I'm worried I could end up with a > completely unusable filesystem if the device delete hits the same > problem than replace. >

Re: [PATCH v3] generic: make 17[1-4] work well when btrfs compression is enabled

On Wed, Oct 26, 2016 at 05:52:11PM +0800, Wang Xiaoguang wrote: > When enabling btrfs compression, original codes can not fill fs > correctly, here we introduce _fill_fs() in common/rc, which'll keep > creating and writing files until enospc error occurs. Note _fill_fs > is copied from

Re: [PATCH v2] btrfs: change btrfs_csum_final result param type to u8

On 2016-10-24 19:05 +0200, David Sterba wrote: > > Ok that's better, just the signed-off-by line is missing. I would add it > myself on behalf of contributors I know, but as a matter of practice, > fix it please and resend the patch. Using the "git commit -s" adds the > line for you. > I added

Re: [PATCH] btrfs-progs: Ignore clang complete file

On Thu, Oct 27, 2016 at 10:32:48AM +0800, Qu Wenruo wrote: > While most guys are using ctags and cscope with vim, new completion tool > like vim-clang_completion is gaining its popularity, due to its compiler > level accuracy simpleness to use. > > Since ctags and cscope are already in

[PATCH] btrfs-progs: repair: Trickle down EEXISTS while freeing

While deleting pending extents, we insert extents to the extent_tree. However, because of corruption, this might already be freed. We trickle the EEXISTS error and let the caller decide what needs to be done with it. In this case, ignore, so the repair may proceed. The primary motivation of this

[PATCH] btrfs: fix races on root_log_ctx lists

btrfs_remove_all_log_ctxs takes a shortcut where it avoids walking the list because it knows all of the waiters are patiently waiting for the commit to finish. But, there's a small race where btrfs_sync_log can remove itself from the list if it finds a log commit is already done. Also, it uses

[PATCH v2 1/6] Btrfs: incremental send, do not skip generation inconsistency check for inode 256.

From: Robbie Ko Example scenario: Parent snapshot: |.(ino 256, gen 5) | a1/ (ino 257, gen 5) | a2/ (ino 258, gen 5) Send snapshot: |.(ino 256, gen 7) | a2 (ino 257, gen 7) rmdir a1 mkfile o257-7-0

[PATCH v2 6/6] Btrfs: incremental send, add generation check in existence demtermination for the parent directory

From: Robbie Ko Exampla scenario: Parent snapshot: | dir258/ (ino 258, gen 7, dir) |--- dir257/ (ino 257, gen 7, dir) | dir259/ (ino 259, gen 7, dir) Send snapshot: | file258 (ino 258, gen 10, file) | new_dir259/ (ino

[PATCH v2 3/6] Btrfs: incremental send, add generation for waiting_dir_move struct and check it in can_rmdir.

From: Robbie Ko Example scenario: Parent snapshot: | dir258/ (ino 258, gen 27) | dir257/ (ino 257, gen 27) | dir259/ (ino 259, gen 27) Send snapshot: | new_dir259/ (ino 259, gen 38) | new_dir258/ (ino 258, gen 38) | new_dir257/ (ino

Re: [PATCH 0/4] Qgroup comment enhance and balance fix

Any comment? Especially the final patch will fix a long standing bug. Thanks, Qu At 10/18/2016 09:31 AM, Qu Wenruo wrote: The patchset does the following things: 1) Enhance comment for qgroup, rename 2 functions Explain the how qgroup works, so new developers won't waste too much time

[PATCH v2 0/4] Btrfs: add serval test case for incremental send

From: Robbie Ko Patch for test btrfs incremental send. v2: * Improve the change log Robbie Ko (4): xfstests: btrfs/131: add test for an incremental send with name collision xfstests: btrfs/132: add test for invaild update time by an incremental send

[PATCH v2 4/4] xfstests: btrfs/134: add test for incremental send which renames a directory already being deleted

From: Robbie Ko Test that an incremental send operation dosen't work because it tries to rename a directory which is already deleted. This test exercises scenarios used to fail in btrfs and are fixed by the following patch for the linux kernel: "Btrfs: incremental send,

[PATCH v2 1/4] xfstests: btrfs/131: add test for an incremental send with name collision

From: Robbie Ko Test that an incremental send operation doesn't work because there's a name collision in the destination and it's not checked corretly before the rename operation applies. This test exercises scenarios used to fail in btrfs and are fixed by the following

[PATCH v2 2/4] xfstests: btrfs/132: add test for invaild update time by an incremental send

From: Robbie Ko Test that an incremental send operation dosen't' work because it tries to update the time to a deleted directory after it finishes a move operation. The other one is that an operation is applied to a file using the old name not the new name. This test

[PATCH v2 5/6] Btrfs: incremental send, add generation check for inode is waiting for move.

From: Robbie Ko Example scenario: Parent snapshot: | d1/ (ino 257, gen 44) | d4/ (ino 258, gen 44) | d3/ (ino 259, gen 44) Send snapshot: | d1/ (ino 258, gen 47) | d4/ (ino 260, gen 47)

[PATCH v2 4/6] Btrfs: incremental send, skip check overwritten if parents' generation are different

From: Robbie Ko Example scenario: Parent snapshot: | d259_old/ (ino 259, gen 96) | d1/ (ino 258, gen 96) | f (ino 257, gen 96) Send snapshot: | d258/ (ino 258, gen 98) | d259/ (ino 259, gen

[PATCH v2 0/6] Btrfs: incremental send, fix serval case for inode 256 and generation

From: Robbie Ko Patch for fix btrfs incremental send. These patches base on v4.8.0-rc8 v2: * Improve the change log * Add a new patch "add generation check in existence demtermination for the parent directory" Robbie Ko (6): Btrfs: incremental send, do not skip

[PATCH v2 2/6] Btrfs: incremental send, add generation check for the inode waiting for rmdir operation.

From: Robbie Ko Example scenario: Parent snapshot: | dir258/ (ino 258, gen 15, dir) |--- dir257/ (ino 257, gen 15, dir) | dir259/ (ino 259, gen 15, dir) Send snapshot: | file258 (ino 258, gen 21, file) | new_dir259/ (ino 259, gen 21, dir) |---

It takes me up to 20 reboots for the system to start sda

Folks, I have a btrfs raid 10 spread out among 4 ssds. The main drive is sda. When i reboot my system after the kernel loads i see "a start job is running for dev-sda1.device. I've let this sit for 3 hours with no progress into the boot, and it seems to be a race condition of some sort. If i

[PATCH v2 3/4] xfstests: btrfs/133: add test for incremental send with rmdir applied on wrong name

From: Robbie Ko Test that an incremental send operation doesn't work for rmdir because it uses the wrong name to delete. This test exercises scenarios used to fail in btrfs and are fixed by the following patch for the linux kernel: "Btrfs: incremental send, skip check

Re: [PATCH v3] generic: make 17[1-4] work well when btrfs compression is enabled

On Wed, Oct 26, 2016 at 05:52:11PM +0800, Wang Xiaoguang wrote: > When enabling btrfs compression, original codes can not fill fs > correctly, here we introduce _fill_fs() in common/rc, which'll keep > creating and writing files until enospc error occurs. Note _fill_fs > is copied from

Re: [PATCH V2] Btrfs-progs: subvol_uuid_search: Return error code on memory allocation failure

On Mon, Oct 17, 2016 at 10:07:17AM +0530, Prasanth K S R wrote: > From: Prasanth K S R > > This commit fixes coverity defect CID 1328695. The patch does more than that. Please split it into two, one that handles the memory allocations (the error message is not needed) and

Re: It takes me up to 20 reboots for the system to start sda

Am Thu, 27 Oct 2016 19:17:52 -0600 schrieb Bearcat Şándor : > Folks, > > I have a btrfs raid 10 spread out among 4 ssds. The main drive is > sda. When i reboot my system after the kernel loads i see "a start job > is running for dev-sda1.device. I've let this sit for 3

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

On 10/26/2016 08:30 PM, Dave Chinner wrote: On Wed, Oct 26, 2016 at 11:11:35AM -0400, Josef Bacik wrote: On 10/25/2016 06:01 PM, Dave Chinner wrote: On Tue, Oct 25, 2016 at 02:41:44PM -0400, Josef Bacik wrote: With anything that populates the inode/dentry cache with a lot of one time use

Re: bio linked list corruption.

On 10/26/2016 08:00 PM, Jens Axboe wrote: > On 10/26/2016 05:47 PM, Dave Jones wrote: >> On Wed, Oct 26, 2016 at 07:38:08PM -0400, Chris Mason wrote: >> >> > >-hctx->queued++; >> > >-data->hctx = hctx; >> > >-data->ctx = ctx; >> > >+data->hctx = alloc_data.hctx; >> > >+

Identifying reflink / CoW files

Hello, Following the previous discussion: https://www.spinics.net/lists/linux-btrfs/msg19075.html I would be interested in finding a way to reliably identify reflink / CoW files in order to use deduplication programs (like fdupes, jdupes, rmlint) efficiently. Using FIEMAP doesn't seem to be

[PATCH 02/19] btrfs-progs: raid56: Introduce tables for RAID6 recovery

Copied from kernel code with minor modification. Signed-off-by: Qu Wenruo --- .gitignore | 2 + Makefile.in | 12 - mktables.c | 142 raid56.h| 7 +++ 4 files changed, 161 insertions(+), 2

[PATCH 16/19] btrfs-progs: check/scrub: Introduce function to recover data parity

Introduce function, recover_from_parities(), to recover data stripes. However this function only support RAID5 yet, but should be good enough for the scrub framework. Signed-off-by: Qu Wenruo --- check/scrub.c | 49 + 1

[PATCH 01/19] btrfs-progs: raid56: Introduce raid56 header for later recovery usage

Although there is only 2 function in raid56 header, later recovery will introduce a lot of tables, so split it from disk-io.h. Signed-off-by: Qu Wenruo --- Makefile.in | 2 +- disk-io.h | 4 raid56.h| 20 volumes.c | 1 + 4 files

[PATCH 15/19] btrfs-progs: extent-tree: Introduce function to check if there is any extent in given range.

Will be used for later scrub usage. Signed-off-by: Qu Wenruo --- ctree.h | 2 ++ extent-tree.c | 52 2 files changed, 54 insertions(+) diff --git a/ctree.h b/ctree.h index e0dd2e4..b511ff2 100644 --- a/ctree.h

[PATCH 12/19] btrfs-progs: check/scrub: Introduce function to scrub one extent

Introduce a new function, scrub_one_extent(), as a wrapper to check one extent. Signed-off-by: Qu Wenruo --- check/scrub.c | 76 +++ 1 file changed, 76 insertions(+) diff --git a/check/scrub.c b/check/scrub.c

[PATCH] Btrfs: fix fsync deadlock in log_new_dir_dentries

From: Robbie Ko We found a fsync deadlock in log_new_dir_dentries, because btrfs_search_forward get path lock, then call btrfs_iget will get another extent_buffer lock, maybe occur deadlock. Fix this by release path before call btrfs_iget, avoid deadlock occur. Example:

[PATCH 08/19] btrfs-progs: check/csum: Introduce function to read out one data csum

Introduce a new function, btrfs_read_one_data_csum(), to read just one data csum for check usage. Unlike original implement in cmds-check.c which checks csum by one CSUM_EXTENT, this just read out one csum(4 bytes). It is not fast but makes code easier to read. And will be used in later fsck

Re: [PATCH 5/5] fs: don't set *REFERENCED unless we are on the lru list

On Thu, Oct 27, 2016 at 09:13:44AM -0400, Josef Bacik wrote: > On 10/26/2016 08:30 PM, Dave Chinner wrote: > >On Wed, Oct 26, 2016 at 11:11:35AM -0400, Josef Bacik wrote: > >>On 10/25/2016 06:01 PM, Dave Chinner wrote: > >>>On Tue, Oct 25, 2016 at 02:41:44PM -0400, Josef Bacik wrote: > With

[PATCH 04/19] btrfs-progs: raid56: Allow raid6 to recover data and p

Copied from kernel raid6 lib, with some naming modification. Signed-off-by: Qu Wenruo --- raid56.c | 36 raid56.h | 3 +++ 2 files changed, 39 insertions(+) diff --git a/raid56.c b/raid56.c index 599533e..3e471d6 100644 ---

[PATCH 10/19] btrfs-progs: check/scrub: Introduce function to scrub mirror based tree block

Introduce a new function, scrub_tree_mirror(), to scrub mirror based tree blocks (Single/DUP/RAID0/1/10) This function can be used on in-memory tree blocks using @data parameter for RAID5/6 full stripe, or just by @bytenr, for other profiles. Signed-off-by: Qu Wenruo

[PATCH 05/19] btrfs-progs: Introduce wrapper to recover raid56 data

Signed-off-by: Qu Wenruo --- raid56.c | 76 raid56.h | 13 +++ 2 files changed, 89 insertions(+) diff --git a/raid56.c b/raid56.c index 3e471d6..8d54e55 100644 --- a/raid56.c +++ b/raid56.c @@

[PATCH 09/19] btrfs-progs: check/scrub: Introduce structures to support fsck scrub

Introuduce new local structures, scrub_full_stripe and scrub_stripe, for incoming offline scrub support. Signed-off-by: Qu Wenruo --- Makefile.in | 2 +- check/scrub.c | 110 ++ 2 files changed, 111

[PATCH 03/19] btrfs-progs: raid56: Allow raid6 to recover 2 data stripes

Mainly copied from kernel code, with minor modification to follow the new naming schema. Signed-off-by: Qu Wenruo --- raid56.c | 63 +++ raid56.h | 4 2 files changed, 67 insertions(+) diff --git

[PATCH 13/19] btrfs-progs: check/scrub: Introduce function to scrub one data stripe

Introduce new function, scrub_one_data_stripe(), to check all data and tree blocks inside the data stripe. Signed-off-by: Qu Wenruo --- check/scrub.c | 117 ++ 1 file changed, 117 insertions(+) diff --git

[PATCH 00/19] Offline scrub support

For any one who wants to try it, it can be get from my repo: https://github.com/adam900710/btrfs-progs/tree/fsck_scrub Currently, I only tested it on SINGLE/DUP/RAID1/RAID5 filesystems, with mirror or parity or data corrupted. The tool are all able to detect them and give recoverbility report.

[PATCH 18/19] btrfs-progs: check/scrub: Introduce function to check a whole block group

Introduce new function, scrub_one_block_group(), to scrub a block group. For Single/DUP/RAID0/RAID1/RAID10, we use old mirror number based map_block, and check extent by extent. For parity based profile (RAID5/6), we use new map_block_v2() and check full stripe by full stripe. Signed-off-by: Qu

[PATCH v2] Btrfs: fix enospc in hole punching

From: Robbie Ko The hole punching can result in adding new leafs (and as a consequence new nodes) to the tree because when we find file extent items that span beyond the hole range we may end up not deleting them (just adjusting them) and add new file extent items

[PATCH 17/19] btrfs-progs: check/scrub: Introduce a function to scrub one full stripe

Introduce a new function, scrub_one_full_stripe(), to check a full stripe. It handles the full stripe scrub in the following steps: 0) Check if we need to check full stripe If full stripe contains no extent, why waste our CPU and IO? 1) Read out full stripe Then we know how many devices

[PATCH 07/19] btrfs-progs: Allow __btrfs_map_block_v2 to remove unrelated stripes

For READ, caller normally hopes to get what they request, other than full stripe map. In this case, we should remove unrelated stripe map, just like the following case: 32K 96K |<-request range->| 0 64k 128K RAID0: |

[PATCH 19/19] btrfs-progs: fsck: Introduce offline scrub function

Now, btrfs check has a kernel scrub equivalent. And even more, it's has stronger csum check against reconstructed data and existing data stripes. It will avoid any possible silent data corruption in kernel scrub. Now it only supports to do read-only check, but is already able to provide info on

[PATCH 06/19] btrfs-progs: Introduce new btrfs_map_block function which returns more unified result.

Introduce a new function, __btrfs_map_block_v2(). Unlike old btrfs_map_block(), which needs different parameter to handle different RAID profile, this new function uses unified btrfs_map_block structure to handle all RAID profile in a more meaningful method: Return physical address along with

[PATCH 11/19] btrfs-progs: check/scrub: Introduce function to scrub mirror based data blocks

Introduce a new function, scrub_data_mirror(), to check mirror based data blocks. Signed-off-by: Qu Wenruo --- check/scrub.c | 69 +++ 1 file changed, 69 insertions(+) diff --git a/check/scrub.c b/check/scrub.c

[PATCH 14/19] btrfs-progs: check/scrub: Introduce function to verify parities

Introduce new function, verify_parities(), to check if parities matches for full stripe which all data stripes matches with their csum. Signed-off-by: Qu Wenruo --- check/scrub.c | 66 +++ 1 file changed, 66

Re: [PATCH] btrfs-progs: fix build for programs including ioctl.h

On 10/13/16 12:36 PM, Eric Sandeen wrote: > This was reported when docker failed to build against > btrfs-progs v4.8.1. > > It includes ioctl.h which now calls BUILD_ASSERT(), which > is defined in kerncompat.h, which was not included in the > ioctl.h header file. Ping? -Eric > Signed-off-by:

Re: bio linked list corruption.

On Wed, Oct 26, 2016 at 11:33 PM, Christoph Hellwig wrote: >> Dave, can you hit the warnings with this? Totally untested... > > Can we just kill off the unhelpful blk_map_ctx structure, e.g.: Yeah, I found that hard to read too. The difference between blk_map_ctx and

Re: bio linked list corruption.

On Thu, Oct 27, 2016 at 04:41:33PM +1100, Dave Chinner wrote: > And that's indicative of a delalloc metadata reservation being > being too small and so we're allocating unreserved blocks. > > Different symptoms, same underlying cause, I think. > > I see the latter assert from time to