Re: [RFC PATCH 00/17] btrfs: implementation of priority aware allocator

2018-12-01 Thread Su Yue
On 2018/11/28 12:04 PM, Qu Wenruo wrote: On 2018/11/28 上午11:11, Su Yue wrote: This patchset can be fetched from repo: https://github.com/Damenly/btrfs-devel/commits/priority_aware_allocator. Since patchset 'btrfs: Refactor find_free_extent()' does a nice work to simplify find_free_extent

Re: [RFC PATCH 01/17] btrfs: priority alloc: prepare of priority aware allocator

2018-11-28 Thread Su Yue
On 11/28/18 4:24 PM, Nikolay Borisov wrote: On 28.11.18 г. 5:11 ч., Su Yue wrote: To implement priority aware allocator, this patch: Introduces struct btrfs_priority_tree which contains block groups in same level. Adds member priority to struct btrfs_block_group_cache and pointer points

[PATCH] btrfs: adjust order of unlocks in do_trimming()

2018-11-27 Thread Su Yue
In function do_trimming(), block_group->lock should be unlocked first. Fixes: 7fe1e6415026 ("Btrfs: rewrite btrfs_trim_block_group()") Signed-off-by: Su Yue --- fs/btrfs/free-space-cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/free-space-cache

[RFC PATCH 12/17] btrfs: priority alloc: introduce find_free_extent_search()

2018-11-27 Thread Su Yue
In origin, find_free_extent() just searches block groups in space_info one by one. In priority aware allocator, we first search block groups in higher priority tree than in lower priority tree. This helper unify above two ways for further use. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c

[RFC PATCH 16/17] btrfs: priority alloc: write bg->priority_tree->groups_sem to avoid race in btrfs_delete_unused_bgs()

2018-11-27 Thread Su Yue
If use priority aware allocator, bg->priority_tree->groups_sem should be written instead of space_info->groups_sem. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 60 +++--- 1 file changed, 44 insertions(+), 16 deletions(-) diff --git a/fs/btr

[RFC PATCH 08/17] btrfs: priority alloc: add btrfs_update_block_group_priority()

2018-11-27 Thread Su Yue
Introduce btrfs_update_block_group_priority() to update block_groups::priority. It will move block group from old tree to new tree if need. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 76 ++ 1 file changed, 76 insertions(+) diff --git a/fs/btrfs

[RFC PATCH 04/17] btrfs: priority alloc: add functions to create/remove priority trees

2018-11-27 Thread Su Yue
Introduce create_priority_trees() to create priority trees in space_info. Introduce remove_priority_trees() to remove priority trees in space_info. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 94 ++ 1 file changed, 94 insertions(+) diff --git a/fs

[RFC PATCH 17/17] btrfs: add mount option "priority_alloc=%s"

2018-11-27 Thread Su Yue
Add mount option "priority_alloc=%s", now %s only supports "usage" and "off". The latter is used for remount. "priority_alloc=usage" will active priority aware allocator. This mount option changes the way of find_free_extent to search block groups and ma

[RFC PATCH 13/17] btrfs: priority alloc: modify find_free_extent() to fit priority allocator

2018-11-27 Thread Su Yue
Add member priority_tree to find_free_extent_ctl to represents the tree using. Modify find_free_extent to use find_free_extent_search, so it can work in default mount option and priorit aware allocator. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 114

[RFC PATCH 06/17] btrfs: priority alloc: introduce three macros to mark block group status

2018-11-27 Thread Su Yue
... unlock block_group up_read tree This patch introduce three macros to represents block group is removing /need to updated / busy. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 9 + 1 file changed, 9 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c

[RFC PATCH 03/17] btrfs: priority alloc: introduce compute_block_group_priority/usage

2018-11-27 Thread Su Yue
Introduce compute_block_group_usage() and compute_block_group_usage(). And call the latter in btrfs_make_block_group() and btrfs_read_block_groups(). compute_priority_level use ilog2(free) to compute priority level. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 60

[RFC PATCH 11/17] btrfs: priority alloc: remove block group from priority tree while removing block group

2018-11-27 Thread Su Yue
Export btrfs_remove_block_group_priority() to header ctree.h. Call btrfs_remove_block_group_priority() while deleting transaction->deleted_bgs, btrfs_free_block_groups() and btrfs_remove_block_group(). Signed-off-by: Su Yue --- fs/btrfs/ctree.h | 1 + fs/btrfs/extent-tree.c | 3 +++

[RFC PATCH 02/17] btrfs: add mount definition BTRFS_MOUNT_PRIORITY_USAGE

2018-11-27 Thread Su Yue
Now implementation of priority allocator only support usage option. Add BTRFS_MOUNT_PRIORITY_USAGE for further commits. Signed-off-by: Su Yue --- fs/btrfs/ctree.h | 1 + 1 file changed, 1 insertion(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 5c4651d8a524..4c56baf9f7cf 100644

[RFC PATCH 15/17] btrfs: priority alloc: write bg->priority_groups_sem while waiting reservation

2018-11-27 Thread Su Yue
Since if use priority alloc, we should down/up_write() bg->priority_groups_sem. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index f530a4344368..6627bbe56ad5 100

[RFC PATCH 10/17] btrfs: priority alloc: call add_block_group_priority while reading or making block group

2018-11-27 Thread Su Yue
Add block group to priority tree in btrfs_read_block_groups() and btrfs_make_block_groups(). Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 2dec02782df1..fc40901b4772 100644 --- a/fs

[RFC PATCH 14/17] btrfs: priority alloc: introduce btrfs_set_bg_updating and call btrfs_update_block_group_prioriy

2018-11-27 Thread Su Yue
oup() to update priority tree if needed. Signed-off-by: Su Yue --- fs/btrfs/ctree.h| 2 ++ fs/btrfs/extent-tree.c | 40 + fs/btrfs/free-space-cache.c | 3 +++ 3 files changed, 45 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctre

[RFC PATCH 05/17] btrfs: priority alloc: introduce functions to add block group to priority tree

2018-11-27 Thread Su Yue
Introduce compute_priority_level() to compute priority level according priority, now just divides PRIORITY_USAGE_FACOTR. Introduce add_block_group_priority() to add block groups to priority tree. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 76

[RFC PATCH 07/17] btrfs: priority alloc: add functions to remove block group from priority tree

2018-11-27 Thread Su Yue
Introduce btrfs_remove_block_group_priority() to remove block group from priority tree. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 37 + 1 file changed, 37 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 5bae757786dc

[RFC PATCH 09/17] btrfs: priority alloc: call create/remove_priority_trees in space_info

2018-11-27 Thread Su Yue
Call create_priority_trees() in create_space_info(). Call remove_priority_trees() before free of space_info. Signed-off-by: Su Yue --- fs/btrfs/extent-tree.c | 21 + 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent

[RFC PATCH 01/17] btrfs: priority alloc: prepare of priority aware allocator

2018-11-27 Thread Su Yue
to represents priority trees in different raid types. Signed-off-by: Su Yue --- fs/btrfs/ctree.h | 24 1 file changed, 24 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index e62824cae00a..5c4651d8a524 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h

[RFC PATCH 00/17] btrfs: implementation of priority aware allocator

2018-11-27 Thread Su Yue
es time and hardware, the use-case is not outstanding enough. And some codes are dirty but I can't found another way. So I named it as RFC. Any comments and suggestions are welcome. Su Yue (17): btrfs: priority alloc: prepare of priority aware allocator btrfs: add mount definition BTRFS_

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-11-04 Thread Su Yue
On 11/3/18 5:20 PM, Nikolay Borisov wrote: On 3.11.18 г. 3:34 ч., Su Yue wrote: On 2018/11/2 10:10 PM, Christoph Anton Mitterer wrote: Hey Su. Sorry for the late reply cause I'm busy at other things. Anything further I need to do in this matter or can I consider it "solved&quo

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-11-02 Thread Su Yue
On 2018/11/2 10:10 PM, Christoph Anton Mitterer wrote: Hey Su. Sorry for the late reply cause I'm busy at other things. Anything further I need to do in this matter or can I consider it "solved" and you won't need further testing by my side, but just PR the patches of that branch? :-)

Re: [PATCH 01/13] btrfs-progs: lowmem: add argument path to punch_extent_hole()

2018-10-23 Thread Su Yue
On 10/23/18 6:04 PM, Qu Wenruo wrote: On 2018/10/23 下午5:41, Su Yue wrote: Since repair will do CoW, the outer path may be invalid, add an argument path to punch_extent_hole(). When punch_extent_hole() returns, path will still point to the item before calling punch_extent_hole(); Signed

Re: [PATCH 03/13] btrfs-progs: lowmem: fix false alert if extent item has been repaired

2018-10-23 Thread Su Yue
On 10/23/18 6:30 PM, Qu Wenruo wrote: On 2018/10/23 下午5:41, Su Yue wrote: Previously, @err are assigned immediately after check but before repair. repair_extent_item()'s return value also confuses the caller. If error has been repaired and returns 0, check_extent_item() will try

[PATCH 13/13] btrfs-progs: fsck-test: enable lowmem repair for case 001

2018-10-23 Thread Su Yue
Lowmem can repair after commit 'btrfs-progs: lowmem: move nbytes check before isize check', so add the beacon file. Signed-off-by: Su Yue --- tests/fsck-tests/001-bad-file-extent-bytenr/.lowmem_repairable | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests

[PATCH 08/13] btrfs-progs: Revert "btrfs-progs: Add repair and report function for orphan file extent."

2018-10-23 Thread Su Yue
From: Su Yanjun The reason for revert is that according to the existing situation, the probability of problem in the extent tree is higher than in the fs tree. So this feature should be removed. Signed-off-by: Su Yanjun --- check/main.c | 103

[PATCH 11/13] btrfs-progs: check: Delete file extent item with unaligned extent backref

2018-10-23 Thread Su Yue
From: Su Yanjun In original mode, if some file extent item has unaligned extent backref, fixup_extent_refs can't repair it. This patch will check extent alignment then delete file extent with unaligned extent backref. Signed-off-by: Su Yanjun --- check/main.c | 278

[PATCH 10/13] btrfs-progs: check: fix bug in find_possible_backrefs

2018-10-23 Thread Su Yue
From: Su Yanjun It may cost more time to search all extent data of correspond files but should not influence total speed too much cause that only corrupted extent items are participated in. Signed-off-by: Su Yanjun --- check/main.c | 110 ++- 1

[PATCH 12/13] btrfs-progs: tests: add case for inode lose one file extent

2018-10-23 Thread Su Yue
From: Lu Fengqi The missing extent will lead to the existence of the gap between adjacent extents. The fsck should can detect the gap correctly and repair by punch a hole. Signed-off-by: Lu Fengqi --- .../.lowmem_repairable | 0

[PATCH 02/13] btrfs-progs: lowmem: move nbytes check before isize check

2018-10-23 Thread Su Yue
For files, lowmem repair will try to check nbytes and isize, but isize check depends nbytes. Once bytes has been repaired, then isize should be checked and repaired. So move nbytes check before isize check. Also set nbytes to extent_size once repaired successfully. Signed-off-by: Su Yue

[PATCH 06/13] btrfs-progs: lowmem: rename delete_extent_tree_item() to delete_item()

2018-10-23 Thread Su Yue
The function can delete items in trees besides extent tree. Rename and move it for further use. Signed-off-by: Su Yue --- check/mode-lowmem.c | 99 +++-- 1 file changed, 51 insertions(+), 48 deletions(-) diff --git a/check/mode-lowmem.c b/check/mode

[PATCH 09/13] btrfs-progs: Revert "btrfs-progs: Record orphan data extent ref to corresponding root."

2018-10-23 Thread Su Yue
From: Su Yanjun The reason for revert is that according to the existing situation, the probability of problem in the extent tree is higher than in the fs Tree. So this feature should be removed. Signed-off-by: Su Yanjun --- check/main.c | 120

[PATCH 04/13] btrfs-progs: lowmem: fix false alert about the existence of gaps in the check_file_extent

2018-10-23 Thread Su Yue
From: Lu Fengqi The 'end' parameter of check_file_extent tracks the ending offset of the last checked extent. This is used to detect gaps between adjacent extents. Currently such gaps are wrongly detected since for regular extents only the size of the extent is added to the 'end' parameter.

[PATCH 01/13] btrfs-progs: lowmem: add argument path to punch_extent_hole()

2018-10-23 Thread Su Yue
Since repair will do CoW, the outer path may be invalid, add an argument path to punch_extent_hole(). When punch_extent_hole() returns, path will still point to the item before calling punch_extent_hole(); Signed-off-by: Su Yue --- check/mode-lowmem.c | 21 - 1 file changed

[PATCH 05/13] btrfs-progs: lowmem: check unaligned disk_bytenr for extent_data

2018-10-23 Thread Su Yue
Add support to check unaligned disk_bytenr for extent_data. Signed-off-by: Su Yue --- check/mode-lowmem.c | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index 35fe1adf58e6..e8a2e825c0f3 100644 --- a/check/mode-lowmem.c +++ b

[PATCH 03/13] btrfs-progs: lowmem: fix false alert if extent item has been repaired

2018-10-23 Thread Su Yue
return codes only represents status of the extent item, error bits are passed by pointer. Move the change of @err after repair. Signed-off-by: Su Yue --- check/mode-lowmem.c | 106 1 file changed, 68 insertions(+), 38 deletions(-) diff --git a/check

[PATCH 00/13] btrfs-progs: fixes of file extent in original and lowmem check

2018-10-23 Thread Su Yue
epair and report function for orphan file extent." btrfs-progs: Revert "btrfs-progs: Record orphan data extent ref to corresponding root." btrfs-progs: check: fix bug in find_possible_backrefs btrfs-progs: check: Delete file extent item with unaligned extent backref Su Yu

[PATCH 07/13] btrfs-progs: lowmem: delete unaligned bytes extent data under repair

2018-10-23 Thread Su Yue
If found a extent data item has unaligned part, lowmem repair just deletes it. Signed-off-by: Su Yue --- check/mode-lowmem.c | 8 1 file changed, 8 insertions(+) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index 3c9ecff7e498..5381096fa8b2 100644 --- a/check/mode-lowmem.c

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-10-18 Thread Su Yue
[Bad format in previous reply, send again] On 10/18/18 10:41 PM, Christoph Anton Mitterer wrote: Hey. So I'm back from a longer vacation and had now the time to try out your patches from below: On Wed, 2018-09-05 at 15:04 +0800, Su Yue wrote: I found the errors should blame to something

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-10-18 Thread Su Yue
On 10/18/18 10:41 PM, Christoph Anton Mitterer wrote: Hey. So I'm back from a longer vacation and had now the time to try out your patches from below: On Wed, 2018-09-05 at 15:04 +0800, Su Yue wrote: I found the errors should blame to something about inode_extref check in lowmem mode. I

Re: [PATCH v3 1/4] btrfs: Introduce find_free_extent_ctl structure for later rework

2018-10-12 Thread Su Yue
during find_free_extent() refactor. Signed-off-by: Qu Wenruo Reviewed-by: Su Yue --- fs/btrfs/extent-tree.c | 253 ++--- 1 file changed, 161 insertions(+), 92 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index de6f75f5547b

Re: [PATCH v2 4/4] btrfs: Refactor find_free_extent() loops update into find_free_extent_update_loop()

2018-10-10 Thread Su Yue
in that block group | |- Check if the result is valid | | |- If valid, then exit | |- Jump to next block group | |- Push harder to find free extents |- If not found, re-iterate all block groups Clean enough. Signed-off-by: Qu Wenruo Reviewed-by: Su Yue --- fs/btrfs/extent-tree.c | 218

Re: [PATCH v2 3/4] btrfs: Refactor unclustered extent allocation into find_free_extent_unclustered()

2018-10-10 Thread Su Yue
Wenruo Reviewed-by: Su Yue --- fs/btrfs/extent-tree.c | 112 - 1 file changed, 66 insertions(+), 46 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index a603900e0eb8..5bc8919edac2 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs

Re: [PATCH v2 2/4] btrfs: Refactor clustered extent allocation into find_free_extent_clustered()

2018-10-10 Thread Su Yue
of jumping between different labels in find_free_extent(), the helper function will use return value to indicate different behavior. LGTM for code. One nit in comment. Reviewed-by: Su Yue Signed-off-by: Qu Wenruo --- fs/btrfs/extent-tree.c | 239 - 1 file

Re: [PATCH v2 1/4] btrfs: Introduce find_free_extent_ctrl structure for later rework

2018-10-10 Thread Su Yue
On 8/21/18 4:44 PM, Qu Wenruo wrote: Instead of tons of different local variables in find_free_extent(), extract them into find_free_extent_ctrl structure, and add better explanation for them. Some modification may looks redundant, but will later greatly simplify function parameter list

Re: [PATCH v3 3/6] btrfs-progs: original check: Add ability to detect bad dev extents

2018-10-08 Thread Su Yue
Wenruo Reviewed-by: Su Yue --- check/main.c | 99 1 file changed, 99 insertions(+) diff --git a/check/main.c b/check/main.c index bc2ee22f7943..ff9a785ce555 100644 --- a/check/main.c +++ b/check/main.c @@ -8224,6 +8224,99 @@ out

Re: [PATCH v3 2/6] btrfs-progs: lowmem check: Add check for overlapping dev extents

2018-10-08 Thread Su Yue
On 10/8/18 8:30 PM, Qu Wenruo wrote: Add such check at check_dev_item(), since at that timing we're also iterating dev extents for dev item accounting. Signed-off-by: Qu Wenruo LGTM. Reviewed-by: Su Yue --- check/mode-lowmem.c | 34 -- 1 file changed

Re: [PATCH 1/5] btrfs-progs: lowmem check: Add check for overlapping dev extents

2018-10-08 Thread Su Yue
On 10/8/18 3:00 PM, Qu Wenruo wrote: Add such check at check_dev_item(), since at that timing we're also iterating dev extents for dev item accounting. Signed-off-by: Qu Wenruo --- check/mode-lowmem.c | 32 ++-- 1 file changed, 30 insertions(+), 2 deletions(-)

Re: [PATCH 5/5] btrfs-progs: fsck-tests: Add test image for dev extents beyond device boundary

2018-10-08 Thread Su Yue
On 10/8/18 3:00 PM, Qu Wenruo wrote: Now two locations can detect such problem, either by device item used/total bytes check, or by early dev extents check against device boundary. The image is hand-crafted image which uses DATA SINGLE chunk to feed btrfs check. As expected, as long as block

Re: [PATCH 02/10] btrfs-progs: Add extent buffer bitmap manipulation infrastructure

2018-10-03 Thread Su Yue
have to deal with page mappings their implementation is vastly simplified by simply setting each bit in the passed range. Signed-off-by: Nikolay Borisov Reviewed-by: Su Yue --- extent_io.c | 56 extent_io.h | 4 2 files changed

Re: [PATCH] btrfs: tree-checker: Check level for leaves and nodes

2018-09-27 Thread Su Yue
On 9/28/18 7:59 AM, Qu Wenruo wrote: Although we have tree level check at tree read runtime, it's completely based on its parent level. We still need to do accurate level check to avoid invalid tree blocks sneak into kernel space. The check itself is simple, for leaf its level should always

Re: [PATCH] btrfs: tree-checker: Check level for leaves and nodes

2018-09-27 Thread Su Yue
s parent level. > We still need to do accurate level check to avoid invalid tree blocks > sneak into kernel space. > > The check itself is simple, for leaf its level should always be 0. > For nodes its level should be in range [1, BTRFS_MAX_LEVEL - 1]. > > Signed-off-by: Qu

Re: [PATCH 1/2] btrfs: Use NAME_MAX to replace intermediate number of BTRFS_NAME_LEN

2018-09-24 Thread Su Yue
On 9/25/18 8:06 AM, Qu Wenruo wrote: Since we're following the name size limit of linux, just use NAME_MAX. Signed-off-by: Qu Wenruo Reviewed-by: Su Yue --- fs/btrfs/ctree.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index

Re: [PATCH v3 5/7] btrfs-progs: lowmem: do missing check of last item after check_inode_item()

2018-09-18 Thread Su Yue
On 9/18/18 1:32 PM, Qu Wenruo wrote: On 2018/9/17 下午9:24, Su Yue wrote: On 2018/9/17 8:53 PM, Qu Wenruo wrote: On 2018/9/17 下午3:28, Su Yue wrote: After call of check_inode_item(), path may point to the last unchecked slot of the leaf. The outer walk_up_tree() always treats

Re: [PATCH v3 5/7] btrfs-progs: lowmem: do missing check of last item after check_inode_item()

2018-09-17 Thread Su Yue
On 2018/9/17 8:53 PM, Qu Wenruo wrote: On 2018/9/17 下午3:28, Su Yue wrote: After call of check_inode_item(), path may point to the last unchecked slot of the leaf. The outer walk_up_tree() always treats the position as checked slot then skips to the next. The last item will never be checked

Re: [PATCH v3 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() after repair

2018-09-17 Thread Su Yue
On 2018/9/17 8:51 PM, Qu Wenruo wrote: On 2018/9/17 下午3:28, Su Yue wrote: In check_fs_roots_lowmem(), we do search and follow the resulted path to call check_fs_root(), then call btrfs_next_item() to check next root. However, if repair is enabled, the root tree can be cowed, the existed

[PATCH v3 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() after repair

2018-09-17 Thread Su Yue
check_fs_root, search the saved key again before checking next root. Signed-off-by: Su Yue --- check/mode-lowmem.c | 22 ++ 1 file changed, 22 insertions(+) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index 4db12cc7f9fe..db44456fd85b 100644 --- a/check/mode

[PATCH v3 7/7] btrfs-progs: fsck-tests: add test case inode_extref without dir_item and dir_index

2018-09-17 Thread Su Yue
default_case.img to inode_ref_without_dir_item_and_index.img. Signed-off-by: Su Yue --- .../inode_extref_without_dir_item_and_index.img | Bin 0 -> 4096 bytes ... => inode_ref_without_dir_item_and_index.img} | Bin 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/fsck-tes

[PATCH v3 1/7] btrfs-progs: adjust arguments of btrfs_lookup_inode_extref()

2018-09-17 Thread Su Yue
The argument index is not used in btrfs_lookup_inode_extref(), so remove it. And adjust positions its arguments to make it consistent with kernel part. No functional change. Fixes: 260675657767 ("btrfs-progs: Import btrfs_insert/del/lookup_extref() functions.") Signed-off-by: Su Yue

[PATCH v3 6/7] btrfs-progs: lowmem: optimization and repair for check_inode_extref()

2018-09-17 Thread Su Yue
errors. Signed-off-by: Su Yue --- check/mode-lowmem.c | 125 +++- 1 file changed, 99 insertions(+), 26 deletions(-) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index 612e5e28e45b..53e4fdccd740 100644 --- a/check/mode-lowmem.c +++ b/check/mode

[PATCH v3 2/7] btrfs-progs: make btrfs_unlink() lookup inode_extref

2018-09-17 Thread Su Yue
tions.") Signed-off-by: Su Yue Reviewed-by: Qu Wenruo --- inode.c | 14 +- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/inode.c b/inode.c index 2398bca4a109..598ad0ab6b4c 100644 --- a/inode.c +++ b/inode.c @@ -277,6 +277,7 @@ int btrfs_unlink(struct btrfs_trans_han

[PATCH v3 5/7] btrfs-progs: lowmem: do missing check of last item after check_inode_item()

2018-09-17 Thread Su Yue
t;) Signed-off-by: Su Yue --- check/mode-lowmem.c | 37 + 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/check/mode-lowmem.c b/check/mode-lowmem.c index db44456fd85b..612e5e28e45b 100644 --- a/check/mode-lowmem.c +++ b/check/mode-lowmem.c @

[PATCH v3 3/7] btrfs-progs: lowmem check: find dir_item by di_key in check_dir_item()

2018-09-17 Thread Su Yue
be @di_key. Change comment about parameters too. To keep compactness, move declarations into while loop in check_dir_item(). Fixes: 564901eac7a4 ("btrfs-progs: check: introduce print_dir_item_err()") Signed-off-by: Su Yue --- check/mode-lowmem.c | 22 +++--- 1 file ch

[PATCH v3 0/7] btrfs-progs: lowmem: bug fixes and inode_extref repair

2018-09-17 Thread Su Yue
with inode_extref now. Suggested by Qu Wenruo. v2: Resend with patches in right order. Su Yue (7): btrfs-progs: adjust arguments of btrfs_lookup_inode_extref() btrfs-progs: make btrfs_unlink() lookup inode_extref btrfs-progs: lowmem check: find dir_item by di_key in check_dir_item() btrfs-progs

Re: [PATCH v2 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() under repair

2018-09-14 Thread Su Yue
On 09/14/2018 02:27 PM, Nikolay Borisov wrote: On 14.09.2018 03:58, Su Yue wrote: On 09/14/2018 07:37 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue In check_fs_roots_lowmem(), we do search and follow the resulted path to call check_fs_root

Re: [PATCH v2 5/7] btrfs-progs: lowmem: continue to check item in last slot while checking inodes

2018-09-13 Thread Su Yue
On 09/14/2018 07:43 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue After call of check_inode_item(), path may point to the last unchecked slot of the leaf. The outer walk_up_tree() always treats the position as checked item then skips to next item

Re: [PATCH v2 4/7] btrfs-progs: lowmem: search key of root again after check_fs_root() under repair

2018-09-13 Thread Su Yue
On 09/14/2018 07:37 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue In check_fs_roots_lowmem(), we do search and follow the resulted path to call check_fs_root(), then call btrfs_next_item() to check next root. However, if repair is enabled, the root

Re: [PATCH v2 3/7] btrfs-progs: lowmem check: find dir_item by di_key in check_dir_item()

2018-09-13 Thread Su Yue
On 09/14/2018 07:33 AM, Qu Wenruo wrote: On 2018/9/13 上午4:49, damenly...@gmail.com wrote: From: Su Yue In check_dir_item, we are going to search corresponding dir_item/index. Commit 564901eac7a4 ("btrfs-progs: check: introduce print_dir_item_err()") Changed argument nam

Re: [PATCH 0/6] btrfs-progs: lowmem: bug fixes and inode_extref repair

2018-09-12 Thread Su Yue
Sorry for the noise, Please ignore this patchset. Will send v2. On 2018/9/13 3:20 AM, damenly...@gmail.com wrote: From: Su Yue This patchset can be fetched from my github(based on v4.17.1): https://github.com/Damenly/btrfs-progs/tree/lowmem_extref The patchset aims to support check

Re: [PATCH 00/15] Add delayed-refs support to btrfs-progs

2018-09-12 Thread Su Yue
On 2018/7/16 11:39 PM, David Sterba wrote: On Fri, Jun 08, 2018 at 03:47:43PM +0300, Nikolay Borisov wrote: Hello, Here is a series which adds support for delayed refs. This is needed to enable later work on

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-09-05 Thread Su Yue
On 2018/9/5 8:33 PM, Christoph Anton Mitterer wrote: On Wed, 2018-09-05 at 15:04 +0800, Su Yue wrote: Agreed with Qu, btrfs-check shall not try to do any write. Well.. it could have been just some coincidence :-) I found the errors should blame to something about inode_extref check

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-09-05 Thread Su Yue
On 09/04/2018 04:24 AM, Christoph Anton Mitterer wrote: Hey. On Fri, 2018-08-31 at 10:33 +0800, Su Yue wrote: Can you please fetch btrfs-progs from my repo and run lowmem check in readonly? Repo: https://github.com/Damenly/btrfs-progs/tree/lowmem_debug It's based on v4.17.1 plus additonal

[PATCH] btrfs: defrag: use btrfs_mod_outstanding_extents in cluster_pages_for_defrag

2018-09-04 Thread Su Yue
62f87bad9c ("Btrfs: rework outstanding_extents") Signed-off-by: Su Yue --- fs/btrfs/ioctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 63600dc2ac4c..c180ded27092 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -1308

Re: bug: btrfs-progs scrub -R flag doesn't show per device stats

2018-09-02 Thread Su Yue
On 09/01/2018 02:43 AM, Chris Murphy wrote: btrfs-progs v4.17.1 man btrfs-scrub: -R print raw statistics per-device instead of a summary However, on a two device Btrfs volume, -R does not show per device statistics. See screenshot:

[PATCH] btrfs-progs: docs: update document about option -R of btrfs-scrub

2018-09-02 Thread Su Yue
-by: Chris Murphy Signed-off-by: Su Yue --- Documentation/btrfs-scrub.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/btrfs-scrub.asciidoc b/Documentation/btrfs-scrub.asciidoc index d2d206275c36..4c49269e2f36 100644 --- a/Documentation/btrfs-scrub.asci

Re: fsck lowmem mode only: ERROR: errors found in fs roots

2018-08-30 Thread Su Yue
Thank for the report. On 08/31/2018 12:47 AM, Christoph Anton Mitterer wrote: Hey. I've the following on a btrfs that's basically the system fs for my notebook: When booting from a USB stick with: # uname -a Linux heisenberg 4.17.0-3-amd64 #1 SMP Debian 4.17.17-1 (2018-08-18) x86_64

[PATCH] btrfs-progs: dump-tree: print invalid argument and strerror

2018-08-30 Thread Su Yue
. This patch let check_arg_type return -errno if realpath failed. And print strerror if check_arg_type failed and the returned code is negative. Like: $ btrfs inspect-internal dump-tree nothingness ERROR: invalid argument: nothingness: No such file or directory Signed-off-by: Su Yue --- cmds

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-08-23 Thread Su Yue
. This is especially handy to inspect high level trees, e.g. comparing tree reloc tree with its source tree. Signed-off-by: Qu Wenruo LGTM. Reviewed-by: Su Yue --- Documentation/btrfs-inspect-internal.asciidoc | 5 ++ cmds-inspect-dump-tree.c | 26 -- print-tree.c

Re: [PATCH] btrfs-progs: rescue: Add ability to disable quota offline

2018-08-21 Thread Su Yue
Merillat > Signed-off-by: Qu Wenruo Some nitpicks below. After fixes them, you can add tag: Reviewed-by: Su Yue > --- > This can patch can be fetched from github repo: > https://github.com/adam900710/btrfs-progs/tree/quota_disable > --- > Documentation/btrf

Re: [PATCH 0/2] btrfs-progs: task: add prefn() to task_info and simplify task_start()

2018-08-21 Thread Su Yue
> Sent: Wednesday, August 22, 2018 at 3:38 AM > From: "Nikolay Borisov" > To: damenly...@gmail.com, linux-btrfs@vger.kernel.org > Cc: "Su Yue" > Subject: Re: [PATCH 0/2] btrfs-progs: task: add prefn() to task_info and > simplify task_start()

Re: [RFC PATCH] btrfs: Remove 'objectid' member from struct btrfs_root

2018-08-06 Thread Su Yue
On 08/06/2018 02:17 PM, Qu Wenruo wrote: On 2018年08月06日 13:25, Misono Tomohiro wrote: There are two members in struct btrfs_root which indicate root's objectid: ->objectid and ->root_key.objectid. They are both set to the same value in __setup_root(): static void __setup_root(struct

Re: [PATCH v2 4/6] btrfs: Introduce mount time chunk <-> dev extent mapping check

2018-07-31 Thread Su Yue
havior (BUG_ON). Reported-by: Xu Wen Link: https://bugzilla.kernel.org/show_bug.cgi?id=200403 Link: https://bugzilla.kernel.org/show_bug.cgi?id=200407 Signed-off-by: Qu Wenruo LGTM. Reviewed-by: Su Yue --- fs/btrfs/disk-io.c | 7 ++ fs/btrfs/volumes.c

Re: [PATCH v2 6/6] btrfs: locking: Allow btrfs_tree_lock() to return error to avoid deadlock

2018-07-31 Thread Su Yue
-by: Qu Wenruo Reviewed-by: Su Yue --- fs/btrfs/ctree.c | 57 +++--- fs/btrfs/extent-tree.c | 28 +++ fs/btrfs/extent_io.c | 8 -- fs/btrfs/free-space-tree.c | 4 ++- fs/btrfs/locking.c | 13 +++-- fs

Re: [PATCH v2 1/6] btrfs: Check each block group has corresponding chunk at mount time

2018-07-31 Thread Su Yue
or each block group at mount time. Here we reuse the original find_first_block_group(), which is already doing basic bg -> chunk check, adding more check on start/len and type flags. Link: https://bugzilla.kernel.org/show_bug.cgi?id=199837 Reported-by: Xu Wen Signed-off-by: Qu Wenruo

Re: [PATCH 1/1] btrfs: locking: Allow btrfs_tree_lock() to return error to avoid deadlock

2018-07-31 Thread Su Yue
-by: Qu Wenruo OK.. I looked through every places where the callee is called. Those errors are handled gracefully as my thought. Except one nitpick, the %llu about pid_t in btrfs_tree_lock. However, you can add the tag: Reviewed-by: Su Yue --- fs/btrfs/ctree.c | 57

Re: [PATCH] btrfs: tree-checker: Add DATA_RELOC tree to essential non-empty tree list

2018-07-19 Thread Su Yue
On 07/19/2018 03:05 PM, Qu Wenruo wrote: DATA RELOC tree should also never be empty. Link: https://bugzilla.kernel.org/show_bug.cgi?id=200407 Reported-by: Xu Wen Signed-off-by: Qu Wenruo Reviewed-by: Su Yue --- It's possible just fold this fix into commit "btrfs: tree-checker: D

Re: [PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Su Yue
On 07/17/2018 04:01 PM, Nikolay Borisov wrote: On 17.07.2018 10:46, Qu Wenruo wrote: [BUG] For certain fuzzed btrfs image, if we create any csum data, it would cause the following kernel warning and deadlock when trying to update csum tree: -- [ 278.113360] WARNING: CPU: 1 PID: 41 at

[PATCH v2] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Su Yue
,u64' for users(developers more likely) to appoint a specific key to search and dump, then the path from root node down the leaf will be dumped. The search of the key costs most time of this way. Signed-off-by: Su Yue --- Change log: v2: Rename btrfs_search_spec_key() to btrfs_search_print_path

Re: [PATCH] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Su Yue
On 07/16/2018 03:45 PM, Qu Wenruo wrote: On 2018年07月16日 13:55, Su Yue wrote: For big filesystems, there are many items in trees(extent tree specially). For example, to dump one extent, we usually dump extent tree then pipe result to grep. The time-consuming part is that dump tree traverses

[PATCH] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-15 Thread Su Yue
,u64' for users(developer more likely) to appoint a specific key to search and dump, then the path from root node down the leaf will be dumped. The search of the key costs most time of this way. Signed-off-by: Su Yue --- In my experiment, not usual case though. Image was provided by Chris Murphy

Re: btrfs check lowmem, take 2

2018-07-10 Thread Su Yue
On 07/11/2018 11:36 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 09:58:36AM +0800, Su Yue wrote: On 07/11/2018 09:44 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 09:08:40AM +0800, Su Yue wrote: On 07/11/2018 08:58 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 08:53:58AM +0800, Su

Re: btrfs check lowmem, take 2

2018-07-10 Thread Su Yue
On 07/11/2018 09:44 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 09:08:40AM +0800, Su Yue wrote: On 07/11/2018 08:58 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 08:53:58AM +0800, Su Yue wrote: Problems 1) btrfs check --repair _still_ takes all 32GB of RAM and crashes the server

Re: btrfs check lowmem, take 2

2018-07-10 Thread Su Yue
On 07/11/2018 08:58 AM, Marc MERLIN wrote: On Wed, Jul 11, 2018 at 08:53:58AM +0800, Su Yue wrote: Problems 1) btrfs check --repair _still_ takes all 32GB of RAM and crashes the server, despite my deleting lots of snapshots. Is it because I have too many files then? Yes. Original check

Re: So, does btrfs check lowmem take days? weeks?

2018-07-10 Thread Su Yue
On 07/10/2018 06:53 PM, Su Yue wrote: On 07/10/2018 12:10 PM, Marc MERLIN wrote: On Tue, Jul 10, 2018 at 08:56:15AM +0800, Su Yue wrote: I'm just not clear if my FS is still damaged and btrfsck was just hacked to ignore the damage it can't deal with, or whether it was able to repair

Re: btrfs check lowmem, take 2

2018-07-10 Thread Su Yue
On 07/11/2018 02:09 AM, Marc MERLIN wrote: Thanks to Su and Qu, I was able to get my filesystem to a point that it's mountable. I then deleted loads of snapshots and I'm down to 26. IT now looks like this: gargamel:~# btrfs fi show /mnt/mnt Label: 'dshelf2' uuid:

Re: So, does btrfs check lowmem take days? weeks?

2018-07-10 Thread Su Yue
On 07/10/2018 12:55 PM, Qu Wenruo wrote: On 2018年07月10日 11:50, Marc MERLIN wrote: On Tue, Jul 10, 2018 at 09:34:36AM +0800, Qu Wenruo wrote: Ok, this is where I am now: WARNING: debug: end of checking extent item[18457780273152 169 1] type: 176 offset: 2 checking extent items

Re: So, does btrfs check lowmem take days? weeks?

2018-07-09 Thread Su Yue
[CC to linux-btrfs] Here is the log of wrong extent data. On 07/08/2018 01:21 AM, Marc MERLIN wrote: On Fri, Jul 06, 2018 at 10:56:36AM -0700, Marc MERLIN wrote: On Fri, Jul 06, 2018 at 09:05:23AM -0700, Marc MERLIN wrote: Ok, this is where I am now: WARNING: debug: end of checking extent

Re: Fwd: Re: So, does btrfs check lowmem take days? weeks?

2018-07-09 Thread Su Yue
Forgot to CC Marc. On 07/10/2018 09:33 AM, Su Yue wrote: [FWD to linux-btrfs] Thanks to Marc's patient of running and tests btrfsck lowmem mode in recent days. The FS has a large extent tree but luckily few are corrupted, they are all fixed by special version. Reloc trees were cleaned too. So

Fwd: Re: So, does btrfs check lowmem take days? weeks?

2018-07-09 Thread Su Yue
. Error messages are showed in the last. Forwarded Message Subject: Re: So, does btrfs check lowmem take days? weeks? Date: Mon, 9 Jul 2018 10:48:18 -0700 From: Marc MERLIN To: Su Yue CC: quwenruo.bt...@gmx.com, Su Yue Success! Well done Su, this is a huge improvement

  1   2   3   4   5   >