Re: [PATCH 2/5] New btrfs command: "btrfs inspect physical-find"

2016-07-28 Thread Qu Wenruo
Hi, Goffredo, Sorry I forgot to mention that, even btrfs-map-logcal is an offline tool, it can still handle mount fs too. Although it's also true that it still lacks the needed RAID flags and stripe info. At 07/29/2016 04:25 AM, Goffredo Baroncelli wrote: Hi Qu, On 2016-07-28 03:4

Re: [PATCH 2/5] New btrfs command: "btrfs inspect physical-find"

2016-07-28 Thread Qu Wenruo
On 07/29/2016 01:08 PM, Goffredo Baroncelli wrote: On 2016-07-29 03:34, Qu Wenruo wrote: I am not against about your proposal; however I have to point out that the goal of these command was not to *traverse* the file, but only to found the physical location of a file offset. My use case was

Btrfs send to send out metadata and data separately

2016-07-29 Thread Qu Wenruo
Hi Filipe, and maintainers, I'm recently working on the root fix to free send from calling backref walk. My current idea is to send data and metadata separately, and only do clone detection inside the send subvolume. This method needs two new send commands: (And new send attribute, A_DATA_BYT

Re: [PATCH 2/5] New btrfs command: "btrfs inspect physical-find"

2016-07-29 Thread Qu Wenruo
On 07/30/2016 01:14 AM, Goffredo Baroncelli wrote: On 2016-07-29 08:44, Qu Wenruo wrote: On 07/29/2016 01:08 PM, Goffredo Baroncelli wrote: On 2016-07-29 03:34, Qu Wenruo wrote: I am not against about your proposal; however I have to point out that the goal of these command was not to

Re: btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-07-29 Thread Qu Wenruo
, Qu Wenruo wrote: When balancing data extents, qgroup will leak all its numbers for balanced data extents. The root cause is the non-standard extent reference update used in balance code. The problem happens in the following steps: (Use 4M as original data extent size, and 257 as src root objectid

Re: btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-07-30 Thread Qu Wenruo
On 07/30/2016 11:57 PM, Goldwyn Rodrigues wrote: On 07/29/2016 08:06 PM, Qu Wenruo wrote: Hi, Goldwyn, This patch is replaced by the following patchset: https://patchwork.kernel.org/patch/9213915/ https://patchwork.kernel.org/patch/9213913/ Would you mind testing the new patch? Sorry

Re: Btrfs send to send out metadata and data separately

2016-07-31 Thread Qu Wenruo
At 07/29/2016 09:14 PM, Libor Klepáč wrote: Hello, just a little question on receiver point 0), see bellow Dne pátek 29. července 2016 20:40:38 CEST, Qu Wenruo napsal(a): Hi Filipe, and maintainers, Receive will do the following thing first before recovering the subvolume/snapshot: 0

Re: Btrfs send to send out metadata and data separately

2016-07-31 Thread Qu Wenruo
At 07/31/2016 02:49 AM, g.bt...@cobb.uk.net wrote: On 29/07/16 13:40, Qu Wenruo wrote: Cons: 1) Not full fs clone detection Such clone detection is only inside the send snapshot. For case that one extent is referred only once in the send snapshot, but also referred by source

[PATCH 2/3] btrfs-progs: inspect: Introduce dump-send-stream subcommand

2016-07-31 Thread Qu Wenruo
Introduce a new subcommand "dump-send-stream" for "inspect-internal" command group. It will exam and output all operations inside a send stream. It's quite a useful tool to learn and dig kernel send stream. Signed-off-by: Qu Wenruo --- Documentation/btrfs-inspe

[PATCH 3/3] btrfs-progs: Remove send-test tool

2016-07-31 Thread Qu Wenruo
Since new "inspect dump-send" has better output and structure, it's time to remove old and function-weak send-test tool. New "inspect dump-send" can handle them all better. Signed-off-by: Qu Wenruo --- Makefile.in |

[PATCH 1/3] btrfs-progs: Introduce new send-dump object

2016-07-31 Thread Qu Wenruo
dump-send" command. Signed-off-by: Qu Wenruo --- Makefile.in | 2 +- send-dump.c | 367 send-dump.h | 24 3 files changed, 392 insertions(+), 1 deletion(-) create mode 100644 send-dump.c create mode 100644 send-dump.h di

Re: Btrfs send to send out metadata and data separately

2016-08-01 Thread Qu Wenruo
At 08/02/2016 02:00 AM, Filipe Manana wrote: On Fri, Jul 29, 2016 at 1:40 PM, Qu Wenruo wrote: Hi Filipe, and maintainers, I'm recently working on the root fix to free send from calling backref walk. My current idea is to send data and metadata separately, and only do clone dete

Re: Btrfs send to send out metadata and data separately

2016-08-03 Thread Qu Wenruo
At 08/03/2016 05:05 PM, Filipe Manana wrote: On Tue, Aug 2, 2016 at 2:20 AM, Qu Wenruo wrote: At 08/02/2016 02:00 AM, Filipe Manana wrote: On Fri, Jul 29, 2016 at 1:40 PM, Qu Wenruo wrote: [snipped] Hi Filipe, Thanks for your comment, it helps to refine the idea to fix the

[PATCH v2 0/3] Qgroup fix for dirty hack routines

2016-08-04 Thread Qu Wenruo
patch to handle cases where the whole subtree, not only level 2 nodes get updated. Qu Wenruo (3): btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent() btrfs: relocation: Fix leaking qgroups numbers on data extents btrfs: qgroup: Fix qgroup incorrectness caused by log replay fs/btrfs

[PATCH v2 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-04 Thread Qu Wenruo
since more btrfs hacks exposed, like replacing path in balance, needs us to record dirty extents manually, so we have to add such functions. Also, add comment for both functions, to info developers how to keep qgroup correct when doing hacks. Cc: Mark Fasheh Signed-off-by: Qu Wenruo --- fs

[PATCH v2 2/3] btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-08-04 Thread Qu Wenruo
up to track all file extents in data reloc tree. Since at commit transaction time, the tree swapping is done, and qgroup will account these data extents correctly. Cc: Mark Fasheh Reported-by: Mark Fasheh Reported-by: Filipe Manana Signed-off-by: Qu Wenruo --- changelog: v2: Iterate all file ex

[PATCH v2 3/3] btrfs: qgroup: Fix qgroup incorrectness caused by log replay

2016-08-04 Thread Qu Wenruo
When doing log replay at mount time(after power loss), qgroup will leak numbers of replayed data extents. The cause is almost the same of balance. So fix it by manually informing qgroup for owner changed extents. The bug can be detected by btrfs/119 test case. Cc: Mark Fasheh Signed-off-by: Qu

Re: [PATCH v2 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-07 Thread Qu Wenruo
At 08/07/2016 01:19 AM, Goldwyn Rodrigues wrote: Thanks Qu, This patch set fixes all the reported problems. However, I have some minor issues with coding style. Thanks for the test. On 08/04/2016 09:29 PM, Qu Wenruo wrote: Refactor btrfs_qgroup_insert_dirty_extent() function, to two

Re: [PATCH v2 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-08 Thread Qu Wenruo
At 08/08/2016 10:54 AM, Goldwyn Rodrigues wrote: On 08/07/2016 07:39 PM, Qu Wenruo wrote: At 08/07/2016 01:19 AM, Goldwyn Rodrigues wrote: Thanks Qu, This patch set fixes all the reported problems. However, I have some minor issues with coding style. Thanks for the test. On 08/04

Re: [PATCH v2 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-08 Thread Qu Wenruo
At 08/09/2016 09:01 AM, Goldwyn Rodrigues wrote: On 08/08/2016 07:26 PM, Qu Wenruo wrote: At 08/08/2016 10:54 AM, Goldwyn Rodrigues wrote: On 08/07/2016 07:39 PM, Qu Wenruo wrote: At 08/07/2016 01:19 AM, Goldwyn Rodrigues wrote: Thanks Qu, This patch set fixes all the reported

Re: [PATCH v12.2 01/15] btrfs: expand btrfs_set_extent_delalloc() and its friends to support in-band dedupe and subpage size patchset

2016-08-08 Thread Qu Wenruo
Hi David and Chandan, I see the expansion of cow_file_range() function got merged into 4.8 window, while this patch is still not merged yet, is there anything wrong with this patch? Thanks, Qu At 07/19/2016 04:50 PM, Qu Wenruo wrote: Extract btrfs_set_extent_delalloc() and

Re: [PATCH v2 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-08 Thread Qu Wenruo
At 08/09/2016 10:24 AM, Goldwyn Rodrigues wrote: On 08/08/2016 08:12 PM, Qu Wenruo wrote: At 08/09/2016 09:01 AM, Goldwyn Rodrigues wrote: On 08/08/2016 07:26 PM, Qu Wenruo wrote: At 08/08/2016 10:54 AM, Goldwyn Rodrigues wrote: On 08/07/2016 07:39 PM, Qu Wenruo wrote: At 08

[PATCH v3 3/3] btrfs: qgroup: Fix qgroup incorrectness caused by log replay

2016-08-09 Thread Qu Wenruo
When doing log replay at mount time(after power loss), qgroup will leak numbers of replayed data extents. The cause is almost the same of balance. So fix it by manually informing qgroup for owner changed extents. The bug can be detected by btrfs/119 test case. Cc: Mark Fasheh Signed-off-by: Qu

[PATCH v2 0/3] Qgroup fix for dirty hack routines

2016-08-09 Thread Qu Wenruo
record_dirty_extent()' to 'btrfs_qgroup_insert_dirty_extent()' Qu Wenruo (3): btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent() btrfs: relocation: Fix leaking qgroups numbers on data extents btrfs: qgroup: Fix qgroup incorrectness caused by log replay fs/btrfs/delayed-ref.c | 7 +-- fs/btrfs/

[PATCH v3 2/3] btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-08-09 Thread Qu Wenruo
up to track all file extents in data reloc tree. Since at commit transaction time, the tree swapping is done, and qgroup will account these data extents correctly. Cc: Mark Fasheh Reported-by: Mark Fasheh Reported-by: Filipe Manana Signed-off-by: Qu Wenruo --- fs/btrfs/relocation.c

[PATCH v3 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-09 Thread Qu Wenruo
: Qu Wenruo --- fs/btrfs/delayed-ref.c | 7 ++- fs/btrfs/extent-tree.c | 37 + fs/btrfs/qgroup.c | 41 +++-- fs/btrfs/qgroup.h | 33 + 4 files changed, 71 insertions(+), 47

Re: [PATCH v3 2/3] btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-08-14 Thread Qu Wenruo
At 08/12/2016 09:33 PM, Filipe Manana wrote: On Tue, Aug 9, 2016 at 9:30 AM, Qu Wenruo wrote: When balancing data extents, qgroup will leak all its numbers for relocated data extents. The relocation is done in the following steps for data extents: 1) Create data reloc tree and inode 2) Copy

Re: btrfs quota issues

2016-08-14 Thread Qu Wenruo
At 08/12/2016 01:32 AM, Rakesh Sankeshi wrote: I set 200GB limit to one user and 100GB to another user. as soon as I reached 139GB and 53GB each, hitting the quota errors. anyway to workaround quota functionality on btrfs LZO compressed filesystem? Please paste "btrfs qgroup show -prce " ou

[PATCH v3.1 1/3] btrfs: qgroup: Refactor btrfs_qgroup_insert_dirty_extent()

2016-08-14 Thread Qu Wenruo
since more btrfs hacks exposed, like replacing path in balance, we need to record dirty extents manually, so we have to add such functions. Also, add comment for both functions, to info developers how to keep qgroup correct when doing hacks. Cc: Mark Fasheh Signed-off-by: Qu Wenruo Reviewed-and

[PATCH v3.1 0/3] Qgroup fix for dirty hack routines

2016-08-14 Thread Qu Wenruo
_qgroup_insert_dirty_extent()' v3.1: Update commit message for 2nd patch, to identify itself as a regression fix patch, suggested by Filipe. Removed commented out debug command of 2nd patch. Update Tested-by and Reviewed-by tags from Goldwyn. Qu Wenruo (3): btrfs: qgroup: Refactor bt

[PATCH v3.1 3/3] btrfs: qgroup: Fix qgroup incorrectness caused by log replay

2016-08-14 Thread Qu Wenruo
When doing log replay at mount time(after power loss), qgroup will leak numbers of replayed data extents. The cause is almost the same of balance. So fix it by manually informing qgroup for owner changed extents. The bug can be detected by btrfs/119 test case. Cc: Mark Fasheh Signed-off-by: Qu

[PATCH v3.1 2/3] btrfs: relocation: Fix leaking qgroups numbers on data extents

2016-08-14 Thread Qu Wenruo
ed-by: Filipe Manana Signed-off-by: Qu Wenruo Tested-by: Goldwyn Rodrigues --- fs/btrfs/relocation.c | 109 +++--- 1 file changed, 103 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c index b26a5ae..27480ef 100644 ---

About minimal device number for RAID5/6

2016-08-15 Thread Qu Wenruo
Hi, Recently I found that manpage of mkfs is saying minimal device number for RAID5 and RAID6 is 2 and 3. Personally speaking, although I understand that RAID5/6 only requires 1/2 devices for parity stripe, it is still quite strange behavior. Under most case, user use raid5/6 for striping A

Re: btrfs quota issues

2016-08-15 Thread Qu Wenruo
ent size (non-compression). So qgroup + compression is not the main concern before and is buggy. If without compression, qgroup works sanely, at least we can be sure that the cause is qgroup + compression. Thanks, Qu On Sun, Aug 14, 2016 at 7:11 PM, Qu Wenruo wrote: At 08/12/2016 01:32

Re: About minimal device number for RAID5/6

2016-08-15 Thread Qu Wenruo
At 08/15/2016 10:10 PM, Austin S. Hemmelgarn wrote: On 2016-08-15 10:08, Anand Jain wrote: IMHO it's better to warn user about 2 devices RAID5 or 3 devices RAID6. Any comment is welcomed. Based on looking at the code, we do in fact support 2/3 devices for raid5/6 respectively. Personall

Re: [PATCH] btrfs: properly track when rescan worker is running

2016-08-15 Thread Qu Wenruo
. Cc: # v4.4+ Signed-off-by; Jeff Mahoney Reviewed-by: Qu Wenruo Looks good to me. Would you mind to submit a test case for it? Thanks, Qu --- fs/btrfs/ctree.h |1 + fs/btrfs/disk-io.c |1 + fs/btrfs/qgroup.c |9 - 3 files changed, 10 insertions(+), 1 deletion

Re: [PATCH 1/3] btrfs-progs: Introduce new send-dump object

2016-08-16 Thread Qu Wenruo
At 08/16/2016 10:31 PM, David Sterba wrote: On Mon, Aug 01, 2016 at 02:29:42PM +0800, Qu Wenruo wrote: Introduce send-dump.[ch] which implements a new btrfs_send_ops to exam and output all operations inside a send stream. It has a better output format than the old and no longer compilable

Re: btrfs quota issues

2016-08-16 Thread Qu Wenruo
orting error message like "No space left on device" (ENOSPC) or "Quota exceeded"(EDQUOT)? Thanks, Qu In case of ext4 and xfs, I can see visually that it's hitting the quota limit. On Mon, Aug 15, 2016 at 6:01 PM, Qu Wenruo wrote: At 08/16/2016 03:11 AM, Rake

Re: [PATCH] btrfs: Fix leaking bytes_may_use after hitting EDQUOTA

2016-08-16 Thread Qu Wenruo
Hi David, I think this patch is forgot for v4.8 merge windows. Xfstest btrfs/022 is complaining about the problem if the patch is not applied. Thanks, Qu At 06/28/2016 09:57 AM, Qu Wenruo wrote: If one mount btrfs with enospc_debug mount option and hit qgroup limits in

Re: [PATCH 1/2] btrfs/131: test for umount of read-only fs when quota rescan is paused

2016-08-17 Thread Qu Wenruo
At 08/17/2016 04:45 PM, Eryu Guan wrote: On Tue, Aug 16, 2016 at 02:30:21PM -0400, Jeff Mahoney wrote: Ensure that we can unmount a read-only file system when quota rescan is paused from a previous read-write mount. If the kernel has a separate bug where we are returning early while waiting f

[PATCH 2/4] btrfs-progs: convert-test: Check if the ext2_save/image is read only

2016-08-19 Thread Qu Wenruo
rollback fails. This test case will test the regression. Signed-off-by: Qu Wenruo --- tests/convert-tests/008-readonly-image/test.sh | 26 ++ 1 file changed, 26 insertions(+) create mode 100755 tests/convert-tests/008-readonly-image/test.sh diff --git a/tests/convert-tests

[PATCH 4/4] btrfs-progs: print-tree: Print hex and human readable root flags

2016-08-19 Thread Qu Wenruo
Signed-off-by: Qu Wenruo --- print-tree.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/print-tree.c b/print-tree.c index f33ddad..81ab81f 100644 --- a/print-tree.c +++ b/print-tree.c @@ -496,18 +496,31 @@ static int count_bytes(void *buf, int len, char b) return

[PATCH 3/4] btrfs-progs: print-tree: Print human readable inode flags

2016-08-19 Thread Qu Wenruo
Signed-off-by: Qu Wenruo --- print-tree.c | 34 -- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/print-tree.c b/print-tree.c index 9f9e11e..f33ddad 100644 --- a/print-tree.c +++ b/print-tree.c @@ -835,6 +835,33 @@ static void print_uuid_item

[PATCH 0/4] Convert regression fix with print-tree enhancement

2016-08-19 Thread Qu Wenruo
This patchset fixes one convert regression, adds regression test. Then with 2 print-tree enhancement, which is quite handy to find how inode and subvolume READONLY flag is set. Qu Wenruo (4): btrfs-progs: convert: Fix a regression that ext2_save/image is not readonly btrfs-progs: convert

[PATCH 1/4] btrfs-progs: convert: Fix a regression that ext2_save/image is not readonly

2016-08-19 Thread Qu Wenruo
old code behavior, use 0400 permission and add back READONLY flag to fix it. Signed-off-by: Qu Wenruo --- btrfs-convert.c | 9 - ctree.h | 2 ++ inode.c | 36 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/btrfs

[Regression/Behavior change]dm-flakey corrupt read bio, even the feature is drop_writes

2016-08-22 Thread Qu Wenruo
Hi, Mike and btrfs and dm guys When doing regression test on v4.8-rc1, we found that fstests/btrfs/056 always fails. With the following dmesg: --- Buffer I/O error on dev dm-0, logical block 1310704, async page read Buffer I/O error on dev dm-0, logical block 16, async page read Buffer I/O erro

Re: [dm-devel] [Regression/Behavior change]dm-flakey corrupt read bio, even the feature is drop_writes

2016-08-23 Thread Qu Wenruo
- https://www.redhat.com/archives/dm-devel/2016-July/msg00561.html - https://www.redhat.com/archives/dm-devel/2016-August/msg00109.html Lukas On Mon, Aug 22, 2016 at 9:31 AM, Qu Wenruo wrote: Hi, Mike and btrfs and dm guys When doing regression test on v4.8-rc1, we found that fstests/btrfs

Re: [Regression/Behavior change]dm-flakey corrupt read bio, even the feature is drop_writes

2016-08-23 Thread Qu Wenruo
https://www.redhat.com/archives/dm-devel/2016-August/msg00109.html Lukas On Mon, Aug 22, 2016 at 9:31 AM, Qu Wenruo wrote: Hi, Mike and btrfs and dm guys When doing regression test on v4.8-rc1, we found that fstests/btrfs/056 always fails. With the following dmesg: --- Buffer I/O error on dev dm-0, lo

Re: Btrfs send to send out metadata and data separately

2016-08-23 Thread Qu Wenruo
At 08/04/2016 09:52 AM, Qu Wenruo wrote: At 08/03/2016 05:05 PM, Filipe Manana wrote: On Tue, Aug 2, 2016 at 2:20 AM, Qu Wenruo wrote: At 08/02/2016 02:00 AM, Filipe Manana wrote: On Fri, Jul 29, 2016 at 1:40 PM, Qu Wenruo wrote: [snipped] Hi Filipe, Thanks for your comment

[PATCH] btrfs: Fix extent map leak in find_first_block_group

2016-08-23 Thread Qu Wenruo
The following commit introduced the extent map leak: commit 6fb37b756acce6d6e045f79c3764206033f617b4 Author: Liu Bo Date: Wed Jun 22 18:31:27 2016 -0700 Btrfs: check inconsistence between chunk and block group This leads to slab warning at rmmod time. Cc: Liu Bo Signed-off-by: Qu Wenruo

Re: btrfs quota issues

2016-08-25 Thread Qu Wenruo
ese 2 subvolumes seems to be 200G, maybe just coincident?) Thanks, Qu On Tue, Aug 16, 2016 at 5:56 PM, Qu Wenruo wrote: At 08/17/2016 12:05 AM, Rakesh Sankeshi wrote: 2) after EDQUOT, can't write anymore. I can delete the data, but still can't write further So it's a

Re: btrfs and systemd

2016-08-28 Thread Qu Wenruo
At 08/29/2016 04:15 AM, Stefan Priebe - Profihost AG wrote: Hi, i'm trying to get my 60TB btrfs volume to mount with systemd at boot. But this always fails with: "mounting timed out. Stopping." after 90s. 60TB is quite large, and under most case it will already cause mount speed problem.

Re: btrfs send extremely slow (almost stuck)

2016-08-28 Thread Qu Wenruo
At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to simplify the process). It seems that on my two machines, btrfs send gets stuck after transferring some GiB - it's not fully halted, bu

Re: btrfs send extremely slow (almost stuck)

2016-08-28 Thread Qu Wenruo
At 08/29/2016 10:11 AM, Qu Wenruo wrote: At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to simplify the process). It seems that on my two machines, btrfs send gets stuck after

Re: Multiple bugs found by fuzzing BTRFS

2016-08-28 Thread Qu Wenruo
Thanks for your fuzzing images. Quite helpful. At 08/29/2016 02:06 PM, Lukas Lueg wrote: Hi, I've now spent around 160 hours of fuzzing BTRFS, here are the crashes I found so far. Every type of crash is reported only once although there are usually multiple locations where they show up (especi

[PATCH 3/4] btrfs-progs: Ignore invalid key in invalid root

2016-08-29 Thread Qu Wenruo
Btrfs tree implies a lot of restriction on which key types are allowed in specific roots. Like CHUNK_ITEM keys are only valid in chunk root. This patch will add such check at run_next_block() for original mode. Reported-by: Lukas Lueg Signed-off-by: Qu Wenruo --- cmds-check.c | 62

[PATCH 2/4] btrfs-progs: Enhance and export print_key_type function

2016-08-29 Thread Qu Wenruo
Just the same thing done for print_objectid(). Signed-off-by: Qu Wenruo --- print-tree.c | 86 ++-- print-tree.h | 1 + 2 files changed, 44 insertions(+), 43 deletions(-) diff --git a/print-tree.c b/print-tree.c index 601fc0f..f97482f

[PATCH 1/4] btrfs-progs: Enhance and export print_objectid function

2016-08-29 Thread Qu Wenruo
This function is quite useful for a lot of error report. Enhance it to support custom output other than stdout. And export it for later btrfsck enhancement. Signed-off-by: Qu Wenruo --- print-tree.c | 59 ++- print-tree.h | 1 + 2 files

[PATCH 0/4] Fix chunk num_stripes FPE error found by fuzzed image

2016-08-29 Thread Qu Wenruo
is check itself can already fix the problem. But 1) can give more info on which is really going wrong in the fs. Thanks for Lukas for his fuzzed images and report. Qu Wenruo (4): btrfs-progs: Enhance and export print_objectid function btrfs-progs: Enhance and export print_key_type function

[PATCH 4/4] btrfs-progs: Do extra chunk check before processing chunk item

2016-08-29 Thread Qu Wenruo
type check against leaf owner, it is still possible to modify the leaf owner to by-pass it. So we still need to check chunk validation before processing it. Reported-by: Lukas Lueg Signed-off-by: Qu Wenruo --- cmds-check.c | 16 volumes.c| 8 volumes.h| 4 ++

Re: does btrfs-receive use/compare the checksums from the btrfs-send side?

2016-08-29 Thread Qu Wenruo
At 08/28/2016 11:46 AM, Christoph Anton Mitterer wrote: Hey. I've often wondered: When I do a send/receive, does the receiving side use the checksums from the sending side (either by directly storing them or by comparing them with calculated checksums and failing if they don't match after the

Re: btrfs send extremely slow (almost stuck)

2016-08-29 Thread Qu Wenruo
At 08/29/2016 06:02 PM, Oliver Freyermuth wrote: Am 29.08.2016 um 04:11 schrieb Qu Wenruo: Unknown bug, while unfortunately no good idea to solve yet. I sent a RFC patch to completely disable shared extent detection, while got strong objection. I also submitted some other ideas on fixing it

Re: [PATCH 0/4] Fix chunk num_stripes FPE error found by fuzzed image

2016-08-29 Thread Qu Wenruo
At 08/30/2016 12:28 AM, David Sterba wrote: On Mon, Aug 29, 2016 at 04:08:58PM +0800, Qu Wenruo wrote: This patchset will fix chunk num_stripes FPE error, by introducing another 2 layers of check. 1) Check key type validation against leaf owner(for original mode) This will prevent invalid

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

2016-08-29 Thread Qu Wenruo
At 08/29/2016 11:20 PM, David Sterba wrote: On Thu, Aug 25, 2016 at 01:30:09PM +0800, Wang Xiaoguang wrote: Hi, On 08/24/2016 08:44 PM, David Sterba wrote: On Fri, Aug 19, 2016 at 05:59:46PM +0800, Wang Xiaoguang wrote: The basic idea is simple. Assume a middle tree node A is shared and its

[PATCH] btrfs-progs: fuzz-test: Add image for wrong chunk item in root tree

2016-08-29 Thread Qu Wenruo
From: Lukas Lueg Reported by Lukas and the same image from him. DATA_RELOC tree's key type is modifed to CHUNK_ITEM, causing btrfsck interpret it as CHUNK_ITEM and cause 0 num_stripes. Add the image to fuzz-test. Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- .../images/wrong-

[PATCH 1/2] btrfs-progs: fsck: Do early check for read_tree_block

2016-08-29 Thread Qu Wenruo
eb for the usage. So we should not allow invalid bytenr/blocksize even passed to btrfs_find_create_tree_block(). This patch will add such check so we won't trigger use-after-free bug then. Reported-by: Lukas Lueg Signed-off-by: Qu Wenruo --- disk-io.c | 18 ++ 1 file change

[PATCH 2/2] btrfs-progs: fuzz-test: Add image for unaligned tree block ptr

2016-08-29 Thread Qu Wenruo
From: Lukas Lueg Add test case image for unaligned tree block ptr. It should lead to BUG_ON in free_extent_buffer(). Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- .../images/unaligned-tree-block-bytenr.raw.txt | 33 + .../images/unaligned-tree-block

[PATCH 1/5] btrfs-progs: fsck: Check drop level before walking through fs tree

2016-08-30 Thread Qu Wenruo
Exposed by fuzzed image from Lukas, which contains invalid drop level (16), causing segfault when accessing path->nodes[drop_level]. This patch will check drop level against fs tree level and BTRFS_MAX_LEVEL to avoid such problem. Reported-by: Lukas Lueg Signed-off-by: Qu Wenruo --- c

[PATCH 0/5] Fuzzer test fix

2016-08-30 Thread Qu Wenruo
safe against such things. I can't wait to see how the full low-memory mode works under fuzzer tests. Lukas Lueg (2): btrfs-progs: fuzz-test: Add test case for invalid drop level btrfs-progs: fuzz-test: Add test case for unaligned extent item Qu Wenruo (3): btrfs-progs: fsck: Check

[PATCH 5/5] btrfs-progs: fuzz-test: Add test case for unaligned extent item

2016-08-30 Thread Qu Wenruo
From: Lukas Lueg Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- tests/fuzz-tests/images/unaligned-extent-item.raw.txt | 8 tests/fuzz-tests/images/unaligned-extent-item.raw.xz | Bin 0 -> 3684 bytes 2 files changed, 8 insertions(+) create mode 100644 tests/fuzz-te

[PATCH 3/5] btrfs-progs: fsck: Check bytenr alignment for extent item

2016-08-30 Thread Qu Wenruo
Check bytenr alignment for extent item to filter invalid items early. Signed-off-by: Qu Wenruo --- cmds-check.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/cmds-check.c b/cmds-check.c index 2aa0a7b..c56b176 100644 --- a/cmds-check.c +++ b/cmds-check.c @@ -5422,6

[PATCH 2/5] btrfs-progs: fuzz-test: Add test case for invalid drop level

2016-08-30 Thread Qu Wenruo
From: Lukas Lueg Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- tests/fuzz-tests/images/invalid-drop-level.raw.txt | 30 + tests/fuzz-tests/images/invalid-drop-level.raw.xz | Bin 0 -> 3788 bytes 2 files changed, 30 insertions(+) create mode 100644 tests/f

[PATCH 4/5] btrfs-progs: fsck: Avoid abort and BUG_ON in add_tree_backref

2016-08-30 Thread Qu Wenruo
Add_tree_backref() can cause BUG_ON() and abort() in quite a lot of cases, from the ENOMEM to existing tree backref records. Change all these BUG_ON() and abort() to return proper values. And modify all callers to handle such problems. Reported-by: Lukas Lueg Signed-off-by: Qu Wenruo --- cmds

Re: [PATCH] btrfs-progs: fuzz-test: Add image for wrong chunk item in root tree

2016-08-30 Thread Qu Wenruo
At 08/30/2016 10:17 PM, David Sterba wrote: On Tue, Aug 30, 2016 at 10:15:50AM +0800, Qu Wenruo wrote: From: Lukas Lueg Reported by Lukas and the same image from him. DATA_RELOC tree's key type is modifed to CHUNK_ITEM, causing btrfsck interpret it as CHUNK_ITEM and cause 0 num_st

Re: btrfs send extremely slow (almost stuck)

2016-08-30 Thread Qu Wenruo
At 08/31/2016 09:35 AM, Jeff Mahoney wrote: On 8/28/16 10:12 PM, Qu Wenruo wrote: At 08/29/2016 10:11 AM, Qu Wenruo wrote: At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to

Re: [PATCH] Btrfs: fix endless loop in balancing block groups

2016-08-31 Thread Qu Wenruo
At 09/01/2016 07:43 AM, Liu Bo wrote: Qgroup function may overwrite the saved error 'err' with 0 in case quota is not enabled, and this ends up with a endless loop in balance because we keep going back to balance the same block group. In which case? If join_trans() fails, we won't go through

Re: [PATCH] Btrfs: fix endless loop in balancing block groups

2016-08-31 Thread Qu Wenruo
At 09/01/2016 08:54 AM, Liu Bo wrote: On Thu, Sep 01, 2016 at 08:32:00AM +0800, Qu Wenruo wrote: At 09/01/2016 07:43 AM, Liu Bo wrote: Qgroup function may overwrite the saved error 'err' with 0 in case quota is not enabled, and this ends up with a endless loop in balance becau

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-09-01 Thread Qu Wenruo
At 09/01/2016 05:44 AM, Chris Murphy wrote: On Wed, Aug 31, 2016 at 2:49 PM, Ronan Arraes Jardim Chagas wrote: Hi guys! And the problem happened again. This time, I was only using Mozilla Firefox. I could get the very first message after the error. I hope it brings more information: [28039.

[PATCH 2/2] btrfs-progs: Doc: Add warning for build RAID btrfs on partions from the same device

2016-09-01 Thread Qu Wenruo
Quite a common sense for any RAID-like multi-device setup, just in case. Signed-off-by: Qu Wenruo --- Documentation/mkfs.btrfs.asciidoc | 4 1 file changed, 4 insertions(+) diff --git a/Documentation/mkfs.btrfs.asciidoc b/Documentation/mkfs.btrfs.asciidoc index 846c08f..c291714 100644

[PATCH 1/2] btrfs-progs: mkfs: Warn user for minimal RAID5/6 devices setup

2016-09-01 Thread Qu Wenruo
For RAID5, 2 devices setup is just RAID1 with more overhead. For RAID6, 3 devices setup is RAID1 with 3 copies, not what most user want. So warn user at mkfs time for such case, and add explain in man pages. Signed-off-by: Qu Wenruo --- Documentation/mkfs.btrfs.asciidoc | 15

Re: btrfs send extremely slow (almost stuck)

2016-09-04 Thread Qu Wenruo
At 09/05/2016 05:41 AM, Oliver Freyermuth wrote: Am 30.08.2016 um 02:48 schrieb Qu Wenruo: Yes. And more specifically, it doesn't even affect delta backup. For shared extents caused by reflink/dedupe(out-of-band or even incoming in-band), it will be send as individual files. For con

[PATCH] fstests: common: Enhance _exclude_scratch_mount_option to handle multiply options and generic fs type

2016-09-05 Thread Qu Wenruo
t mount options for given fs(mainly for btrfs though) Finally, allow it to accept multiple options at the same time. No need for multiple _exclude_scratch_mount_option lines now Signed-off-by: Qu Wenruo --- common/rc | 41 ++--- tests/ext4/271 | 6 ++

Re: does btrfs-receive use/compare the checksums from the btrfs-send side?

2016-09-05 Thread Qu Wenruo
At 09/04/2016 12:29 PM, Christoph Anton Mitterer wrote: On Mon, 2016-08-29 at 16:25 +0800, Qu Wenruo wrote: Send will generate checksum for each command. What does "command" mean here? Or better said how much data is secured with one CRC32? Command is one send command stream,

Re: [PATCH] Btrfs: remove unnecessary code of chunk_root assignment in btrfs_read_chunk_tree.

2016-09-05 Thread Qu Wenruo
At 09/05/2016 09:19 AM, Zhao Lei wrote: Hi, Sean Fu From: Sean Fu [mailto:fxinr...@gmail.com] Sent: Sunday, September 04, 2016 7:54 PM To: dste...@suse.com Cc: c...@fb.com; anand.j...@oracle.com; fdman...@suse.com; zhao...@cn.fujitsu.com; linux-btrfs@vger.kernel.org; linux-ker...@vger.kernel.

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

2016-09-05 Thread Qu Wenruo
Just like what Wang has mentioned, would you please paste all the output of the contents of /sys/fs/btrfs//allocation? It's recommended to use "grep . -IR " to get all the data as it will show the file name. Thanks, Qu At 09/03/2016 03:25 AM, Ronan Arraes Jardim Chagas wrote: Hi guys! Jeff

Re: [PATCH 1/3] btrfs-progs: Introduce new send-dump object

2016-09-05 Thread Qu Wenruo
At 08/24/2016 08:49 PM, David Sterba wrote: On Wed, Aug 17, 2016 at 08:42:48AM +0800, Qu Wenruo wrote: At 08/16/2016 10:31 PM, David Sterba wrote: On Mon, Aug 01, 2016 at 02:29:42PM +0800, Qu Wenruo wrote: Introduce send-dump.[ch] which implements a new btrfs_send_ops to exam and output

Re: btrfs send extremely slow (almost stuck)

2016-09-05 Thread Qu Wenruo
At 09/06/2016 05:29 AM, Oliver Freyermuth wrote: Am 05.09.2016 um 07:21 schrieb Qu Wenruo: Did you get the half way send stream? Luckily, yes! If the send stream has something, please use "--no-data" option to send the subvolume again to get the metadata only dump, and upl

Re: [PATCH] fstests: common: Enhance _exclude_scratch_mount_option to handle multiply options and generic fs type

2016-09-05 Thread Qu Wenruo
At 09/06/2016 12:20 PM, Eryu Guan wrote: On Mon, Sep 05, 2016 at 03:13:33PM +0800, Qu Wenruo wrote: Enhance _exclude_scratch_mount_option() function to get real mount options from $MOUNT_OPTIONS. This seems unnecessarily complex to me. Considering the fact that we didn't specify any

[PATCH 0/3] Introduce dump option for btrfs-receive

2016-09-06 Thread Qu Wenruo
v2: Move from inspect subcommand to receive subcommand. Qu Wenruo (3): btrfs-progs: Introduce new send-dump object btrfs-progs: receive: Introduce option to exam and dump send stream btrfs-progs: Remove send-test tool Documentation/btrfs-receive.asciidoc | 17 +- Makefile.in

[PATCH 1/3] btrfs-progs: Introduce new send-dump object

2016-09-06 Thread Qu Wenruo
dump-send" command. Signed-off-by: Qu Wenruo --- Makefile.in | 2 +- send-dump.c | 367 send-dump.h | 24 3 files changed, 392 insertions(+), 1 deletion(-) create mode 100644 send-dump.c create mode 100644 send-dump.h di

[PATCH 3/3] btrfs-progs: Remove send-test tool

2016-09-06 Thread Qu Wenruo
Since new "receive --dump" has better output and structure, it's time to remove old and function-weak send-test tool. Signed-off-by: Qu Wenruo --- Makefile.in | 6 +- send-test.c | 447 2 files changed, 1 insertion(+)

[PATCH v2 2/3] btrfs-progs: receive: Introduce option to exam and dump send stream

2016-09-06 Thread Qu Wenruo
Introduce new option, '--dump' for receive subcommand. With this command, user can exam and dump the metadata info of a send stream. Which is quite useful for education purpose or bug reporting. Signed-off-by: Qu Wenruo --- Documentation/btrfs-receive.asciidoc | 17 +++

Re: [PATCH] Btrfs: remove unnecessary code of chunk_root assignment in btrfs_read_chunk_tree.

2016-09-06 Thread Qu Wenruo
At 09/07/2016 09:38 AM, Sean Fu wrote: On Mon, Sep 05, 2016 at 03:56:41PM +0800, Qu Wenruo wrote: At 09/05/2016 09:19 AM, Zhao Lei wrote: Hi, Sean Fu From: Sean Fu [mailto:fxinr...@gmail.com] Sent: Sunday, September 04, 2016 7:54 PM To: dste...@suse.com Cc: c...@fb.com; anand.j

[PATCH] btrfs-progs: qgroup: Fix regression leads to corrupted qgroup status

2016-09-06 Thread Qu Wenruo
correct path->slot[0]. This leads to wrong slot[0] and read_qgroup_status() will read out wrong flags, leading to regression. Fix read_qgroup_status() by using eb and slot instread of wrong path strucutre. Reported-by: Tsutomu Itoh Cc: Mark Fasheh Signed-off-by: Qu Wenruo --- qgroup-verify.c

[PATCH v2] fstests: common: Enhance _exclude_scratch_mount_option to handle multiple options

2016-09-07 Thread Qu Wenruo
t multiple options at the same time. No need for multiple _exclude_scratch_mount_option lines now Signed-off-by: Qu Wenruo --- changelog: v2: Don't introduce new 'fstype' parameter, suggested by Dave and Eryu. Use easier grep -w method, suggested by Dave and Eryu.

[PATCH v13 09/15] btrfs: ordered-extent: Add support for dedupe

2016-09-08 Thread Qu Wenruo
From: Wang Xiaoguang Add ordered-extent support for dedupe. Note, current ordered-extent support only supports non-compressed source extent. Support for compressed source extent will be added later. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik --- fs

[PATCH v13 01/15] btrfs: expand btrfs_set_extent_delalloc() and its friends to support in-band dedupe and subpage size patchset

2016-09-08 Thread Qu Wenruo
Extract btrfs_set_extent_delalloc() and extent_clear_unlock_delalloc() parameters for both in-band dedupe and subpage sector size patchset. This should reduce conflict of both patchset and the effort to rebase them. Cc: Chandan Rajendra Cc: David Sterba Signed-off-by: Qu Wenruo --- fs/btrfs

[PATCH v13 12/15] btrfs: relocation: Enhance error handling to avoid BUG_ON

2016-09-08 Thread Qu Wenruo
ned -ENOENT in relocate_tree_block() and continue balancing. Reported-by: Satoru Takeuchi Signed-off-by: Qu Wenruo --- fs/btrfs/relocation.c | 22 +- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c index 6e8086a..6ae287f 1

[PATCH v13 06/15] btrfs: delayed-ref: Add support for increasing data ref under spinlock

2016-09-08 Thread Qu Wenruo
For in-band dedupe, btrfs needs to increase data ref with delayed_ref locked, so add a new function btrfs_add_delayed_data_ref_lock() to increase extent ref with delayed_refs already locked. Signed-off-by: Qu Wenruo Reviewed-by: Josef Bacik --- fs/btrfs/delayed-ref.c | 30

[PATCH v13 05/15] btrfs: dedupe: Introduce function to remove hash from in-memory tree

2016-09-08 Thread Qu Wenruo
possible race. Cc: Mark Fasheh Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang --- fs/btrfs/dedupe.c | 132 +++--- 1 file changed, 126 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/dedupe.c b/fs/btrfs/dedupe.c index e51412b..14c57fa

  1   2   3   4   5   6   7   8   9   10   >