Re: [PATCH 0/4] 3- and 4- copy RAID1

2018-07-16 Thread Goffredo Baroncelli
On 07/15/2018 04:37 PM, waxhead wrote: > David Sterba wrote: >> An interesting question is the naming of the extended profiles. I picked >> something that can be easily understood but it's not a final proposal. >> Years ago, Hugo proposed a naming scheme that described the >> non-standard raid

Re: [PATCH 1/3] btrfs-progs: check: orig: Don't panic out when unexpected root item is referring to one extent

2018-07-16 Thread David Sterba
On Thu, Jul 05, 2018 at 03:45:56PM +0800, Qu Wenruo wrote: > With crafted image, expected root item can refer to certain extent, and > original mode uses BUG_ON() to handle such case. > > Fix it by gracefully return error. > > Link: https://bugzilla.kernel.org/show_bug.cgi?id=200403 >

Re: [PATCH RESEND 2/2] btrfs-progs: make all programs and libraries optional

2018-07-16 Thread Omar Sandoval
On Mon, Jul 16, 2018 at 04:56:57PM +0200, David Sterba wrote: > On Thu, Jul 12, 2018 at 04:11:19PM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > We have a build system internally which only needs to build the > > libraries out of a repository, not any binaries. I looked at how this

Re: [PATCH] btrfs-progs: tests/fuzz: Use correct suffix for raw image

2018-07-16 Thread David Sterba
On Thu, Jul 05, 2018 at 01:41:24PM +0800, Qu Wenruo wrote: > * Please fold this patch into original commit in devel branch * > > It's raw image, so it should be ".raw.xz" not ".img.xz" Ah right, thanks, commit updated. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

Re: [PATCH 1/2] btrfs-progs: lowmem: fix false alerts of referencer count mismatch for blocks relocated

2018-07-16 Thread David Sterba
On Tue, Jul 03, 2018 at 03:58:50PM +0800, Su Yue wrote: > Btrfs lowmem check reports such false alerts: ... 1 and 2 applied, thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: [PATCH 2/2] btrfs-progs: check: enhanced progress indicator

2018-07-16 Thread David Sterba
On Wed, Jul 11, 2018 at 01:18:55PM +0800, Qu Wenruo wrote: > On 2018年07月05日 03:20, Stéphane Lesimple wrote: > > We reuse the task_position enum and task_ctx struct of the original progress > > indicator, adding more values and fields for our needs. > > > > Then add hooks in all steps of the check

Healthy amount of free space?

2018-07-16 Thread Wolf
Greetings, I would like to ask what what is healthy amount of free space to keep on each device for btrfs to be happy? This is how my disk array currently looks like [root@dennas ~]# btrfs fi usage /raid Overall: Device size: 29.11TiB Device allocated:

Re: [PATCH 0/4] 3- and 4- copy RAID1

2018-07-16 Thread waxhead
waxhead wrote: David Sterba wrote: An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme that described the non-standard raid varieties of the btrfs flavor:

Re: [PATCH] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Qu Wenruo
On 2018年07月16日 13:55, Su Yue wrote: > For big filesystems, there are many items in trees(extent tree > specially). > For example, to dump one extent, we usually dump extent tree then pipe > result to grep. The time-consuming part is that dump tree traverses > items. And it eats cpu and memory

Re: [PATCH] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Su Yue
On 07/16/2018 03:45 PM, Qu Wenruo wrote: On 2018年07月16日 13:55, Su Yue wrote: For big filesystems, there are many items in trees(extent tree specially). For example, to dump one extent, we usually dump extent tree then pipe result to grep. The time-consuming part is that dump tree traverses

Re: Corrupted FS with "open_ctree failed" and "failed to recover balance: -5"

2018-07-16 Thread Qu Wenruo
On 2018年07月16日 16:15, Udo Waechter wrote: > Hello, > > noone any ideas? Do you need more information? > > Cheers, > udo. > > On 11/07/18 17:37, Udo Waechter wrote: >> Hello everyone, >> >> I have a corrupted filesystem which I can't seem to recover. >> >> The machine is: >> Debian Linux,

Re: [PATCH] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Qu Wenruo
On 2018年07月16日 16:14, Su Yue wrote: > > > On 07/16/2018 03:45 PM, Qu Wenruo wrote: >> >> >> On 2018年07月16日 13:55, Su Yue wrote: >>> For big filesystems, there are many items in trees(extent tree >>> specially). >>> For example, to dump one extent, we usually dump extent tree then pipe >>>

Re: Corrupted FS with "open_ctree failed" and "failed to recover balance: -5"

2018-07-16 Thread Udo Waechter
Hello, noone any ideas? Do you need more information? Cheers, udo. On 11/07/18 17:37, Udo Waechter wrote: > Hello everyone, > > I have a corrupted filesystem which I can't seem to recover. > > The machine is: > Debian Linux, kernel 4.9 and btrfs-progs v4.13.3 > > I have a HDD RAID5 with LVM

Re: [PATCH v2] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Qu Wenruo
On 2018年07月16日 17:13, Su Yue wrote: > For big filesystems, there are many items in trees(extent tree > specially). > For example, to dump one extent, we usually dump extent tree then pipe > result to grep. The time-consuming part is that dump tree traverses > items. And it eats cpu and memory

[PATCH v2] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread Su Yue
For big filesystems, there are many items in trees(extent tree specially). For example, to dump one extent, we usually dump extent tree then pipe result to grep. The time-consuming part is that dump tree traverses items. And it eats cpu and memory too. This patch introduces an option '-k

[PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-16 Thread Gu Jinxiang
Instead of pointer to btrfs_fs_devices as an arg in btrfs_scan_one_device, better to make it as a return value. And since btrfs_fs_devices can be get by btrfs_device, better to return btrfs_device than fs_btrfs_devices. Signed-off-by: Gu Jinxiang --- Changelog: v5: rebase to misc-next. v4: as

[PATCH v5 1/2] btrfs: make fs_devices to be a local variable

2018-07-16 Thread Gu Jinxiang
fs_devices is always passed to btrfs_scan_one_device which overrides it. And in the call stack below fs_devices is passed to btrfs_scan_one_device from btrfs_mount_root. And in btrfs_mount_root the output fs_devices of this call stack is not used. btrfs_mount_root ->

[PATCH v5 0/2] btrfs: do some parameters optimization

2018-07-16 Thread Gu Jinxiang
Do parameters optimization for function btrfs_parse_early_options and btrfs_scan_one_device. Gu Jinxiang (2): btrfs: make fs_devices to be a local variable btrfs: get device pointer from btrfs_scan_one_device fs/btrfs/super.c | 38 -- fs/btrfs/volumes.c

Re: [PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-16 Thread Anand Jain
:: @@ -1561,12 +1564,15 @@ static struct dentry *btrfs_mount_root(struct file_system_type *fs_type, goto error_fs_info; } - error = btrfs_scan_one_device(device_name, mode, fs_type, _devices); - if (error) { + device =

RE: [PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-16 Thread Gu, Jinxiang
> -Original Message- > From: David Sterba [mailto:dste...@suse.cz] > Sent: Monday, July 16, 2018 8:34 PM > To: Gu, Jinxiang/顾 金香 > Cc: linux-btrfs@vger.kernel.org; anand.j...@oracle.com > Subject: Re: [PATCH v5 2/2] btrfs: get device pointer from > btrfs_scan_one_device > > On Mon,

Re: [PATCH 0/2] btrfs-progs: check: enhanced progress indicator

2018-07-16 Thread David Sterba
On Wed, Jul 04, 2018 at 09:20:12PM +0200, Stéphane Lesimple wrote: > This patch replaces the current ".oOo."-style progress indicator of > btrfs check (-p), by a more helpful live indication of the check progress. > I've been using it on a custom btrfs-progs version since 2015. > > Here's how the

Re: [PATCH 0/4] 3- and 4- copy RAID1

2018-07-16 Thread Austin S. Hemmelgarn
On 2018-07-16 14:29, Goffredo Baroncelli wrote: On 07/15/2018 04:37 PM, waxhead wrote: David Sterba wrote: An interesting question is the naming of the extended profiles. I picked something that can be easily understood but it's not a final proposal. Years ago, Hugo proposed a naming scheme

Re: [PATCH v2] fstests: btrfs/168 verify device ready after device delete

2018-07-16 Thread Eryu Guan
On Fri, Jul 06, 2018 at 02:01:37PM +0800, Anand Jain wrote: > This test case verifies if the device ready return success after the > device delete. > > Signed-off-by: Anand Jain Need some helps from btrfs folks to see if it's a valid test. Thanks! Eryu > --- > v1->v2: use _run_btrfs_util_prog

Re: [PATCH v2] btrfs: Use btrfs_mark_bg_unused() to replace open code

2018-07-16 Thread David Sterba
On Sun, May 27, 2018 at 09:25:25AM +0800, Qu Wenruo wrote: > > > On 2018年05月22日 20:14, David Sterba wrote: > > On Tue, May 22, 2018 at 04:43:47PM +0800, Qu Wenruo wrote: > >> Introduce a small helper, btrfs_mark_bg_unused(), to accquire needed > >> locks and add a block group to unused_bgs list.

Re: [PATCH v5 1/2] btrfs: make fs_devices to be a local variable

2018-07-16 Thread Nikolay Borisov
On 16.07.2018 12:11, Gu Jinxiang wrote: > fs_devices is always passed to btrfs_scan_one_device which > overrides it. And in the call stack below fs_devices is passed to > btrfs_scan_one_device from btrfs_mount_root. > And in btrfs_mount_root the output fs_devices of this call stack > is not

Re: [PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-16 Thread David Sterba
On Mon, Jul 16, 2018 at 05:11:17PM +0800, Gu Jinxiang wrote: > Instead of pointer to btrfs_fs_devices as an arg in > btrfs_scan_one_device, better to make it as a return value. > > And since btrfs_fs_devices can be get by btrfs_device, > better to return btrfs_device than fs_btrfs_devices. > >

Re: [PATCH v2] btrfs: Rewrite retry logic in do_chunk_alloc

2018-07-16 Thread David Sterba
On Wed, Apr 18, 2018 at 10:27:57AM +0300, Nikolay Borisov wrote: > do_chunk_alloc implements logic to detect whether there is currently > pending chunk allocation (by means of space_info->chunk_alloc being > set) and if so it loops around to the 'again' label. Additionally, > based on the state

[PATCH] btrfs: rename btrfs_parse_early_options

2018-07-16 Thread Anand Jain
Rename btrfs_parse_early_options() to btrfs_parse_device_options(). As btrfs_parse_early_options() parses the -o device options and scan the device provided. So this rename specifies its action. Also the function name is inline with btrfs_parse_subvol_options(). No functional changes.

Re: [PATCH 2nd try v4] btrfs: drop uuid_mutex in btrfs_free_extra_devids()

2018-07-16 Thread Anand Jain
On 05/29/2018 07:19 PM, David Sterba wrote: On Tue, May 29, 2018 at 01:40:34PM +0800, Anand Jain wrote: v3->v4: As we traverse through the seed device, fs_device gets updated with the child seed fs_devices, so make sure we use the same fs_devices pointer for the mutex_unlock as

Re: [PATCH 1/2] btrfs: Introduce mount time chunk <-> dev extent mapping check

2018-07-16 Thread David Sterba
On Mon, Jul 09, 2018 at 02:42:02PM +0800, Qu Wenruo wrote: > This patch will introduce chunk <-> dev extent mapping check, to protect > us against invalid dev extents or chunks. > > Since chunk mapping is the fundamental infrastructure of btrfs, extra > check at mount time could prevent a lot of

Re: [PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-16 Thread Qu Wenruo
On 2018年07月16日 21:16, David Sterba wrote: > On Fri, Jul 06, 2018 at 07:44:37AM +0800, Qu Wenruo wrote: >> >> >> On 2018年07月05日 23:18, David Sterba wrote: >>> On Tue, Jul 03, 2018 at 05:10:09PM +0800, Qu Wenruo wrote: If a crafted btrfs has missing block group items, it could cause

Re: [PATCH 0/2] btrfs: chunk and dev-extent related error handler enhancement

2018-07-16 Thread David Sterba
On Mon, Jul 09, 2018 at 02:42:01PM +0800, Qu Wenruo wrote: > Can be fetched with all existing tree-checker/bg<->chunk error detector > from github: > https://github.com/adam900710/linux/tree/tree_checker_enhance > > Still some fuzzed images reported from Xu Wen. > This time, 2 can be fixed by

Re: [PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-16 Thread David Sterba
On Fri, Jul 06, 2018 at 07:44:37AM +0800, Qu Wenruo wrote: > > > On 2018年07月05日 23:18, David Sterba wrote: > > On Tue, Jul 03, 2018 at 05:10:09PM +0800, Qu Wenruo wrote: > >> If a crafted btrfs has missing block group items, it could cause > >> unexpected behavior and breaks our expectation on

Re: [PATCH RESEND 1/2] btrfs-progs: remove stale dir-test and quick-test

2018-07-16 Thread David Sterba
On Thu, Jul 12, 2018 at 04:11:18PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > These don't build anymore and don't appear to be used for anything. I've fixed build of quick-test and will keep the code for now, it can be used for some stress testing. The dir-test does not seem to be

[PATCH 3/7] btrfs: do device clone using the btrfs_scan_one_device

2018-07-16 Thread Anand Jain
When we add a device to the RO mounted seed device, it becomes a RW sprout FS. The following steps are used to hold the seed and sprout fs_devices. (first two steps are not mandatory for the sprouting, they are there to ensure the seed device remains in the scanned state) . Clone the

[PATCH v4 1/7] btrfs: drop uuid_mutex in btrfs_free_extra_devids()

2018-07-16 Thread Anand Jain
btrfs_free_extra_devids() is called only in the mount context which traverses through the fs_devices::devices and frees the orphan devices devices in the given %fs_devices if any. As the search for the orphan device is limited to fs_devices::devices so we don't need the global uuid_mutex. There

[PATCH v2 2/7] btrfs: fix race between free_stale_devices and close_fs_devices

2018-07-16 Thread Anand Jain
From: Anand Jain %fs_devices can be free-ed by btrfs_free_stale_devices() when the close_fs_devices() drops fs_devices::opened to zero, but close_fs_devices tries to access the %fs_devices again without the device_list_mutex. Fix this by bringing the %fs_devices access with in the

[PATCH 0/7] Misc volume patch set part2

2018-07-16 Thread Anand Jain
As the last set if the pull wasn't integrated due to its pending review corrections. Here, I have done them (not much except for adding comments and function renames) and so sending it again. Also this set clubs other independent patches which are in the mailing list. I am explaining the changes

[PATCH 4/7] btrfs: use the assigned fs_devices instead of the dereference

2018-07-16 Thread Anand Jain
We have assigned the %fs_info->fs_devices in %fs_devices as its not modified just use it for the mutex_lock(). Signed-off-by: Anand Jain --- I don't see this in the ML at all, I remember sending this. Anyway I am marking this as V1. fs/btrfs/volumes.c | 4 ++-- 1 file changed, 2 insertions(+),

[PATCH 5/7] btrfs: warn for num_devices below 0

2018-07-16 Thread Anand Jain
In preparation to de-duplicate a section of code where we deduce the num_devices, use warn instead of bug. Signed-off-by: Anand Jain --- fs/btrfs/volumes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index

[PATCH 6/7] btrfs: add helper btrfs_num_devices() to deduce num_devices

2018-07-16 Thread Anand Jain
When the replace is running the fs_devices::num_devices also includes the replace device, however in some operations like device delete and balance it needs the actual num_devices without the repalce devices, so now the function btrfs_num_devices() just provides that. Signed-off-by: Anand Jain

[PATCH 7/7] btrfs: add helper function check device delete able

2018-07-16 Thread Anand Jain
Move the section of the code which performs the check if the device is indelible, move that into a helper function. Signed-off-by: Anand Jain --- v1->v2: Rename function to btrfs_get_device_for_delete(), thanks Nikolay. fs/btrfs/volumes.c | 49 ++---

Re: [PATCH RESEND 2/2] btrfs-progs: make all programs and libraries optional

2018-07-16 Thread David Sterba
On Thu, Jul 12, 2018 at 04:11:19PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > We have a build system internally which only needs to build the > libraries out of a repository, not any binaries. I looked at how this > works with other projects, and the best example was util-linux, which

Re: [PATCH] btrfs: rename btrfs_parse_early_options

2018-07-16 Thread Anand Jain
On 07/16/2018 10:18 PM, Anand Jain wrote: Rename btrfs_parse_early_options() to btrfs_parse_device_options(). As btrfs_parse_early_options() parses the -o device options and scan the device provided. So this rename specifies its action. Also the function name is inline with

Re: [PATCH 00/15] Add delayed-refs support to btrfs-progs

2018-07-16 Thread David Sterba
On Fri, Jun 08, 2018 at 03:47:43PM +0300, Nikolay Borisov wrote: > Hello, > > > > Here is a series which adds support for delayed refs. This

Re: [PATCH 0/7] Misc volume patch set part2

2018-07-16 Thread Anand Jain
On 07/16/2018 10:58 PM, Anand Jain wrote: As the last set if the pull wasn't integrated due to its pending review corrections. Here, I have done them (not much except for adding comments and function renames) and so sending it again. Also this set clubs other independent patches which are in

Re: [PATCH 1/2] btrfs-progs: Exit gracefully when overlap chunks are detected

2018-07-16 Thread David Sterba
On Mon, Jul 09, 2018 at 02:50:53PM +0800, Qu Wenruo wrote: > BUG_ON() can be triggered if some image contains overlap chunks. > volumes.c:1930: read_one_chunk: BUG_ON `ret` triggered, value -17 > btrfs(+0x2cf12)[0x5601efa17f12] > btrfs(+0x2fd8b)[0x5601efa1ad8b] >

Re: [PATCH v2] btrfs-progs: inspect-internal: add option '-k u64,u8,u64' of dump-tree

2018-07-16 Thread David Sterba
On Mon, Jul 16, 2018 at 05:13:25PM +0800, Su Yue wrote: > For big filesystems, there are many items in trees(extent tree > specially). > For example, to dump one extent, we usually dump extent tree then pipe > result to grep. The time-consuming part is that dump tree traverses > items. And it eats

Re: [PATCH 2/2] btrfs-progs: check: enhanced progress indicator

2018-07-16 Thread Misono Tomohiro
On 2018/07/05 4:20, Stéphane Lesimple wrote: > We reuse the task_position enum and task_ctx struct of the original progress > indicator, adding more values and fields for our needs. > > Then add hooks in all steps of the check to properly record progress. > > Signed-off-by: Stéphane Lesimple >