[PULL] uuid_mutex fixes and cleanups part2

2018-07-17 Thread Anand Jain
These patches we sent independently before and are in the mailing list. They have been tested successfully using the xfstests. The cyclical lockdep warning aren't due to these set of patches and they (2) can be ignored as they are harmless because the threads involved are ioctl/commit and mount

[PATCH FIXED v7] Add cli and ioctl to forget scanned device(s)

2018-07-17 Thread Anand Jain
[ Left out changes is now commited ]. v7: Availalbe for pull from btrfs-progs: g...@github.com:asj/btrfs-progs.git forget btrfs.ko: g...@github.com:asj/btrfs-devel.git misc-next-jul18 Use struct btrfs_ioctl_vol_args (instead of struct btrfs_ioctl_vol_args_v2) as its inline with

[PATCH FIXED v7] btrfs: introduce feature to forget a btrfs device

2018-07-17 Thread Anand Jain
Support for a new command 'btrfs dev forget [dev]' is proposed here, to undo the effects of 'btrfs dev scan [dev]'. For this purpose, this patch proposes to use ioctl #5 as it was empty. IOW(BTRFS_IOCTL_MAGIC, 5, ..) This patch adds new ioctl BTRFS_IOC_FORGET_DEV which can be sent from the

[PATCH FIXED v7] btrfs-progs: add cli to forget one or all scanned devices

2018-07-17 Thread Anand Jain
This patch adds cli btrfs device forget [dev] to remove the given device structure in the kernel if the device is unmounted. If no argument is given it shall remove all stale (device which are not mounted) from the kernel. Signed-off-by: Anand Jain --- cmds-device.c | 58

Re: [PATCH v7] Add cli and ioctl to forget scanned device(s)

2018-07-17 Thread Anand Jain
Please ignore this. There is a line of code which is un-commit. I am sending this series again. Sorry for the noise. Thanks, Anand On 07/18/2018 11:07 AM, Anand Jain wrote: v7: Availalbe for pull from btrfs-progs: g...@github.com:asj/btrfs-progs.git forget btrfs.ko:

[PATCH v7] Add cli and ioctl to forget scanned device(s)

2018-07-17 Thread Anand Jain
v7: Availalbe for pull from btrfs-progs: g...@github.com:asj/btrfs-progs.git forget btrfs.ko: g...@github.com:asj/btrfs-devel.git misc-next-jul18 Use struct btrfs_ioctl_vol_args (instead of struct btrfs_ioctl_vol_args_v2) as its inline with other ioctl btrfs-control The CLI usage

[PATCH v7] btrfs-progs: add cli to forget one or all scanned devices

2018-07-17 Thread Anand Jain
This patch adds cli btrfs device forget [dev] to remove the given device structure in the kernel if the device is unmounted. If no argument is given it shall remove all stale (device which are not mounted) from the kernel. Signed-off-by: Anand Jain --- cmds-device.c | 58

[PATCH v7] btrfs: introduce feature to forget a btrfs device

2018-07-17 Thread Anand Jain
Support for a new command 'btrfs dev forget [dev]' is proposed here, to undo the effects of 'btrfs dev scan [dev]'. For this purpose, this patch proposes to use ioctl #5 as it was empty. IOW(BTRFS_IOCTL_MAGIC, 5, ..) This patch adds new ioctl BTRFS_IOC_FORGET_DEV which can be sent from the

Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)

2018-07-17 Thread Marc MERLIN
On Wed, Jul 18, 2018 at 08:05:51AM +0800, Qu Wenruo wrote: > No OOM triggers? That's a little strange. > Maybe it's related to how kernel handles memory over-commit? Yes, I think you are correct. > And for the hang, I think it's related to some memory allocation failure > and error handler just

Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)

2018-07-17 Thread Qu Wenruo
On 2018年07月18日 04:59, Marc MERLIN wrote: > Ok, I did more testing. Qu is right that btrfs check does not crash the > kernel. > It just takes all the memory until linux hangs everywhere, and somehow (no > idea why) > the OOM killer never triggers. No OOM triggers? That's a little strange.

[PATCH resend 1/2] btrfs: allow defrag on a file opened ro that has rw permissions

2018-07-17 Thread Adam Borowski
Requiring a rw descriptor conflicts both ways with exec, returning ETXTBSY whenever you try to defrag a program that's currently being run, or causing intermittent exec failures on a live system being defragged. As defrag doesn't change the file's contents in any way, there's no reason to

[PATCH resend 2/2] btrfs: defrag: return EPERM not EINVAL when only permissions fail

2018-07-17 Thread Adam Borowski
We give EINVAL when the request is invalid; here it's ok but merely the user has insufficient privileges. Thus, this return value reflects the error better -- as discussed in the identical case for dedupe. According to codesearch.debian.net, no userspace program distinguishes these values beyond

[PATCH resend 0/2] btrfs: fix races between exec and defrag

2018-07-17 Thread Adam Borowski
Hi! Here's a ping for a patch to fix ETXTBSY races between defrag and exec, just like the dedupe counterpart. Unlike that one which is shared to multiple filesystems and thus lives in Al Viro's land, it is btrfs only. Attached: a simple tool to fragment a file, by ten O_SYNC rewrites of length 1

Re: [PATCH 0/4] 3- and 4- copy RAID1

2018-07-17 Thread Duncan
Goffredo Baroncelli posted on Mon, 16 Jul 2018 20:29:46 +0200 as excerpted: > On 07/15/2018 04:37 PM, waxhead wrote: > Striping and mirroring/pairing are orthogonal properties; mirror and > parity are mutually exclusive. I can't agree. I don't know whether you meant that in the global sense,

Re: btrfs check (not lowmem) and OOM-like hangs (4.17.6)

2018-07-17 Thread Marc MERLIN
Ok, I did more testing. Qu is right that btrfs check does not crash the kernel. It just takes all the memory until linux hangs everywhere, and somehow (no idea why) the OOM killer never triggers. Details below: On Tue, Jul 17, 2018 at 01:32:57PM -0700, Marc MERLIN wrote: > Here is what I got

btrfs check (not lowmem) and OOM-like hangs (4.17.6)

2018-07-17 Thread Marc MERLIN
On Tue, Jul 17, 2018 at 10:50:32AM -0700, Marc MERLIN wrote: > I got the following on 4.17.6 while running btrfs check --repair on an > unmounted filesystem (not the lowmem version) > > I understand that btrfs check is userland only, although it seems that > it caused these FS hangs on a

Re: [RESEND][PATCH v4 0/2] vfs: better dedupe permission check

2018-07-17 Thread Mark Fasheh
CCing Michael Kerrisk and linux-api The patch at the end of this e-mail updates our man page for ioctl_fideduperange to reflect the changes proposed in this patch series: https://marc.info/?l=linux-fsdevel=153185457324037=2 Any feedback would be appreciated. Thanks in advance. --Mark

Re: [RESEND][PATCH v4 0/2] vfs: better dedupe permission check

2018-07-17 Thread Mark Fasheh
On Tue, Jul 17, 2018 at 12:48:18PM -0700, Darrick J. Wong wrote: > On Tue, Jul 17, 2018 at 12:09:04PM -0700, Mark Fasheh wrote: > > From: Mark Fasheh > > > > [PATCH] ioctl_fideduperange.2: clarify permission requirements > > > > dedupe permission checks were recently relaxed - update our man

Re: [RESEND][PATCH v4 0/2] vfs: better dedupe permission check

2018-07-17 Thread Darrick J. Wong
On Tue, Jul 17, 2018 at 12:09:04PM -0700, Mark Fasheh wrote: > Hi Al, > > The following patches fix a couple of issues with the permission check > we do in vfs_dedupe_file_range(). I sent them out for a few times now, > a changelog is attached. If they look ok to you, I'd appreciate them > being

[PATCH v4 1/2] vfs: allow dedupe of user owned read-only files

2018-07-17 Thread Mark Fasheh
The permission check in vfs_dedupe_file_range() is too coarse - We only allow dedupe of the destination file if the user is root, or they have the file open for write. This effectively limits a non-root user from deduping their own read-only files. In addition, the write file descriptor that the

[PATCH v4 2/2] vfs: dedupe should return EPERM if permission is not granted

2018-07-17 Thread Mark Fasheh
Right now we return EINVAL if a process does not have permission to dedupe a file. This was an oversight on my part. EPERM gives a true description of the nature of our error, and EINVAL is already used for the case that the filesystem does not support dedupe. Signed-off-by: Mark Fasheh

[RESEND][PATCH v4 0/2] vfs: better dedupe permission check

2018-07-17 Thread Mark Fasheh
Hi Al, The following patches fix a couple of issues with the permission check we do in vfs_dedupe_file_range(). I sent them out for a few times now, a changelog is attached. If they look ok to you, I'd appreciate them being pushed upstream. You can get them from git if you like: git pull

Re: Healthy amount of free space?

2018-07-17 Thread Martin Steigerwald
Nikolay Borisov - 17.07.18, 10:16: > On 17.07.2018 11:02, Martin Steigerwald wrote: > > Nikolay Borisov - 17.07.18, 09:20: > >> On 16.07.2018 23:58, Wolf wrote: > >>> Greetings, > >>> I would like to ask what what is healthy amount of free space to > >>> keep on each device for btrfs to be happy?

task btrfs-transacti:921 blocked for more than 120 seconds during check repair

2018-07-17 Thread Marc MERLIN
I got the following on 4.17.6 while running btrfs check --repair on an unmounted filesystem (not the lowmem version) I understand that btrfs check is userland only, although it seems that it caused these FS hangs on a different filesystem (the trace of course does not provide info on which FS)

Re: [PATCH 2/4] btrfs: add support for 3-copy replication (raid1c3)

2018-07-17 Thread David Sterba
On Fri, Jul 13, 2018 at 11:02:03PM +0200, Goffredo Baroncelli wrote: > As general comment, good to hear that something is moving around raid5/6 + > write hole and multiple mirroring. > However I am guessing if this is time to simplify the RAID code. There are a > lot of "if" which could be

Re: [PATCH] btrfs: rename btrfs_parse_early_options

2018-07-17 Thread David Sterba
On Mon, Jul 16, 2018 at 11:01:37PM +0800, Anand Jain wrote: > On 07/16/2018 10:18 PM, Anand Jain wrote: > > Rename btrfs_parse_early_options() to btrfs_parse_device_options(). As > > btrfs_parse_early_options() parses the -o device options and scan the > > device provided. So this rename specifies

Re: [PATCH] btrfs: qgroup: cleanup the unused srcroot from btrfs_qgroup_inherit

2018-07-17 Thread David Sterba
On Tue, Jul 17, 2018 at 04:58:22PM +0800, Lu Fengqi wrote: > Since commit 0b246afa62b0 ("btrfs: root->fs_info cleanup, add fs_info > convenience variables"), the srcroot is no longer used to get fs_fino. > In fact, it can be dropped after commit 707e8a071528 ("btrfs: use nodesize > everywhere,

Re: [PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-17 Thread David Sterba
On Tue, Jul 17, 2018 at 12:54:00PM +0800, Anand Jain wrote: > > :: > > >>> @@ -1561,12 +1564,15 @@ static struct dentry *btrfs_mount_root(struct > >>> file_system_type *fs_type, > >>> goto error_fs_info; > >>> } > >>> > >>> - error =

Re: [PATCH v5 2/2] btrfs: get device pointer from btrfs_scan_one_device

2018-07-17 Thread David Sterba
On Tue, Jul 17, 2018 at 01:08:34AM +, Gu, Jinxiang wrote: > > > - device = device_list_add(path, disk_super, _device_added); > > > - if (IS_ERR(device)) { > > > - ret = PTR_ERR(device); > > > - } else { > > > - *fs_devices_ret = device->fs_devices; > > > - if

Re: [PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-17 Thread Qu Wenruo
On 2018年07月17日 20:33, David Sterba wrote: > On Mon, Jul 16, 2018 at 09:57:43PM +0800, Qu Wenruo wrote: > -EUCLEAN ? Either works for me. >>> >>> That's not just a cosmetic change, there's a semantic difference between >>> the error codes, I maybe make that more explicit and not

Re: [PATCH 5/5] btrfs: Verify every chunk has corresponding block group at mount time

2018-07-17 Thread David Sterba
On Mon, Jul 16, 2018 at 09:57:43PM +0800, Qu Wenruo wrote: > >>> -EUCLEAN ? > >> > >> Either works for me. > > > > That's not just a cosmetic change, there's a semantic difference between > > the error codes, I maybe make that more explicit and not expect that this > > is obvious. > > > > ENOENT

Re: [PATCH 2/2] btrfs-progs: check: enhanced progress indicator

2018-07-17 Thread David Sterba
On Tue, Jul 17, 2018 at 10:23:47AM +0900, Misono Tomohiro wrote: > On 2018/07/05 4:20, Stéphane Lesimple wrote: > > @@ -9806,16 +9839,26 @@ int cmd_check(int argc, char **argv) > > error("errors found in csum tree"); > > err |= !!ret; > > > > - fprintf(stderr, "checking root

Re: [PATCH RFC v2 1/2] btrfs: scrub: Don't use inode page cache in scrub_handle_errored_block()

2018-07-17 Thread David Sterba
On Wed, Jul 11, 2018 at 01:41:21PM +0800, Qu Wenruo wrote: > In commit ac0b4145d662 ("btrfs: scrub: Don't use inode pages for device > replace") we removed the branch of copy_nocow_pages() to avoid > corruption for compressed nodatasum extents. > > However above commit only solves the problem in

Re: Healthy amount of free space?

2018-07-17 Thread Austin S. Hemmelgarn
On 2018-07-16 16:58, Wolf wrote: Greetings, I would like to ask what what is healthy amount of free space to keep on each device for btrfs to be happy? This is how my disk array currently looks like [root@dennas ~]# btrfs fi usage /raid Overall: Device size:

Re: [PATCH RESEND 2/2] btrfs-progs: make all programs and libraries optional

2018-07-17 Thread David Sterba
On Mon, Jul 16, 2018 at 10:44:52AM -0700, Omar Sandoval wrote: > On Mon, Jul 16, 2018 at 04:56:57PM +0200, David Sterba wrote: > > On Thu, Jul 12, 2018 at 04:11:19PM -0700, Omar Sandoval wrote: > > > From: Omar Sandoval > > > > > > We have a build system internally which only needs to build the

[PATCH] btrfs: qgroup: cleanup the unused srcroot from btrfs_qgroup_inherit

2018-07-17 Thread Lu Fengqi
Since commit 0b246afa62b0 ("btrfs: root->fs_info cleanup, add fs_info convenience variables"), the srcroot is no longer used to get fs_fino. In fact, it can be dropped after commit 707e8a071528 ("btrfs: use nodesize everywhere, kill leafsize"). Signed-off-by: Lu Fengqi --- fs/btrfs/qgroup.c |

Re: [PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Qu Wenruo
On 2018年07月17日 16:28, Nikolay Borisov wrote: > > > On 17.07.2018 11:24, Qu Wenruo wrote: >> And it's causing problem for certain test cases. >> Please ignore this (at least for now). >> >> But on the other hand, we indeed have a lot of reports on corrupted >> extent tree, it's possible to hit

Re: [PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Qu Wenruo
On 2018年07月17日 16:01, Nikolay Borisov wrote: > > > On 17.07.2018 10:46, Qu Wenruo wrote: >> [BUG] >> For certain fuzzed btrfs image, if we create any csum data, it would >> cause the following kernel warning and deadlock when trying to update >> csum tree: >> -- >> [ 278.113360] WARNING:

Re: Healthy amount of free space?

2018-07-17 Thread Nikolay Borisov
On 17.07.2018 11:02, Martin Steigerwald wrote: > Hi Nikolay. > > Nikolay Borisov - 17.07.18, 09:20: >> On 16.07.2018 23:58, Wolf wrote: >>> Greetings, >>> I would like to ask what what is healthy amount of free space to >>> keep on each device for btrfs to be happy? >>> >>> This is how my disk

Re: [PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Su Yue
On 07/17/2018 04:01 PM, Nikolay Borisov wrote: On 17.07.2018 10:46, Qu Wenruo wrote: [BUG] For certain fuzzed btrfs image, if we create any csum data, it would cause the following kernel warning and deadlock when trying to update csum tree: -- [ 278.113360] WARNING: CPU: 1 PID: 41 at

Re: Healthy amount of free space?

2018-07-17 Thread Martin Steigerwald
Hi Nikolay. Nikolay Borisov - 17.07.18, 09:20: > On 16.07.2018 23:58, Wolf wrote: > > Greetings, > > I would like to ask what what is healthy amount of free space to > > keep on each device for btrfs to be happy? > > > > This is how my disk array currently looks like > > > > [root@dennas

Re: [PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Nikolay Borisov
On 17.07.2018 10:46, Qu Wenruo wrote: > [BUG] > For certain fuzzed btrfs image, if we create any csum data, it would > cause the following kernel warning and deadlock when trying to update > csum tree: > -- > [ 278.113360] WARNING: CPU: 1 PID: 41 at fs/btrfs/locking.c:230 >

Re: Corrupted FS with "open_ctree failed" and "failed to recover balance: -5"

2018-07-17 Thread Udo Waechter
Thanks for the answer. On 16/07/18 10:32, Qu Wenruo wrote: > > > On 2018年07月16日 16:15, Udo Waechter wrote: >>> The weird thing is that I can't really find information about the >>> "failed to recover balance: -5" error. - There was no rebalancing >>> running when during the crash. > > Can only

[PATCH] btrfs: extent-tree: Check if the newly reserved tree block is already in use

2018-07-17 Thread Qu Wenruo
[BUG] For certain fuzzed btrfs image, if we create any csum data, it would cause the following kernel warning and deadlock when trying to update csum tree: -- [ 278.113360] WARNING: CPU: 1 PID: 41 at fs/btrfs/locking.c:230 btrfs_tree_lock+0x3e2/0x400 [ 278.113737] CPU: 1 PID: 41 Comm:

Re: Healthy amount of free space?

2018-07-17 Thread Nikolay Borisov
On 16.07.2018 23:58, Wolf wrote: > Greetings, > I would like to ask what what is healthy amount of free space to keep on > each device for btrfs to be happy? > > This is how my disk array currently looks like > > [root@dennas ~]# btrfs fi usage /raid > Overall: > Device size:

[PATCH 2/2] btrfs-progs: completion: Let dump-tree/dump-super/inode-resolve to accept any file

2018-07-17 Thread Qu Wenruo
For dump-tree/dump-super the completion uses default filedir -d, which is far from convenient. Use filedir for dump-tree/dump-super/inode-resolve just like rootid. Signed-off-by: Qu Wenruo --- btrfs-completion | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/btrfs-completion

[PATCH 1/2] btrfs-progs: completion: Use _filedir() to replace _btrfs_devs()

2018-07-17 Thread Qu Wenruo
For developers it's pretty common to call "btrfs check" on a raw image dump other than real block device. So current _btrfs_devs() is really making things worse. Use _filedir() to replace _btrfs_devs() so it can complete any filenames, no matter if it's just a file or a real block device.

[PATCH 0/2] btrfs-progs: completion: Small fixes to make debug simpler

2018-07-17 Thread Qu Wenruo
For developer, it's pretty common to use "btrfs check" or "btrfs ins dump-tree" on raw dumps. However "btrfs check" can only complete real block devices, and "btrfs inspect dump-tree" can only complete dir. Make them to use _filedir() so any filename can be completed and save us developer a