[PATCH v2 1/1] btrfs: Add mechanism to configure automatic level-0 qgroup removal

to avoid breaking userspace. Once the knob is enabled, it is persisted across remounts in qgroup_flags. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h| 2 +- fs/btrfs/ioctl.c| 14 +++ fs/btrfs/sysfs.c

[PATCH v2 0/1] Add qgroup_autoremove flag

: * Move from mount option to sysfs knob * Persist the knob across remounts Sargun Dhillon (1): btrfs: Add mechanism to configure automatic level-0 qgroup removal fs/btrfs/ctree.h| 2 +- fs/btrfs/ioctl.c| 14 +++ fs/btrfs/sysfs.c| 91

[PATCH] btrfs: Report error on removing qgroup if del_qgroup_item fails

of the error if the error is fatal, (any error other than qgroup not found). Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 770f667269f5..e172d4843eae 100644 --- a/fs/btrfs/qg

[PATCH v3 2/2] btrfs: Add new ioctl uapis for qgroup creation / removal

unless a matching subvolume ID is created. The delete qgroup ioctl has the new semantic that it will not let you delete a currently utilized (referenced by a subvolume) level-0 qgroup unless you pass the BTRFS_QGROUP_NO_SUBVOL_CHECK flag. Signed-off-by: Sargun Dhillon <sar...@sargun.me> -

[PATCH v3 1/2] btrfs: Fail on removing qgroup if del_qgroup_item fails

of the error if the error is fatal, (any error other than qgroup not found). Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 4ce351e..3abc517 100644 --- a/fs/btrfs/qgroup.c ++

[PATCH v3 0/2] New qgroup creation / removal ioctls

d creation * Remove deprecation message for old API Changes since v1: * Remove creation of level-0 qgroups without subvol Sargun Dhillon (2): btrfs: Fail on removing qgroup if del_qgroup_item fails btrfs: Add new ioctl uapis for qgroup creation / removal fs/btrfs/ioctl.c

[PATCH v3 0/2] New qgroup creation / removal ioctl

d creation * Remove deprecation message for old API Changes since v1: * Remove creation of level-0 qgroups without subvol Sargun Dhillon (2): btrfs: Fail on removing qgroup if del_qgroup_item fails btrfs: Add new ioctl uapis for qgroup creation / removal fs/btrfs/ioctl.c

Re: Containers, Btrfs vs Btrfs + overlayfs

On Thu, Jul 13, 2017 at 7:01 PM, Qu Wenruo wrote: > > > On 2017年07月14日 07:26, Chris Murphy wrote: >> >> On Thu, Jul 13, 2017 at 4:32 PM, Liu Bo wrote: >>> >>> On Thu, Jul 13, 2017 at 02:49:27PM -0600, Chris Murphy wrote: Has anyone been

Re: Lock between userspace and btrfs-cleaner on extent_buffer

On Thu, Jun 29, 2017 at 11:49 AM, Jeff Mahoney <je...@suse.com> wrote: > On 6/29/17 2:46 PM, Sargun Dhillon wrote: >> On Thu, Jun 29, 2017 at 11:42 AM, Jeff Mahoney <je...@suse.com> wrote: >>> On 6/28/17 6:02 PM, Sargun Dhillon wrote: >>>> On Wed,

Re: Lock between userspace and btrfs-cleaner on extent_buffer

On Thu, Jun 29, 2017 at 11:42 AM, Jeff Mahoney <je...@suse.com> wrote: > On 6/28/17 6:02 PM, Sargun Dhillon wrote: >> On Wed, Jun 28, 2017 at 2:55 PM, Jeff Mahoney <je...@suse.com> wrote: >>> On 6/27/17 5:12 PM, Jeff Mahoney wrote: >>>> On 6/13/17 9:05 PM,

Re: Lock between userspace and btrfs-cleaner on extent_buffer

On Wed, Jun 28, 2017 at 2:55 PM, Jeff Mahoney <je...@suse.com> wrote: > On 6/27/17 5:12 PM, Jeff Mahoney wrote: >> On 6/13/17 9:05 PM, Sargun Dhillon wrote: >>> On Thu, Jun 8, 2017 at 11:34 AM, Sargun Dhillon <sar...@sargun.me> wrote: >>>> I have a deadloc

[PATCH] btrfs: Add qgroup_auto_cleanup mount flag to automatically cleanup qgroups

it is opt-in. Existing software, and scripts may be doing qgroup cleanup on subvolume deletion explicitly, and the absence of a qgroup may cause failure. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h | 1 + fs/btrfs/ioctl.c | 15 +++ fs/btrfs/super.

Re: [PATCH v2 0/4] Qgroup uapi improvements

On Fri, May 26, 2017 at 1:44 PM, Sargun Dhillon <sar...@sargun.me> wrote: > This patchset has several improvements around the qgroups user-facing > API. It introduces two new ioctls for creating, and removing qgroups. > These ioctls have a new args structure that allows passing fl

Re: Lock between userspace and btrfs-cleaner on extent_buffer

On Thu, Jun 8, 2017 at 11:34 AM, Sargun Dhillon <sar...@sargun.me> wrote: > I have a deadlock caught in the wild between two processes -- > btrfs-cleaner, and userspace process (Docker). Here, you can see both > of the backtraces. btrfs-cleaner is trying to get a lock on > 9

Lock between userspace and btrfs-cleaner on extent_buffer

I have a deadlock caught in the wild between two processes -- btrfs-cleaner, and userspace process (Docker). Here, you can see both of the backtraces. btrfs-cleaner is trying to get a lock on 9859d360caf0, which is owned by Docker's pid. Docker on the other hand is trying to get a lock on

[PATCH v2] btrfs: Convert btrfs_fs_info's free_chunk_space to an atomic64_t

This patch is a small performance optimization to get rid of a spin lock, where instead an atomic64_t can be used. Signed-off-by: Sargun Dhillon <sar...@sargun.me> Reviewed-by: Omar Sandoval <osan...@fb.com> --- fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 3 +-- fs/

Re: [PATCH] btrfs: Convert btrfs_fs_info's free_chunk_space to an atomic64_t

On Wed, Jun 7, 2017 at 11:23 AM, Omar Sandoval <osan...@osandov.com> wrote: > On Wed, Jun 07, 2017 at 06:09:33PM +0000, Sargun Dhillon wrote: >> This patch is a small performance optimization to get rid of a spin >> lock, where instead an atomic64_t can be used. >> >&

[PATCH] btrfs: Convert btrfs_fs_info's free_chunk_space to an atomic64_t

This patch is a small performance optimization to get rid of a spin lock, where instead an atomic64_t can be used. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h | 4 ++-- fs/btrfs/disk-io.c | 3 +-- fs/btrfs/extent-tree.c | 4 +--- fs/btrfs/volumes.c

Re: Debugging Deadlocks?

On Wed, May 31, 2017 at 5:54 AM, David Sterba <dste...@suse.cz> wrote: > On Tue, May 30, 2017 at 09:12:39AM -0700, Sargun Dhillon wrote: >> We've been running BtrFS for a couple months now in production on >> several clusters. We're running on Canonical's 4.8 ker

Debugging Deadlocks?

We've been running BtrFS for a couple months now in production on several clusters. We're running on Canonical's 4.8 kernel, and currently, in the process of moving to our own patchset atop vanilla 4.10+. I'm glad to say it's been a fairly good experience for us. Bar some performance issues, it's

[PATCH v2 3/4] btrfs: Warn the user when the legacy btrfs_qgroup_create API is used

This patch adds a warning to let the user know when the legacy qgroup creation / removal API is in use. Eventually, we can deprecate this API. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ioctl.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs

[PATCH v2 0/4] Qgroup uapi improvements

ed trigger on newly created qgroups. Later on, we can introduce a flag to subvol / qgroup create that marks the qgroup as automatically created, and delete the qgroup as automatically as well. Changes since v1: * Remove creation of level-0 qgroups without subvol * Add deprecation message for old A

[PATCH v2 2/4] btrfs: Add new ioctl uapis for qgroup creation / removal

unless a matching subvolume ID is created. The delete qgroup ioctl has the new semantic that it will not let you delete a currently utilized (referenced by a subvolume) level-0 qgroup unless you pass the BTRFS_QGROUP_NO_SUBVOL_CHECK flag. Signed-off-by: Sargun Dhillon <sar...@sargun.me> -

[PATCH v2 4/4] btrfs: Add qgroup_auto_cleanup mount flag to automatically cleanup qgroups

, as people may already have scripts that do this cleanup on subvolume destruction, this feature, at least for now, must be opt-in. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h | 1 + fs/btrfs/ioctl.c | 15 +++ fs/btrfs/super.c | 15 ++- 3 files c

[PATCH v2 1/4] btrfs: Fail on removing qgroup if del_qgroup_item fails

of the error if the error is fatal, (any error other than qgroup not found). Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index deffbeb..d39ffcc 100644 --- a/fs/btrfs/qgroup.c ++

Re: QGroups Semantics

On Sun, May 21, 2017 at 5:59 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 05/19/2017 05:07 PM, Sargun Dhillon wrote: >> >> I have some questions about the *intended* qgroups semantics, and why >> we allow certain operations: >> >> 1)

Re: [PATCH 4/8] btrfs: autoremove qgroup by default, and add a mount flag to override

On Sun, May 21, 2017 at 7:03 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 05/22/2017 09:58 AM, Sargun Dhillon wrote: >> >> On Sun, May 21, 2017 at 6:20 PM, Qu Wenruo <quwen...@cn.fujitsu.com> >> wrote: >>> >>

Re: [PATCH 6/8] btrfs: Add code to check if a qgroup's subvol exists

On Sun, May 21, 2017 at 6:39 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 05/20/2017 04:39 PM, Sargun Dhillon wrote: >> >> This patch is to prepare for following patches in this patchset. The >> purpose is to make it so that we can prevent

Re: [PATCH 4/8] btrfs: autoremove qgroup by default, and add a mount flag to override

On Sun, May 21, 2017 at 6:20 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 05/20/2017 04:39 PM, Sargun Dhillon wrote: >> >> This change follows the change to automatically remove qgroups >> if the associated subvolume has also been removed. It

Re: [PATCH 2/8] btrfs: Fail on removing qgroup if del_qgroup_item fails

On Sun, May 21, 2017 at 6:14 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 05/20/2017 04:39 PM, Sargun Dhillon wrote: >> >> Previously, we were calling del_qgroup_item, and ignoring the return code >> resulting in a potential to have divergent in-memory s

Re: [PATCH 4/8] btrfs: autoremove qgroup by default, and add a mount flag to override

Just as a follow-up, folks are working around this in userspace: https://github.com/moby/moby/pull/29427 On Sat, May 20, 2017 at 1:39 AM, Sargun Dhillon <sar...@sargun.me> wrote: > This change follows the change to automatically remove qgroups > if the associated subvolume has also

[PATCH 8/8] btrfs: Add new ioctl uapis for qgroup creation / removal

in an override flag. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ioctl.c | 98 ++ include/uapi/linux/btrfs.h | 23 +++ 2 files changed, 121 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 5

[PATCH 6/8] btrfs: Add code to check if a qgroup's subvol exists

This patch is to prepare for following patches in this patchset. The purpose is to make it so that we can prevent accidental removal of qgroups that are actively in use. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ioctl.c | 4 ++-- fs/btrfs/qgroup.

[PATCH 7/8] btrfs: Add code to prevent qgroup creation for a non-existent subvol

) a qgroup when you create a new subvol with quota on. If there is an existing qgroup with this same ID, it will be reset. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ioctl.c | 2 +- fs/btrfs/qgroup.c | 11 ++- fs/btrfs/qgroup.h | 3 ++- 3 files changed, 13 inse

[PATCH 4/8] btrfs: autoremove qgroup by default, and add a mount flag to override

This change follows the change to automatically remove qgroups if the associated subvolume has also been removed. It changes the default behaviour to automatically remove qgroups when a subvolume is deleted, but this can be override with the qgroup_keep mount flag. Signed-off-by: Sargun Dhillon

[PATCH 5/8] btrfs: qgroup.h whitespace change

This patch is just changing whitespace alignment in qgroup.h Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/btrfs/qgroup.h b/fs/btrfs/qgroup.h index fe04d3f..fb6c7da 100644 --- a/fs/btrfs/qg

[PATCH 1/8] btrfs: Split up btrfs_remove_qgroup, no logic changes

This change is purely a style change, so that btrfs_remove_qgroup is split. There is some whitespace changes, but this shouldn't have any effect on the logic of the code. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.

[PATCH 3/8] btrfs: Split up btrfs_create_qgroup, no logic changes

This change is purely a style change, so that btrfs_create_qgroup is split. There is some whitespace changes, but this shouldn't have any effect on the logic of the code. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 32 1 file c

[PATCH 2/8] btrfs: Fail on removing qgroup if del_qgroup_item fails

of the error. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index a2add44..7e772ba 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -1303,6 +1303,8 @@ stat

[PATCH 0/8] BtrFS: QGroups uapi improvements

to prevent the deletion of qgroups that are associated with volumes. Sargun Dhillon (8): btrfs: Split up btrfs_remove_qgroup, no logic changes btrfs: Fail on removing qgroup if del_qgroup_item fails btrfs: Split up btrfs_create_qgroup, no logic changes btrfs: autoremove qgroup by default

QGroups Semantics

I have some questions about the *intended* qgroups semantics, and why we allow certain operations: 1) Why can you create a level 0 qgroup for a non-existent subvolume? It looks like it just gets reset when you create the subvol anyway? Should we prevent this? 2) Why do we allow deleting a level

Re: [PATCH v2 0/2] btrfs: allow mechanism to override quota

On Fri, May 12, 2017 at 7:48 AM, David Sterba <dste...@suse.cz> wrote: > On Thu, May 11, 2017 at 09:17:01PM +0000, Sargun Dhillon wrote: >> This patchset makes it so that on a per-filesystem basis one can disable >> quota enforcement for users with cap_sys_resource. This p

[PATCH v2 2/2] btrfs: Add quota_override knob into sysfs

This patch adds the read-write attribute quota_override into sysfs. Any process which has cap_sys_resource can set this flag to on, and once it is set to true, processes with cap_sys_resource can exceed the quota. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/sysfs.

[PATCH v2 1/2] btrfs: add quota override flag to enable quota override for sys_resource

the simplest solution. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h | 2 ++ fs/btrfs/qgroup.c | 5 + 2 files changed, 7 insertions(+) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index 643c70d..e86cb7c 100644 --- a/fs/btrfs/ctree.h +++ b/fs/btrfs/ctree.h @@

[PATCH v2 0/2] btrfs: allow mechanism to override quota

for the qgroups themselves. Changes since v1: -Rather than a separate member of btrfs_fs_info, use the existing flags field Sargun Dhillon (2): btrfs: add quota override flag to enable quota override for sys_resource btrfs: Add quota_override knob into sysfs fs/btrfs/ctree.h | 2

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

lock down their capabilities, so I agree, making it blindly based on capabilities seems like a poor idea. On Fri, May 5, 2017 at 11:22 AM, David Sterba <dste...@suse.cz> wrote: > On Fri, Apr 21, 2017 at 07:27:23AM -0500, Sargun Dhillon wrote: >> What do you think about putting this b

Re: [PATCH 1/2] btrfs: add quota override attribute

On Sun, Apr 23, 2017 at 8:42 PM, Qu Wenruo <quwen...@cn.fujitsu.com> wrote: > > > At 04/22/2017 07:12 AM, Sargun Dhillon wrote: >> >> This patch introduces the quota override flag to btrfs_fs_info, and >> a change to quota limit checking code to temporarily al

[PATCH 2/2] btrfs: Add quota_override knob into sysfs

This patch adds the read-write attribute quota_override into sysfs. Any process which has cap_sys_resource can set this flag to on, and once it is set to true, processes with cap_sys_resource can exceed the quota. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/sysfs.

[PATCH 1/2] btrfs: add quota override attribute

the simplest solution. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/ctree.h | 3 +++ fs/btrfs/qgroup.c | 9 +++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h index c411590..01a095b 100644 --- a/fs/btrfs/ctree.h ++

[PATCH 0/2] btrfs: allow processes with exceed quota with override

for the qgroups themselves. Sargun Dhillon (2): btrfs: add quota override attribute btrfs: Add quota_override knob into sysfs fs/btrfs/ctree.h | 3 +++ fs/btrfs/qgroup.c | 9 +++-- fs/btrfs/sysfs.c | 36 3 files changed, 46 insertions(+), 2 deletions

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

What do you think about putting this behaviour behind a sysctl? Seems better than to start introducing a new mechanism of marking tasks? On Fri, Apr 21, 2017 at 6:05 AM, Adam Borowski <kilob...@angband.pl> wrote: > On Fri, Apr 21, 2017 at 10:09:46AM +0000, Sargun Dhillon wrote: >

Re: [PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

rator_rfer, and rsv_operator_excl, but that seems to be killing the mosquito with the cannon. On Fri, Apr 21, 2017 at 5:09 AM, Sargun Dhillon <sar...@sargun.me> wrote: > This patch allows processes with CAP_SYS_RESOURCE to exceed the qgroup > limit. It's useful for administrative programs, such as lo

[PATCH] btrfs: allow processes with cap_sys_resource to exceed quota

of userland. Signed-off-by: Sargun Dhillon <sar...@sargun.me> --- fs/btrfs/qgroup.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index a59801d..b0af39c 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -2349,6 +2349,9 @@ int btrfs_qgroup_i

Re: [PATCH ping] btrfs: warn about RAID5/6 being experimental at mount time

On Thu, Apr 20, 2017 at 3:13 PM, Duncan <1i5t5.dun...@cox.net> wrote: > Adam Borowski posted on Wed, 19 Apr 2017 23:07:45 +0200 as excerpted: > >> Too many people come complaining about losing their data -- and indeed, >> there's no warning outside a wiki and the mailing list tribal knowledge. >>

Re: BTRFS as a GlusterFS storage back-end, and what I've learned from using it as such.

Not to change the topic too much but, is there a suite of tracing scripts that one can attach to their BtrFS installation to gather metrics about tree locking performance? We see an awful lot of machines with a task waiting on btrfs_tree_lock, and a bunch of other tasks that are also in disk sleep