Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-12 Thread Qu Wenruo
On 2019/9/12 下午2:28, Nikolay Borisov wrote: > > > On 12.09.19 г. 9:11 ч., Nikolay Borisov wrote: >> >> >> On 12.09.19 г. 7:38 ч., David Newall wrote: >>> On 11/9/19 8:22 pm, Nikolay Borisov wrote: > I saved it tohttps://davidnewall.com/kern.1 Nothing useful in that log, everything seems

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Sasha Levin
Hi, [This is an automated email] This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: 4.4+ The bot has tested the following trees: v5.2.14, v4.19.72, v4.14.143, v4.9.192, v4.4.192. v5.2.14: Build OK! v4.19.72: Fa

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Anand Jain
Thanks for the comments. More below. On 12/9/19 3:16 AM, Josef Bacik wrote: On Wed, Sep 11, 2019 at 03:13:21PM -0400, Eli V wrote: On Wed, Sep 11, 2019 at 2:46 PM Josef Bacik wrote: On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: Function call chain __btrfs_map_block()->find

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Anand Jain
On 12/9/19 12:37 AM, David Sterba wrote: On Mon, Aug 26, 2019 at 05:04:36PM +0800, Anand Jain wrote: Function call chain __btrfs_map_block()->find_live_mirror() uses thread pid to determine the %mirror_num when the mirror_num=0. This patch introduces a framework so that we can add policies to

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Filipe Manana
On Sat, Aug 24, 2019 at 6:53 PM Christoph Anton Mitterer wrote: > > Hey. > > Anything new about the issue described here: > https://www.spinics.net/lists/linux-btrfs/msg91046.html > > It was said that it might be a regression in 5.2 actually and not a > hardware thing... so I just wonder whether I

Re: [PATCH] btrfs: qgroup: Fix the wrong io_tree when freeing reserved data space

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 2:31 AM Qu Wenruo wrote: > > Commit bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by > only freeing reserved ranges") is freeing wrong range in > BTRFS_I()->io_failure_tree, which should be BTRFS_I()->io_tree. I think you meant wrong tree and not wrong

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 8:32 AM Sasha Levin wrote: > > Hi, > > [This is an automated email] > > This commit has been processed because it contains a -stable tag. > The stable tag indicates that it's relevant for the following trees: 4.4+ > > The bot has tested the following trees: v5.2.14, v4.19.7

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread James Harvey
On Thu, Sep 12, 2019 at 3:51 AM Filipe Manana wrote: > ... > > Until the fix gets merged to 5.2 kernels (and 5.3), I don't really > recommend running 5.2 or 5.3. What is your recommendation for distributions that have been shipping 5.2.x for quite some time, where a distro-wide downgrade to 5.1.x

Re: [PATCH] btrfs: qgroup: Fix the wrong io_tree when freeing reserved data space

2019-09-12 Thread Qu Wenruo
On 2019/9/12 下午3:53, Filipe Manana wrote: > On Thu, Sep 12, 2019 at 2:31 AM Qu Wenruo wrote: >> >> Commit bc42bda22345 ("btrfs: qgroup: Fix qgroup reserved space underflow by >> only freeing reserved ranges") is freeing wrong range in >> BTRFS_I()->io_failure_tree, which should be BTRFS_I()->io_

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Swâmi Petaramesh
Hi Filipe, On 9/12/19 9:50 AM, Filipe Manana wrote: > So we definitely have a serious regression introduced on 5.2. > I sent out a fix for it yesterday: > https://patchwork.kernel.org/patch/11141559/ Many thanks for having found and patched it. Kind regards. ॐ -- Swâmi Petaramesh PGP 9076E

Re: [PATCH 2/3] ext4: fix inode rwsem regression

2019-09-12 Thread Ritesh Harjani
cc'd Matthew as well. On 9/11/19 10:15 PM, Goldwyn Rodrigues wrote: From: Goldwyn Rodrigues This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") Apparently our current rwsem code doesn't like doing the trylock, then lock for real scheme. So change our read/write methods to just do the

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 9:24 AM James Harvey wrote: > > On Thu, Sep 12, 2019 at 3:51 AM Filipe Manana wrote: > > ... > > > > Until the fix gets merged to 5.2 kernels (and 5.3), I don't really > > recommend running 5.2 or 5.3. > > What is your recommendation for distributions that have been shippi

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Holger Hoffstätte
On 9/12/19 10:24 AM, James Harvey wrote: On Thu, Sep 12, 2019 at 3:51 AM Filipe Manana wrote: ... Until the fix gets merged to 5.2 kernels (and 5.3), I don't really recommend running 5.2 or 5.3. What is your recommendation for distributions that have been shipping 5.2.x for quite some time,

Re: [PATCH 2/3] ext4: fix inode rwsem regression

2019-09-12 Thread Matthew Bobrowski
On Thu, Sep 12, 2019 at 02:22:35PM +0530, Ritesh Harjani wrote: > cc'd Matthew as well. > > > This is similar to 942491c9e6d6 ("xfs: fix AIM7 regression") > > Apparently our current rwsem code doesn't like doing the trylock, then > > lock for real scheme. So change our read/write methods to just

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Josef Bacik
On Thu, Sep 12, 2019 at 03:41:42PM +0800, Anand Jain wrote: > > > Thanks for the comments. More below. > > On 12/9/19 3:16 AM, Josef Bacik wrote: > > On Wed, Sep 11, 2019 at 03:13:21PM -0400, Eli V wrote: > > > On Wed, Sep 11, 2019 at 2:46 PM Josef Bacik wrote: > > > > > > > > On Mon, Aug 26,

Re: [PATCH RFC] btrfs: Introduce btrfs child tree block verification system

2019-09-12 Thread Josef Bacik
On Thu, Sep 12, 2019 at 07:38:14AM +0800, Qu Wenruo wrote: > > > On 2019/9/12 上午12:02, Josef Bacik wrote: > > On Wed, Sep 11, 2019 at 03:46:24PM +0800, Qu Wenruo wrote: > >> Although we have btrfs_verify_level_key() function to check the first > >> key and level at tree block read time, it has it

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Anand Jain
> On 12 Sep 2019, at 5:50 PM, Josef Bacik wrote: > > On Thu, Sep 12, 2019 at 03:41:42PM +0800, Anand Jain wrote: >> >> >> Thanks for the comments. More below. >> >> On 12/9/19 3:16 AM, Josef Bacik wrote: >>> On Wed, Sep 11, 2019 at 03:13:21PM -0400, Eli V wrote: On Wed, Sep 11, 2019 at

[PATCH for-4.19.x ] btrfs: correctly validate compression type

2019-09-12 Thread Johannes Thumshirn
[ Upstream commit aa53e3bfac7205fb3a8815ac1c937fd6ed01b41e ] Nikolay reported the following KASAN splat when running btrfs/048: [ 1843.470920] == [ 1843.471971] BUG: KASAN: slab-out-of-bounds in strncmp+0x66/0xb0 [ 1843.472775] Read

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Josef Bacik
On Thu, Sep 12, 2019 at 06:00:21PM +0800, Anand Jain wrote: > > > > On 12 Sep 2019, at 5:50 PM, Josef Bacik wrote: > > > > On Thu, Sep 12, 2019 at 03:41:42PM +0800, Anand Jain wrote: > >> > >> > >> Thanks for the comments. More below. > >> > >> On 12/9/19 3:16 AM, Josef Bacik wrote: > >>> On

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Anand Jain
> On 12 Sep 2019, at 6:03 PM, Josef Bacik wrote: > > On Thu, Sep 12, 2019 at 06:00:21PM +0800, Anand Jain wrote: >> >> >>> On 12 Sep 2019, at 5:50 PM, Josef Bacik wrote: >>> >>> On Thu, Sep 12, 2019 at 03:41:42PM +0800, Anand Jain wrote: Thanks for the comments. More below

Re: [PATCH RFC v2 0/2] readmirror feature

2019-09-12 Thread Josef Bacik
On Thu, Sep 12, 2019 at 06:10:08PM +0800, Anand Jain wrote: > > > > On 12 Sep 2019, at 6:03 PM, Josef Bacik wrote: > > > > On Thu, Sep 12, 2019 at 06:00:21PM +0800, Anand Jain wrote: > >> > >> > >>> On 12 Sep 2019, at 5:50 PM, Josef Bacik wrote: > >>> > >>> On Thu, Sep 12, 2019 at 03:41:42P

Re: [PATCH] btrfs: volumes: Allow missing devices to be writeable

2019-09-12 Thread Anand Jain
There is previous work [1] [1] https://lore.kernel.org/linux-btrfs/1461812780-538-1-git-send-email-anand.j...@oracle.com/ I guess it was on purpose that missing device is not part of alloc chunk, so to have lesser impact due to writehole bug. My target is to fix the writehole first, and then

Re: [PATCH RFC] btrfs: Introduce btrfs child tree block verification system

2019-09-12 Thread WenRuo Qu
On 2019/9/12 下午5:59, Josef Bacik wrote: > On Thu, Sep 12, 2019 at 07:38:14AM +0800, Qu Wenruo wrote: >> >> >> On 2019/9/12 上午12:02, Josef Bacik wrote: >>> On Wed, Sep 11, 2019 at 03:46:24PM +0800, Qu Wenruo wrote: Although we have btrfs_verify_level_key() function to check the first key

Re: [PATCH] btrfs: volumes: Allow missing devices to be writeable

2019-09-12 Thread WenRuo Qu
On 2019/9/12 下午6:27, Anand Jain wrote: > > > There is previous work [1] > > [1] > https://lore.kernel.org/linux-btrfs/1461812780-538-1-git-send-email-anand.j...@oracle.com/ > > > > I guess it was on purpose that missing device is not part of > alloc chunk, so to have lesser impact due to wr

Re: [PATCH for-4.19.x ] btrfs: correctly validate compression type

2019-09-12 Thread Sasha Levin
On Thu, Sep 12, 2019 at 12:02:59PM +0200, Johannes Thumshirn wrote: [ Upstream commit aa53e3bfac7205fb3a8815ac1c937fd6ed01b41e ] Nikolay reported the following KASAN splat when running btrfs/048: [ 1843.470920] == [ 1843.471971] B

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Swâmi Petaramesh
Le 12/09/2019 à 10:24, James Harvey a écrit : and should a flashing neon sign be given to users to just run 5.1.x even though the distribution repos have 5.2.x? Yep, I assume that a big flashing red neon sign should be raised for a confirmed bug that can trash your filesystem into ashes, and act

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Nikolay Borisov
On 10.09.19 г. 17:26 ч., fdman...@kernel.org wrote: > From: Filipe Manana > > Sometimes when fsync'ing a file we need to log that other inodes exist and > when we need to do that we acquire a reference on the inodes and then drop > that reference using iput() after logging them. > > That gene

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 12:10 PM Nikolay Borisov wrote: > > > > On 10.09.19 г. 17:26 ч., fdman...@kernel.org wrote: > > From: Filipe Manana > > > > Sometimes when fsync'ing a file we need to log that other inodes exist and > > when we need to do that we acquire a reference on the inodes and then

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Austin S. Hemmelgarn
On 2019-09-11 17:37, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : Given this, defrag isn't willfully unshari

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Nikolay Borisov
On 12.09.19 г. 14:24 ч., Filipe Manana wrote: > On Thu, Sep 12, 2019 at 12:10 PM Nikolay Borisov wrote: >> >> >> >> On 10.09.19 г. 17:26 ч., fdman...@kernel.org wrote: >>> From: Filipe Manana >>> >>> Sometimes when fsync'ing a file we need to log that other inodes exist and >>> when we need to

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Josef Bacik
On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Sometimes when fsync'ing a file we need to log that other inodes exist and > when we need to do that we acquire a reference on the inodes and then drop > that reference using iput() after logging them.

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Josef Bacik
On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Sometimes when fsync'ing a file we need to log that other inodes exist and > when we need to do that we acquire a reference on the inodes and then drop > that reference using iput() after logging them.

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Christoph Anton Mitterer
On Thu, 2019-09-12 at 12:53 +0200, Swâmi Petaramesh wrote: > Yep, I assume that a big flashing red neon sign should be raised for > a > confirmed bug that can trash your filesystem into ashes, and > actually > did so for two of mine... I doubt this will happen... I've asked for something like th

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Christoph Anton Mitterer
Hi. First, thanks for finding&fixing this :-) On Thu, 2019-09-12 at 08:50 +0100, Filipe Manana wrote: > 1) either a hang when committing a transaction, reported by several > users recently and hit it myself too twice when running fstests (test > case generic/475 and generic/561) after I upgradad

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 12:59 PM Josef Bacik wrote: > > On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > Sometimes when fsync'ing a file we need to log that other inodes exist and > > when we need to do that we acquire a reference on the inodes

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 1:18 PM Josef Bacik wrote: > > On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > Sometimes when fsync'ing a file we need to log that other inodes exist and > > when we need to do that we acquire a reference on the inodes a

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-12 Thread David Newall
Hello Qu, Thank you very much for helping me with this. On 12/9/19 4:35 pm, Qu Wenruo wrote: Would you please check how fast (or how slow in this particular case) the related disks are? To me, it really looks like just too slow devices. I discover that you are correct about the underlying sto

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-12 Thread Nikolay Borisov
On 12.09.19 г. 17:03 ч., David Newall wrote: > Hello Qu, > > Thank you very much for helping me with this. > > On 12/9/19 4:35 pm, Qu Wenruo wrote: >> Would you please check how fast (or how slow in this particular case) >> the related disks are? >> To me, it really looks like just too slow de

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-12 Thread Qu Wenruo
On 2019/9/12 下午10:03, David Newall wrote: > Hello Qu, > > Thank you very much for helping me with this. > > On 12/9/19 4:35 pm, Qu Wenruo wrote: >> Would you please check how fast (or how slow in this particular case) >> the related disks are? >> To me, it really looks like just too slow devices

Re: Mount/df/PAM login hangs during rsync to btrfs subvolume, or maybe doing btrfs subvolume snapshot

2019-09-12 Thread Qu Wenruo
On 2019/9/12 下午10:12, Nikolay Borisov wrote: > > > On 12.09.19 г. 17:03 ч., David Newall wrote: >> Hello Qu, >> >> Thank you very much for helping me with this. >> >> On 12/9/19 4:35 pm, Qu Wenruo wrote: >>> Would you please check how fast (or how slow in this particular case) >>> the related di

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Filipe Manana
On Thu, Sep 12, 2019 at 2:09 PM Christoph Anton Mitterer wrote: > > Hi. > > First, thanks for finding&fixing this :-) > > > On Thu, 2019-09-12 at 08:50 +0100, Filipe Manana wrote: > > 1) either a hang when committing a transaction, reported by several > > users recently and hit it myself too twice

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Christoph Anton Mitterer
On Thu, 2019-09-12 at 15:28 +0100, Filipe Manana wrote: > This is case 2), the corruption with the error messages > "parent transid verify failed ..." in dmesg/syslog after mounting the > filesystem again. Hmm so "at least" it will never go unnoticed, right? This is IMO a pretty important advise,

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Swâmi Petaramesh
On 9/12/19 4:39 PM, Christoph Anton Mitterer wrote: > On Thu, 2019-09-12 at 15:28 +0100, Filipe Manana wrote: >> This is case 2), the corruption with the error messages >> "parent transid verify failed ..." in dmesg/syslog after mounting the >> filesystem again. > Hmm so "at least" it will never go

Re: [PATCH] Btrfs: fix assertion failure during fsync and use of stale transaction

2019-09-12 Thread Josef Bacik
On Thu, Sep 12, 2019 at 02:19:55PM +0100, Filipe Manana wrote: > On Thu, Sep 12, 2019 at 1:18 PM Josef Bacik wrote: > > > > On Tue, Sep 10, 2019 at 03:26:49PM +0100, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > Sometimes when fsync'ing a file we need to log that other inodes e

[PATCH] btrfs: Add assert to catch nested transaction commit

2019-09-12 Thread Nikolay Borisov
A recent patch to btrfs showed that there was at least 1 case where a nesed transaction was committed. Nested transaction in this case means a code which has a transaction handle calls some function which in turn obtains a copy of the same transaction handle. In such cases the correct thing to do

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Zdenek Kaspar
On 9/12/19 4:57 PM, Swâmi Petaramesh wrote: However having read that the bug is diagnosed, confirmed and fixed by Filipe, I seriously consider downgrading my kernel back to 5.1 on the 2 Manjaro machines as it is rather straightforward, and maybe my Arch as well... Until I'm sure that the fix mad

[GIT PULL] Last btrfs fixes for 5.3

2019-09-12 Thread David Sterba
Hi, there are two fixes, one of them urgent fixing a bug introduced in 5.2 and reported by many users. It took time to identify the root cause, catching the 5.3 release is higly desired also to push the fix to 5.2 stable tree. The bug is a mess up of return values after adding proper error handli

Re: [PATCH] btrfs-progs: corrupt-block: Fix description of 'r' option

2019-09-12 Thread David Sterba
On Wed, Sep 11, 2019 at 04:55:52PM +0300, Nikolay Borisov wrote: > Since commit 04be0e4b1962 ("btrfs-progs: corrupt-block: Correctlyi > handle -r when passing -I") the 'r' switch is used with both -I and -d > options. So remove the wrong clarificatoin that -r is used only with -d > option. > > Sig

Re: [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data

2019-09-12 Thread David Sterba
On Wed, Sep 04, 2019 at 09:29:47PM +0800, Anand Jain wrote: > As misc-tests/021 image dump is restored on the same original loop > device, this overlaps with the stale data and makes the test pass > falsely. > > Fix this by using a new device for restore. > > And also, the btrfs-image dump and re

Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index

2019-09-12 Thread David Sterba
On Tue, Sep 03, 2019 at 02:06:03PM +0200, David Sterba wrote: > On Mon, Sep 02, 2019 at 06:22:30PM +0200, David Sterba wrote: > > On Mon, Sep 02, 2019 at 04:01:56PM +0800, Anand Jain wrote: > > > > > > David, > > > > > > I don't see this patch is integrated. Can you please integrated this > >

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Swâmi Petaramesh
Le 12/09/2019 à 18:21, Zdenek Kaspar a écrit : On 9/12/19 4:57 PM, Swâmi Petaramesh wrote: However having read that the bug is diagnosed, confirmed and fixed by Filipe, I seriously consider downgrading my kernel back to 5.1 on the 2 Manjaro machines as it is rather straightforward, and maybe my

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread webmaster
Quoting "Austin S. Hemmelgarn" : On 2019-09-11 17:37, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : Given

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Chris Murphy
On Thu, Sep 12, 2019 at 1:18 PM wrote: > > It is normal and common for defrag operation to use some disk space > while it is running. I estimate that a reasonable limit would be to > use up to 1% of total partition size. So, if a partition size is 100 > GB, the defrag can use 1 GB. Lets call this

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Austin S. Hemmelgarn
On 2019-09-12 15:18, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 17:37, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-10 19:32, webmas...@zedlx.com wr

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Chris Murphy : On Thu, Sep 12, 2019 at 1:18 PM wrote: It is normal and common for defrag operation to use some disk space while it is running. I estimate that a reasonable limit would be to use up to 1% of total partition size. So, if a partition size is 100 GB, the defrag can use 1

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting "Austin S. Hemmelgarn" : On 2019-09-12 15:18, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 17:37, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 201

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Chris Murphy
On Thu, Sep 12, 2019 at 3:34 PM General Zed wrote: > > > Quoting Chris Murphy : > > > On Thu, Sep 12, 2019 at 1:18 PM wrote: > >> > >> It is normal and common for defrag operation to use some disk space > >> while it is running. I estimate that a reasonable limit would be to > >> use up to 1% of

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Wed, Sep 11, 2019 at 07:21:31PM -0400, webmas...@zedlx.com wrote: Quoting Zygo Blaxell : > On Wed, Sep 11, 2019 at 01:20:53PM -0400, webmas...@zedlx.com wrote: > > > > Quoting "Austin S. Hemmelgarn" : > > > > > On 2019-09-10 19:32, webmas...@zedlx.com wrote: > > >

btrfs forced readonly | error in btrfs_run_delayed_refs:2907: errno=-17 Object already exists

2019-09-12 Thread Rachel Powers
I've tried using -o usebackuproot but the error persists as soon as I try to modify the filesystem a btrfs scrub fails almost immediately, errors are in the dmesg log a btrfs-check (no --repair) also fails almost imediatly Opening filesystem to check... Checking filesystem on /dev/sda4 UUID: 31b

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting "Austin S. Hemmelgarn" : On 2019-09-12 15:18, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 17:37, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 2019-09-11 13:20, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : On 201

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Chris Murphy : On Thu, Sep 12, 2019 at 3:34 PM General Zed wrote: Quoting Chris Murphy : > On Thu, Sep 12, 2019 at 1:18 PM wrote: >> >> It is normal and common for defrag operation to use some disk space >> while it is running. I estimate that a reasonable limit would be to >> us

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Zygo Blaxell
On Thu, Sep 12, 2019 at 06:57:26PM -0400, General Zed wrote: > > Quoting Chris Murphy : > > > On Thu, Sep 12, 2019 at 3:34 PM General Zed wrote: > > > > > > > > > Quoting Chris Murphy : > > > > > > > On Thu, Sep 12, 2019 at 1:18 PM wrote: > > > >> > > > >> It is normal and common for defrag

[PATCH] btrfs: fix balance convert to single on 32-bit host CPUs

2019-09-12 Thread Zygo Blaxell
Currently, the command: btrfs balance start -dconvert=single,soft . on a Raspberry Pi produces the following kernel message: BTRFS error (device mmcblk0p2): balance: invalid convert data profile single This fails because we use is_power_of_2(unsigned long) to validate the new d

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Thu, Sep 12, 2019 at 06:57:26PM -0400, General Zed wrote: Quoting Chris Murphy : > On Thu, Sep 12, 2019 at 3:34 PM General Zed wrote: > > > > > > Quoting Chris Murphy : > > > > > On Thu, Sep 12, 2019 at 1:18 PM wrote: > > >> > > >> It is normal and common for def

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Wed, Sep 11, 2019 at 04:01:01PM -0400, webmas...@zedlx.com wrote: Quoting "Austin S. Hemmelgarn" : > Not necessarily. Even ignoring the case of data deduplication (which > needs to be considered if you care at all about enterprise usage, and is > part of the who

[PATCH 1/2] btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space

2019-09-12 Thread Qu Wenruo
[BUG] Under the follow case with qgroup enabled, if some error happened after we have reserved delalloc space, then in error handling path, we could cause qgroup data space leakage: >From btrfs_truncate_block() in inode.c: ret = btrfs_delalloc_reserve_space(inode, &data_reserved,

[PATCH 2/2] btrfs: qgroup: Fix reserved data space leak if we have multiple reserve calls

2019-09-12 Thread Qu Wenruo
[BUG] The following script can cause btrfs qgroup data space leak: mkfs.btrfs -f $dev mount $dev -o nospace_cache $mnt btrfs subv create $mnt/subv btrfs quota en $mnt btrfs quota rescan -w $mnt btrfs qgroup limit 128m $mnt/subv for (( i = 0; i < 3; i++)); do # Create 3 64

[PATCH] fstests: btrfs: Verify falloc on multiple holes won't cause qgroup reserved data space leak

2019-09-12 Thread Qu Wenruo
Add a test case where falloc is called on multiple holes with qgroup enabled. This can cause qgroup reserved data space leak and false EDQUOT error even we're not reaching the limit. The fix is titled: "btrfs: qgroup: Fix the wrong target io_tree when freeing reserved data space" Signed-off-by:

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread Zygo Blaxell
On Thu, Sep 12, 2019 at 08:26:04PM -0400, General Zed wrote: > > Quoting Zygo Blaxell : > > > On Thu, Sep 12, 2019 at 06:57:26PM -0400, General Zed wrote: > > > > > > Quoting Chris Murphy : > > > > > > > On Thu, Sep 12, 2019 at 3:34 PM General Zed > > > > wrote: > > > > > > > > > > > > > > >

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Thu, Sep 12, 2019 at 08:26:04PM -0400, General Zed wrote: Quoting Zygo Blaxell : > Don't forget you have to write new checksum and free space tree pages. > In the worst case, you'll need about 1GB of new metadata pages for each > 128MB you defrag (though you get to

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Thu, Sep 12, 2019 at 08:26:04PM -0400, General Zed wrote: Quoting Zygo Blaxell : You mean: all metadata size is 156 GB on one of your systems. However, you don't typically have to put ALL metadata in RAM. You need just some parts needed for defrag operation. So, fo

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Thu, Sep 12, 2019 at 08:26:04PM -0400, General Zed wrote: Quoting Zygo Blaxell : You mean: all metadata size is 156 GB on one of your systems. However, you don't typically have to put ALL metadata in RAM. You need just some parts needed for defrag operation. So, fo

Re: Feature requests: online backup - defrag - change RAID level

2019-09-12 Thread General Zed
Quoting Zygo Blaxell : On Thu, Sep 12, 2019 at 08:26:04PM -0400, General Zed wrote: Quoting Zygo Blaxell : > On Thu, Sep 12, 2019 at 06:57:26PM -0400, General Zed wrote: > > > > At worst, it just has to completely write-out "all metadata", all the way up > > to the super. It needs to be