Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote: > This patch isn't integrated, any idea why ? Because it does not cover all usecases. I've committed the preparatory bits from branch where I've been working on that feature so we can actually get to the hard parts. -- To unsubscribe

reproducable oops in btrfs/130 with latests mainline

btrfs/130 [ 384.645337] run fstests btrfs/130 at 2016-11-14 12:33:26 [ 384.827333] BTRFS: device fsid bf118b00-e2e0-4a96-a177-765789170093 devid 1 transid 3 /dev/vdc [ 384.851643] BTRFS info (device vdc): disk space caching is enabled [ 384.852113] BTRFS info (device vdc): flagging fs

Re: [PATCH] ioctl.h: add missing kernel compatibility header for BUILD_ASSERT

On Sat, Nov 12, 2016 at 10:14:49PM +, Sergei Trofimovich wrote: > > > Basically gcc tries to say us BUILD_ASSERT is not visible. > > > > > > BUILD_ASSERT lives in kerncompat.h which this change adds. > > > > I think including the kerncompat.h is too intrusive here, I've fixed by > >

Re: [PULL] Btrfs fixes for 4.9-rc5

On Fri, Nov 11, 2016 at 03:45:10PM -0500, Chris Mason wrote: > On 11/10/2016 10:00 AM, David Sterba wrote: > > Hi, > > > > two minor error handling fixes and one fix for balance sttus item that goes > > back to 4.4. > > > > The branch continues from my last pull that went to Linus' tree, so it

Re: [Bug 186671] New: OOM on system with just rsync running 32GB of ram 30GB of pagecache

System is an intel dual socket Xeon E5620, 7500/5520/5500/X58 ICH10 family according to lspci. Anyways 4.8.4 OOM'd while I was gone. I'll download the current 4.9rc and reboot, but in the mean time here's xxd, vmstat & kern.log output: 8532039 9324 0100 2226

Re: Btrfs progs release 4.8.3

On Sun, Nov 13, 2016 at 02:28:02PM +0100, Henk Slager wrote: > On Fri, Nov 11, 2016 at 4:38 PM, David Sterba wrote: > > Hi, > > > > btrfs-progs version 4.8.3 have been released. Handful of fixes and lots of > > cleanups. > > > > Changes: > > * check: > > * support for

Re: [PATCH 1/2] btrfs: remove old tree_root dirent processing in btrfs_real_readdir()

On 11/11/16 11:05 AM, Omar Sandoval wrote: > On Sat, Nov 05, 2016 at 01:26:34PM -0400, je...@suse.com wrote: >> From: Jeff Mahoney >> >> Commit 3de4586c527 (Btrfs: Allow subvolumes and snapshots anywhere >> in the directory tree) introduced the current system of placing >>

[PATCH 0/2] RAID5/6 scrub race fix

Fix the so-called famous RAID5/6 scrub error. Thanks Goffredo Baroncelli for reporting the bug, and make it into our sight. (Yes, without the Phoronix report on this, https://www.phoronix.com/scan.php?page=news_item=Btrfs-RAID-56-Is-Bad, I won't ever be aware of it) Unlike many of us(including

[PATCH 1/2] btrfs: scrub: Introduce full stripe lock for RAID56

Unlike mirror based profiles, RAID5/6 recovery needs to read out the whole full stripe. And if we don't do proper protect, it can easily cause race condition. Introduce 2 new functions: lock_full_stripe() and unlock_full_stripe() for RAID5/6. Which stores a rb_tree of mutex for full stripes, so

[PATCH 2/2] btrfs: scrub: Fix RAID56 recovery race condition

When scrubbing a RAID5 which has recoverable data corruption (only one data stripe is corrupted), sometimes scrub will report more csum errors than expected. Sometimes even unrecoverable error will be reported. The problem can be easily reproduced by the following steps: 1) Create a btrfs with

Re: [PATCH 3/6] btrfs-progs: add OPEN_CTREE_INVALIDATE_FST flag

On Mon, Nov 14, 2016 at 09:22:22AM +0800, Qu Wenruo wrote: > > > At 11/14/2016 03:35 AM, Omar Sandoval wrote: > > From: Omar Sandoval > > > > If this flag is passed to open_ctree(), we'll clear the > > FREE_SPACE_TREE_VALID compat_ro bit. The kernel will then reconstruct > >

Re: [PATCH] ioctl.h: add missing kernel compatibility header for BUILD_ASSERT

On Mon, Nov 14, 2016 at 01:34:49PM +0100, David Sterba wrote: > On Sat, Nov 12, 2016 at 10:14:49PM +, Sergei Trofimovich wrote: > > > > Basically gcc tries to say us BUILD_ASSERT is not visible. > > > > > > > > BUILD_ASSERT lives in kerncompat.h which this change adds. > > > > > > I think

Re: [PATCH v3-onstack] btrfs: make block group flags in balance printks human-readable

On Sat, Nov 12, 2016 at 12:59:59AM +0100, Adam Borowski wrote: > They're not even documented anywhere, letting users with no recourse but > to RTFS. It's no big burden to output the bitfield as words. > > Also, display unknown flags as hex. > > Signed-off-by: Adam Borowski

Re: how to understand "btrfs fi show" output? "No space left" issues

On 2016 Sep 20, Peter Becker wrote: > Data, RAID1: total=417.12GiB, used=131.33GiB > > You have 417(total)-131(used) blocks wo are only partial filled. > You should balance your file-system. > > At first you need some free space. You could remove some files / old > snapshots etc. or you add a

Re: [PATCH 3/6] btrfs-progs: add OPEN_CTREE_INVALIDATE_FST flag

On Mon, Nov 14, 2016 at 08:22:56AM -0800, Omar Sandoval wrote: > > > --- a/disk-io.c > > > +++ b/disk-io.c > > > @@ -904,7 +904,7 @@ free_all: > > > return NULL; > > > } > > > > > > -int btrfs_check_fs_compatibility(struct btrfs_super_block *sb, int > > > writable) > > > +int

Re: [PATCH 4/6] btrfs-progs: add btrfs_clear_free_space_tree() from the kernel

On Mon, Nov 14, 2016 at 09:38:19AM +0800, Qu Wenruo wrote: > > > At 11/14/2016 03:35 AM, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Signed-off-by: Omar Sandoval > > Only small nits about the BUG_ON() and return value. > Despite that, feel free to add

Re: About difference in extent sharing in btrfs and xfs

On Tue, Nov 15, 2016 at 02:33:31PM +0800, Qu Wenruo wrote: > Hi, xfs guys and btrfs guys. > > Although the test case generic/372 exists for some time, I noticed that > btrfs always fails the test case, due to the difference in how btrfs > and xfs handle shared extents. > > The difference is,

Re: About difference in extent sharing in btrfs and xfs

At 11/15/2016 03:47 PM, Darrick J. Wong wrote: On Tue, Nov 15, 2016 at 02:33:31PM +0800, Qu Wenruo wrote: Hi, xfs guys and btrfs guys. Although the test case generic/372 exists for some time, I noticed that btrfs always fails the test case, due to the difference in how btrfs and xfs handle

About difference in extent sharing in btrfs and xfs

Hi, xfs guys and btrfs guys. Although the test case generic/372 exists for some time, I noticed that btrfs always fails the test case, due to the difference in how btrfs and xfs handle shared extents. The difference is, btrfs can handle shared extents which points to a subset of a larger

Re: [PATCH] fstests: generic/098 update test for truncating a file into the middle of a hole

On Fri, Nov 11, 2016 at 02:30:04PM -0800, Liu Bo wrote: > This updates generic/098 by adding a sync option, i.e. 'sync' after the second > write, and with btrfs's NO_HOLES, we could still get wrong isize after > remount. > > This gets fixed by the patch > > 'Btrfs: fix truncate down when

[PATCH v4] btrfs: make block group flags in balance printks human-readable

They're not even documented anywhere, letting users with no recourse but to RTFS. It's no big burden to output the bitfield as words. Also, display unknown flags as hex. Signed-off-by: Adam Borowski --- fs/btrfs/relocation.c | 42 +++---

Re: Announcing btrfs-dedupe

On Tue, Nov 08, 2016 at 12:06:01PM +0100, Niccolò Belli wrote: > Nice, you should probably update the btrfs wiki as well, because there is no > mention of btrfs-dedupe. > > First question, why this name? Don't you plan to support xfs as well? Does XFS plan to support LOGICAL_INO, INO_PATHS, and

[PATCH v14.1 05/16] btrfs: dedupe: Introduce function to initialize dedupe info

From: Wang Xiaoguang Add generic function to initialize dedupe info. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang Reviewed-by: Josef Bacik --- fs/btrfs/Makefile | 2 +-

[PATCH v14.1 15/16] btrfs: relocation: Enhance error handling to avoid BUG_ON

Since the introduce of btrfs dedupe tree, it's possible that balance can race with dedupe disabling. When this happens, dedupe_enabled will make btrfs_get_fs_root() return PTR_ERR(-ENOENT). But due to a bug in error handling branch, when this happens backref_cache->nr_nodes is increased but the

[PATCH v14.1 16/16] btrfs: dedupe: Introduce new reconfigure ioctl

Introduce new reconfigure ioctl, and new FORCE flag for in-band dedupe ioctls. Now dedupe enable and reconfigure ioctl are stateful. | Current state | Ioctl| Next state | | Disabled | enable

[PATCH v14.1 14/16] btrfs: dedupe: Add ioctl for inband dedupelication

From: Wang Xiaoguang Add ioctl interface for inband dedupelication, which includes: 1) enable 2) disable 3) status And a pseudo RO compat flag, to imply that btrfs now supports inband dedup. However we don't add any ondisk format change, it's just a pseudo RO compat

[PATCH v14.1 11/16] btrfs: ordered-extent: Add support for dedupe

From: Wang Xiaoguang Add ordered-extent support for dedupe. Note, current ordered-extent support only supports non-compressed source extent. Support for compressed source extent will be added later. Signed-off-by: Qu Wenruo Signed-off-by:

[PATCH v14.1 10/16] btrfs: dedupe: Implement btrfs_dedupe_calc_hash interface

From: Wang Xiaoguang Unlike in-memory or on-disk dedupe method, only SHA256 hash method is supported yet, so implement btrfs_dedupe_calc_hash() interface using SHA256. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang

[PATCH v14.1 01/16] btrfs: improve inode's outstanding_extents computation

From: Wang Xiaoguang This issue was revealed by modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB, When modifying BTRFS_MAX_EXTENT_SIZE(128MB) to 64KB, fsstress test often gets these warnings from btrfs_destroy_inode():

[PATCH v14.1 07/16] btrfs: dedupe: Introduce function to remove hash from in-memory tree

From: Wang Xiaoguang Introduce static function inmem_del() to remove hash from in-memory dedupe tree. And implement btrfs_dedupe_del() and btrfs_dedup_disable() interfaces. Also for btrfs_dedupe_disable(), add new functions to wait existing writer and block incoming

[PATCH v14.1 09/16] btrfs: dedupe: Introduce function to search for an existing hash

From: Wang Xiaoguang Introduce static function inmem_search() to handle the job for in-memory hash tree. The trick is, we must ensure the delayed ref head is not being run at the time we search the for the hash. With inmem_search(), we can implement the

[PATCH v14.1 12/16] btrfs: dedupe: Inband in-memory only de-duplication implement

Core implement for inband de-duplication. It reuse the async_cow_start() facility to do the calculate dedupe hash. And use dedupe hash to do inband de-duplication at extent level. The work flow is as below: 1) Run delalloc range for an inode 2) Calculate hash for the delalloc range at the unit of

[PATCH v14.1 06/16] btrfs: dedupe: Introduce function to add hash into in-memory tree

From: Wang Xiaoguang Introduce static function inmem_add() to add hash into in-memory tree. And now we can implement the btrfs_dedupe_add() interface. Signed-off-by: Qu Wenruo Signed-off-by: Wang Xiaoguang

[PATCH v14.1 13/16] btrfs: Introduce DEDUPE reserve type to fix false enospc for in-band dedupe

From: Wang Xiaoguang We can trigger false ENOSPC error if enabling in-band dedupe. This is the same reason of compress false ENOSPC error: Difference in max extent size can lead to too much space reserved for metadata. Since we already have type-based reserve

[PATCH v14.1 08/16] btrfs: delayed-ref: Add support for increasing data ref under spinlock

For in-band dedupe, btrfs needs to increase data ref with delayed_ref locked, so add a new function btrfs_add_delayed_data_ref_lock() to increase extent ref with delayed_refs already locked. Signed-off-by: Qu Wenruo Reviewed-by: Josef Bacik ---

another latest mainline crash in xfstests

And if this isn't enough it seems generic/166 hangs after finishing the main test (which already takes a very long time): generic/113 16s ...[ 2498.548221] run fstests generic/113 at 2016-11-14 17:04:56 [ 2498.984322] BTRFS info (device vdb): disk space caching is enabled [ 2499.983635] BTRFS

Re: Announcing btrfs-dedupe

On 14/11/16 19:07, Zygo Blaxell wrote: On Mon, Nov 07, 2016 at 07:49:51PM +0100, James Pharaoh wrote: Annoyingly I can't find this now, but I definitely remember reading someone, apparently someone knowledgable, claim that the latest version of the kernel which I was using at the time, still

Re: [Bug 186671] New: OOM on system with just rsync running 32GB of ram 30GB of pagecache

Pretty sure it was the system after the OOM just did a history search to check, though it is 3 days afterwards and several OOMs killed several processes in somewhat rapid succession, I just listed the 1st. I'll turn on CONFIG_DEBUG_VM and reboot again. On Mon, Nov 14, 2016 at 12:04 PM, Vlastimil

Re: [Bug 186671] New: OOM on system with just rsync running 32GB of ram 30GB of pagecache

On 11/14/2016 02:27 PM, E V wrote: > System is an intel dual socket Xeon E5620, 7500/5520/5500/X58 ICH10 > family according to lspci. Anyways 4.8.4 OOM'd while I was gone. I'll > download the current 4.9rc and reboot, but in the mean time here's > xxd, vmstat & kern.log output: > 8532039

Re: Announcing btrfs-dedupe

On Mon, Nov 07, 2016 at 07:49:51PM +0100, James Pharaoh wrote: > Annoyingly I can't find this now, but I definitely remember reading someone, > apparently someone knowledgable, claim that the latest version of the kernel > which I was using at the time, still suffered from issues regarding the >

Re: [PATCH v4] btrfs: make block group flags in balance printks human-readable

On Mon, Nov 14, 2016 at 06:44:34PM +0100, Adam Borowski wrote: > They're not even documented anywhere, letting users with no recourse but > to RTFS. It's no big burden to output the bitfield as words. > > Also, display unknown flags as hex. > > Signed-off-by: Adam Borowski

Re: Announcing btrfs-dedupe

On Mon, Nov 14, 2016 at 07:22:59PM +0100, James Pharaoh wrote: > On 14/11/16 19:07, Zygo Blaxell wrote: > >There is also a still-unresolved problem where the filesystem CPU usage > >rises exponentially for some operations depending on the number of shared > >references to an extent. Files which

[PATCH v2 3/6] btrfs-progs: add OPEN_CTREE_INVALIDATE_FST flag

From: Omar Sandoval If this flag is passed to open_ctree(), we'll clear the FREE_SPACE_TREE_VALID compat_ro bit. The kernel will then reconstruct the free space tree the next time the filesystem is mounted. Reviewed-by: Qu Wenruo Signed-off-by: Omar

[PATCH v2 5/6] btrfs-progs: implement btrfs check --clear-space-cache v2

From: Omar Sandoval Reviewed-by: Qu Wenruo Signed-off-by: Omar Sandoval --- Documentation/btrfs-check.asciidoc | 14 +- cmds-check.c | 34 +- 2 files changed, 34

[PATCH v2 2/6] btrfs-progs: format FREE_SPACE_TREE{,_VALID} nicely in dump-super

From: Omar Sandoval Signed-off-by: Omar Sandoval --- cmds-inspect-dump-super.c | 24 1 file changed, 24 insertions(+) diff --git a/cmds-inspect-dump-super.c b/cmds-inspect-dump-super.c index d9f7bfb..ba0d708 100644 ---

[PATCH v2 0/6] btrfs-progs: better space_cache=v2 support

From: Omar Sandoval Hi, Cover letter from v1: This series implements some support for space_cache=v2 in btrfs-progs. In particular, this adds support for `btrfs check --clear-space-cache v2`, proper printing of the free space tree flags for `btrfs inspect-internal dump-super`,

[PATCH v2 1/6] btrfs-progs: add the FREE_SPACE_TREE_VALID compat_ro bit definition

From: Omar Sandoval This is just the definition; we don't support it yet. Signed-off-by: Omar Sandoval --- ctree.h | 13 + 1 file changed, 13 insertions(+) diff --git a/ctree.h b/ctree.h index d47f0ae..d67b852 100644 --- a/ctree.h +++ b/ctree.h @@

[PATCH v2 6/6] btrfs-progs: document space_cache=v2 more thoroughly

From: Omar Sandoval Signed-off-by: Omar Sandoval --- Documentation/btrfs-man5.asciidoc | 43 +++ 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/Documentation/btrfs-man5.asciidoc

[PATCH v2 4/6] btrfs-progs: add btrfs_clear_free_space_tree() from the kernel

From: Omar Sandoval Reviewed-by: Qu Wenruo Signed-off-by: Omar Sandoval --- ctree.h | 6 extent-tree.c | 11 +++ free-space-tree.c | 91 +++ free-space-tree.h |

Re: Announcing btrfs-dedupe

On Mon, Nov 14, 2016 at 01:39:02PM -0500, Austin S. Hemmelgarn wrote: > On 2016-11-14 13:22, James Pharaoh wrote: > >One thing I am keen to understand is if BTRFS will automatically ignore > >a request to deduplicate a file if it is already deduplicated? Given the > >performance I see when doing a

Re: Announcing btrfs-dedupe

On 2016-11-14 13:22, James Pharaoh wrote: On 14/11/16 19:07, Zygo Blaxell wrote: On Mon, Nov 07, 2016 at 07:49:51PM +0100, James Pharaoh wrote: Annoyingly I can't find this now, but I definitely remember reading someone, apparently someone knowledgable, claim that the latest version of the

Re: [PATCH 1/2] Btrfs: fix file extent corruption

On 11/14/2016 02:06 PM, Josef Bacik wrote: In order to do hole punching we have a block reserve to hold the reservation we need to drop the extents in our range. Since we could end up dropping a lot of extents we set rsv->failfast so we can just loop around again and drop the remaining of the

[PATCH 2/2] Btrfs: abort transaction if fill_holes() fails

At this point we will have dropped extent entries from the file, so if we fail to insert the new hole entries then we are leaving the fs in a corrupt state (albeit an easily fixed one). Abort the transaciton if this happens so we can avoid corrupting the fs. Thanks, Signed-off-by: Josef Bacik

[PATCH 1/2] Btrfs: fix file extent corruption

In order to do hole punching we have a block reserve to hold the reservation we need to drop the extents in our range. Since we could end up dropping a lot of extents we set rsv->failfast so we can just loop around again and drop the remaining of the range. Unfortunately we unconditionally fill

Re: Announcing btrfs-dedupe

On Mon, Nov 14, 2016 at 02:56:51PM -0500, Austin S. Hemmelgarn wrote: > On 2016-11-14 14:51, Zygo Blaxell wrote: > >Deduplicating an extent that may might be concurrently modified during the > >dedup is a reasonable userspace request. In the general case there's > >no way for userspace to ensure

Re: Announcing btrfs-dedupe

On Mon, Nov 14, 2016 at 09:07:51PM +0100, James Pharaoh wrote: > On 14/11/16 20:51, Zygo Blaxell wrote: > >On Mon, Nov 14, 2016 at 01:39:02PM -0500, Austin S. Hemmelgarn wrote: > >>On 2016-11-14 13:22, James Pharaoh wrote: > >>>One thing I am keen to understand is if BTRFS will automatically

Re: [PATCH 1/2] Btrfs: fix file extent corruption

On Mon, Nov 14, 2016 at 02:06:21PM -0500, Josef Bacik wrote: > In order to do hole punching we have a block reserve to hold the reservation > we > need to drop the extents in our range. Since we could end up dropping a lot > of > extents we set rsv->failfast so we can just loop around again and

Re: Announcing btrfs-dedupe

On 2016-11-14 14:51, Zygo Blaxell wrote: On Mon, Nov 14, 2016 at 01:39:02PM -0500, Austin S. Hemmelgarn wrote: On 2016-11-14 13:22, James Pharaoh wrote: One thing I am keen to understand is if BTRFS will automatically ignore a request to deduplicate a file if it is already deduplicated? Given

Re: Announcing btrfs-dedupe

On 14/11/16 20:51, Zygo Blaxell wrote: On Mon, Nov 14, 2016 at 01:39:02PM -0500, Austin S. Hemmelgarn wrote: On 2016-11-14 13:22, James Pharaoh wrote: One thing I am keen to understand is if BTRFS will automatically ignore a request to deduplicate a file if it is already deduplicated? Given

Re: corrupt leaf, slot offset bad

Am Tue, 11 Oct 2016 07:09:49 -0700 schrieb Liu Bo : > On Tue, Oct 11, 2016 at 02:48:09PM +0200, David Sterba wrote: > > Hi, > > > > looks like a lot of random bitflips. > > > > On Mon, Oct 10, 2016 at 11:50:14PM +0200, a...@aron.ws wrote: > > > item 109 has a few strange