Re: [PATCH 1/2] btrfs: introduce priority based delalloc shrink mechanism

hi, On 10/13/2016 01:20 AM, Josef Bacik wrote: On 10/12/2016 05:03 AM, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete tests, it's because shrink_delalloc()

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 05:10:18PM -0400, Zygo Blaxell wrote: > On Wed, Oct 12, 2016 at 09:55:28PM +0200, Adam Borowski wrote: > > On Wed, Oct 12, 2016 at 01:19:37PM -0400, Zygo Blaxell wrote: > > > I had been thinking that we could inject "plug" extents to fill up > > > RAID5 stripes. > > Your

Unable to rescue RAID5

I am using Btrfs RAID5 with 10x4T disks. Around 2 weeks ago, one disk of it was taking long time when read and write. So, I tried replace the disk with new disk by replace command ( btrfs replace ... ). But, it failed. So, I added new disk into the array by add command and then deleted the bad

[PATCH] btrfs: make file clone aware of fatal signals

Indeed this just make the behavior similar to xfs when process has fatal signals pending, and it'll make fstests/generic/298 happy. Signed-off-by: Wang Xiaoguang --- fs/btrfs/ioctl.c | 5 + 1 file changed, 5 insertions(+) diff --git a/fs/btrfs/ioctl.c

Re: RAID system with adaption to changed number of disks

At 10/13/2016 01:19 AM, Zygo Blaxell wrote: On Wed, Oct 12, 2016 at 01:48:58PM +0800, Qu Wenruo wrote: btrfs also doesn't avoid the raid5 write hole properly. After a crash, a btrfs filesystem (like mdadm raid[56]) _must_ be scrubbed (resynced) to reconstruct any parity that was damaged by

Re: [PATCH] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

On Wed, Oct 12, 2016 at 10:23:52PM +0100, Filipe Manana wrote: > On Tue, Sep 6, 2016 at 10:51 PM, Liu Bo wrote: > > Hi Filipe, > > > > On Mon, Sep 05, 2016 at 04:28:09PM +0100, Filipe Manana wrote: > >> On Fri, Sep 2, 2016 at 8:35 PM, Liu Bo wrote: >

Re: Incremental send robustness question

On 10/13/2016 01:47 AM, Sean Greenslade wrote: > On Thu, Oct 13, 2016 at 01:14:51AM +0200, Hans van Kranenburg wrote: >> On 10/13/2016 12:29 AM, Sean Greenslade wrote: >>> And while we're at it, what are the failure modes for incremental sends? >>> Will it throw an error if the parents don't

Re: Incremental send robustness question

On Thu, Oct 13, 2016 at 01:14:51AM +0200, Hans van Kranenburg wrote: > On 10/13/2016 12:29 AM, Sean Greenslade wrote: > > Hi, all. I have a question about a backup plan I have involving > > send/receive. As far as I can tell, there's no way to to resume a send > > that has been interrupted. In

Re: Incremental send robustness question

On 10/13/2016 12:29 AM, Sean Greenslade wrote: > Hi, all. I have a question about a backup plan I have involving > send/receive. As far as I can tell, there's no way to to resume a send > that has been interrupted. In this case, my interruption comes from an > overbearing firewall that doesn't

Re: Incremental send robustness question

On Wed, Oct 12, 2016 at 06:29:55PM -0400, Sean Greenslade wrote: > Hi, all. I have a question about a backup plan I have involving > send/receive. As far as I can tell, there's no way to to resume a send > that has been interrupted. In this case, my interruption comes from an > overbearing

Re: Incremental send robustness question

On Wed, Oct 12, 2016 at 4:29 PM, Sean Greenslade wrote: > Hi, all. I have a question about a backup plan I have involving > send/receive. As far as I can tell, there's no way to to resume a send > that has been interrupted. In this case, my interruption comes from an >

Incremental send robustness question

Hi, all. I have a question about a backup plan I have involving send/receive. As far as I can tell, there's no way to to resume a send that has been interrupted. In this case, my interruption comes from an overbearing firewall that doesn't like long-lived connections. I'm trying to do the initial

Re: raid6 file system in a bad state

On Wed, Oct 12, 2016 at 11:59 AM, Jason D. Michaelson wrote: > With the bad disc in place: > > root@castor:~/btrfs-progs# ./btrfs restore -t 4844272943104 -D /dev/sda > /dev/null > parent transid verify failed on 4844272943104 wanted 161562 found 161476 > parent

Re: [PATCH] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

On Tue, Sep 6, 2016 at 10:51 PM, Liu Bo wrote: > Hi Filipe, > > On Mon, Sep 05, 2016 at 04:28:09PM +0100, Filipe Manana wrote: >> On Fri, Sep 2, 2016 at 8:35 PM, Liu Bo wrote: >> > This can only happen with CONFIG_BTRFS_FS_CHECK_INTEGRITY=y. >> > >> >

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 09:55:28PM +0200, Adam Borowski wrote: > On Wed, Oct 12, 2016 at 01:19:37PM -0400, Zygo Blaxell wrote: > > On Wed, Oct 12, 2016 at 01:48:58PM +0800, Qu Wenruo wrote: > > > In fact, the _concept_ to solve such RMW behavior is quite simple: > > > > > > Make sector size equal

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 11:19 AM, Zygo Blaxell wrote: > Degraded RAID5 is not RAID0. RAID5 has strict constraints that RAID0 > does not. The way a RAID5 implementation behaves in degraded mode is > the thing that usually matters after a disk fails. Is there

Re: RAID system with adaption to changed number of disks

On Thu, Oct 13, 2016 at 12:33:31AM +0500, Roman Mamedov wrote: > On Wed, 12 Oct 2016 15:19:16 -0400 > Zygo Blaxell wrote: > > > I'm not even sure btrfs does this--I haven't checked precisely what > > it does in dup mode. It could send both copies of metadata to

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 01:19:37PM -0400, Zygo Blaxell wrote: > On Wed, Oct 12, 2016 at 01:48:58PM +0800, Qu Wenruo wrote: > > In fact, the _concept_ to solve such RMW behavior is quite simple: > > > > Make sector size equal to stripe length. (Or vice versa if you like) > > > > Although the

Re: RAID system with adaption to changed number of disks

On Wed, 12 Oct 2016 15:19:16 -0400 Zygo Blaxell wrote: > I'm not even sure btrfs does this--I haven't checked precisely what > it does in dup mode. It could send both copies of metadata to the > disks with a single barrier to separate both metadata updates from >

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 01:31:41PM -0400, Zygo Blaxell wrote: > On Wed, Oct 12, 2016 at 12:25:51PM +0500, Roman Mamedov wrote: > > Zygo Blaxell wrote: > > > > > A btrfs -dsingle -mdup array on a mdadm raid[56] device might have a > > > snowball's chance in hell of

Re: [PATCH 1/7] Btrfs: replace BUG() with WARN_ONCE in raid56

On Wed, Oct 12, 2016 at 05:06:55PM +0200, David Sterba wrote: > On Mon, May 16, 2016 at 10:32:48AM +0200, David Sterba wrote: > > On Sun, May 15, 2016 at 04:19:28PM +0200, Holger Hoffstätte wrote: > > > On 05/14/16 02:06, Liu Bo wrote: > > > > This BUG() has been triggered by a fuzz testing image,

RE: raid6 file system in a bad state

> -Original Message- > From: ch...@colorremedies.com [mailto:ch...@colorremedies.com] On > Behalf Of Chris Murphy > Sent: Tuesday, October 11, 2016 3:38 PM > To: Jason D. Michaelson; Btrfs BTRFS > Cc: Chris Murphy > Subject: Re: raid6 file system in a bad state > > readding btrfs > >

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 12:25:51PM +0500, Roman Mamedov wrote: > Zygo Blaxell wrote: > > > A btrfs -dsingle -mdup array on a mdadm raid[56] device might have a > > snowball's chance in hell of surviving a disk failure on a live array > > with only data losses.

Re: Btrfs progs build fails for 4.8

On Wed, Oct 12, 2016 at 10:01:27PM +0800, Qu Wenruo wrote: > > > On 10/12/2016 09:58 PM, Abhay Sachan wrote: > > Hi, > > I tried building latest btrfs progs on CentOS 6, "./configure" failed > > with the error: > > > > checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... no > >

Re: [PATCH 1/2] btrfs: introduce priority based delalloc shrink mechanism

On 10/12/2016 05:03 AM, Wang Xiaoguang wrote: Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete tests, it's because shrink_delalloc() does not write enough delalloc bytes and wait

Re: [PATCH 2/2] btrfs: try to satisfy metadata requests if wen can overcommit

On 10/12/2016 05:03 AM, Wang Xiaoguang wrote: In shrink_delalloc(), if can_overcommit() returns true, shrink_delalloc() will give up shrinking delalloc bytes, in this case we should check whether some tickcts' requests can overcommit, if some can, we can satisfy them timely and directly.

Re: RAID system with adaption to changed number of disks

On Wed, Oct 12, 2016 at 01:48:58PM +0800, Qu Wenruo wrote: > >btrfs also doesn't avoid the raid5 write hole properly. After a crash, > >a btrfs filesystem (like mdadm raid[56]) _must_ be scrubbed (resynced) > >to reconstruct any parity that was damaged by an incomplete data stripe > >update. > >

Re: [PATCH 1/2] btrfs: try to satisfy metadata requests when every flush_space() returns

On 10/12/2016 03:27 AM, Wang Xiaoguang wrote: hi, On 10/07/2016 09:16 PM, Josef Bacik wrote: On 09/21/2016 02:59 AM, Wang Xiaoguang wrote: In flush_space()->shrink_delalloc(), if can_overcommit() returns true, though no bytes added to space_info, we still may satisfy some requests.

Re: [PATCH] btrfs: kill unused writepage_io_hook callback

On Wed, Oct 12, 2016 at 06:40:09PM +0200, David Sterba wrote: > It seems to be long time unused, since 2008 and > 6885f308b5570 ("Btrfs: Misc 2.6.25 updates"). > > Propagating the removal touches some code but has no functional effect. > > Signed-off-by: David Sterba Please

[PATCH] btrfs: kill unused writepage_io_hook callback

It seems to be long time unused, since 2008 and 6885f308b5570 ("Btrfs: Misc 2.6.25 updates"). Propagating the removal touches some code but has no functional effect. Signed-off-by: David Sterba --- fs/btrfs/extent_io.c | 39 ---

Re: [PATCH v2] btrfs: block incompatible optional features at scan

On Wed, Mar 16, 2016 at 08:32:35AM +0800, Anand Jain wrote: > For the matter of completeness we need to check if the device > being scanned has features that are known to the kernel. As of > now if it doesn't - the mount will fails, then what is the point > in having those devices added to the

Re: [PATCH 1/7] Btrfs: replace BUG() with WARN_ONCE in raid56

On Mon, May 16, 2016 at 10:32:48AM +0200, David Sterba wrote: > On Sun, May 15, 2016 at 04:19:28PM +0200, Holger Hoffstätte wrote: > > On 05/14/16 02:06, Liu Bo wrote: > > > This BUG() has been triggered by a fuzz testing image, but in fact > > > btrfs can handle this gracefully by returning -EIO.

Re: [PATCH v1] btrfs: Avoid reading out unnecessary extent tree blocks when mounting

On Thu, Jul 07, 2016 at 10:11:59AM +0800, Qu Wenruo wrote: > Btrfs_read_block_groups() function is the most time consuming function > if the fs is large and filled with small extents. > > For a btrfs filled with 100,000,000 16K sized files, mount the fs takes > about 10 seconds. > > While ftrace

Re: Btrfs progs build fails for 4.8

On 10/12/16 9:45 AM, Abhay Sachan wrote: > Hi Qu, > I am running latest 4.8.1, which I compiled on the machine itself. You likely still have the fiemap.h from Centos' kernel-headers rpm, which is from an older kernel and does not define it. -Eric > Linux vm88 4.8.1 #1 SMP Thu Oct 13 10:33:08

Re: btrfs bio linked list corruption.

On Wed, Oct 12, 2016 at 09:47:17AM -0400, Dave Jones wrote: > On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > > This is from Linus' current tree, with Al's iovec fixups on top. > > > > > > [ cut here

Re: Btrfs progs build fails for 4.8

Hi Qu, I am running latest 4.8.1, which I compiled on the machine itself. Linux vm88 4.8.1 #1 SMP Thu Oct 13 10:33:08 IST 2016 x86_64 x86_64 x86_64 GNU/Linux Regards, Abhay On Wed, Oct 12, 2016 at 7:31 PM, Qu Wenruo wrote: > > > On 10/12/2016 09:58 PM, Abhay Sachan

Re: btrfs bio linked list corruption.

On 10/12/2016 10:40 AM, Dave Jones wrote: On Wed, Oct 12, 2016 at 09:47:17AM -0400, Dave Jones wrote: > On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > > This is from Linus' current tree, with Al's iovec fixups on

Re: [PATCH] btrfs: change btrfs_csum_final result param type to u8

On Mon, Sep 19, 2016 at 07:22:28PM +0200, David Sterba wrote: > On Sun, Sep 18, 2016 at 12:10:34AM +0100, Domagoj Tršan wrote: > > csum member of struct btrfs_super_block has array type of u8. It makes sense > > that function btrfs_csum_final should be also declared to accept u8 *. I > > changed

Re: Btrfs progs build fails for 4.8

On 10/12/2016 09:58 PM, Abhay Sachan wrote: Hi, I tried building latest btrfs progs on CentOS 6, "./configure" failed with the error: checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... no configure: error: no definition of FIEMAP_EXTENT_SHARED found Any ideas what am I lacking in

Btrfs progs build fails for 4.8

Hi, I tried building latest btrfs progs on CentOS 6, "./configure" failed with the error: checking for FIEMAP_EXTENT_SHARED defined in linux/fiemap.h... no configure: error: no definition of FIEMAP_EXTENT_SHARED found Any ideas what am I lacking in the system? Regards, Abhay -- To unsubscribe

Re: btrfs bio linked list corruption.

On Tue, Oct 11, 2016 at 11:54:09AM -0400, Chris Mason wrote: > > > On 10/11/2016 10:45 AM, Dave Jones wrote: > > This is from Linus' current tree, with Al's iovec fixups on top. > > > > [ cut here ] > > WARNING: CPU: 1 PID: 3673 at lib/list_debug.c:33

Btrfs progs release 4.8.1

Hi, btrfs-progs 4.8.1 have been released, with a short delay than estimated. This is a bugfix release, namely the build has been fixed (32bit and no-backtrace). Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/ Git:

Re: [PATCH] btrfs: fix silent data corruption while reading compressed inline extents

On Wed, Oct 12, 2016 at 09:59:03AM +0100, Filipe Manana wrote: > On Wed, Oct 12, 2016 at 3:28 AM, Zygo Blaxell > wrote: > > rsync -S causes a large number of small writes separated by small seeks > > to form sparse holes in files that contain runs of zero bytes.

Re: Btrfs unmountable - claims it is missing a volume

Hi, On Sat, Sep 10, 2016 at 08:59:21AM +0200, Grdykopląs Namorzyn wrote: > Hello, > > While my btrfs mounts ok on several instances of my home OpenElec, but > on ArchLinux it fails with: > > [174828.954975] BTRFS error (device sdc): super_num_devices 3 mismatch > with num_devices 2 found here

Re: raid levels and NAS drives

On 2016-10-11 18:10, Nicholas D Steeves wrote: On Mon, Oct 10, 2016 at 08:07:53AM -0400, Austin S. Hemmelgarn wrote: On 2016-10-09 19:12, Charles Zeitler wrote: Is there any advantage to using NAS drives under RAID levels, as oppposed to regular 'desktop' drives for BTRFS? [...] So, as for

Re: [PATCH 3/5] Btrfs: incremental send, add gen in waiting_dir_move for some corner case

On Wed, Oct 12, 2016 at 9:12 AM, robbieko wrote: > From: Robbie Ko > > There a some case similar as before. As before what? Each change log should be complete and the reader is not supposed to guess what's the previous patch or commit this is

Re: [PATCH 4/5] Btrfs: incremental send, add gen check in did_overwrite_ref

On Wed, Oct 12, 2016 at 9:12 AM, robbieko wrote: > From: Robbie Ko > > There a some case similar as before. As before what? Each change log should be complete and the reader is not supposed to guess what's the previous patch or commit this is

Re: [PATCH] btrfs: fix silent data corruption while reading compressed inline extents

On Wed, Oct 12, 2016 at 3:28 AM, Zygo Blaxell wrote: > rsync -S causes a large number of small writes separated by small seeks > to form sparse holes in files that contain runs of zero bytes. Rarely, > this can lead btrfs to write a file with a compressed inline

Re: [PATCH] Btrfs: fix enospc in punch hole

On Tue, Oct 11, 2016 at 9:58 AM, robbieko wrote: > Hi Filipe: > > because btrfs_calc_trunc_metadata_size is reserved leafsize + nodesize * (8 > - 1) > assume leafsize is the same as nodesize The leaf size is always the same as the node size. There are no exceptions. > ,

Re: [PATCH 2/5] Btrfs: incremental send, add gen for is_waiting_for_rm when some corner case

On Wed, Oct 12, 2016 at 9:12 AM, robbieko wrote: > From: Robbie Ko > > There a one case for old_gen waiting_for_rm, > but new_gen use get_cur_path with the same inode. > > Example: > Parent snapshot: > | dir258/ (ino 258, dir) >

Re: [PATCH 1/5] Btrfs: incremental send, fix don't skip root inode in overwrite_ref

On Wed, Oct 12, 2016 at 9:12 AM, robbieko wrote: > From: Robbie Ko > > When root dir item change, don't skip will_overwrite_ref, > because root inode always exist. What do you mean by root dir item change? You mean indoe 256 changed, how did it

Re: [PATCH] Btrfs: fix fsync deadlock in log_new_dir_dentries

On Tue, Oct 11, 2016 at 8:47 AM, robbieko wrote: > Hi Filipe: > > why did you replace the continue statement with a break statement: > because we released ahead of the path, it can not continue to use, > need to jump out, and then go to again. Seeing the code and not only

Re: [PATCH 5/5] Btrfs: incremental send, add gen check if has waiting_dir_move in the will_overwrite_ref

On Wed, Oct 12, 2016 at 9:12 AM, robbieko wrote: > From: Robbie Ko > > There a some case similar as before. As before what? Each change log should be complete and the reader is not supposed to guess what's the previous patch or commit this is

[PATCH 1/2] btrfs: introduce priority based delalloc shrink mechanism

Since commit b02441999efcc6152b87cd58e7970bb7843f76cf, we don't wait all ordered extents, but I run into some enospc errors when doing large file create and delete tests, it's because shrink_delalloc() does not write enough delalloc bytes and wait them finished: From: Miao Xie

[PATCH 2/2] btrfs: try to satisfy metadata requests if wen can overcommit

In shrink_delalloc(), if can_overcommit() returns true, shrink_delalloc() will give up shrinking delalloc bytes, in this case we should check whether some tickcts' requests can overcommit, if some can, we can satisfy them timely and directly. Signed-off-by: Wang Xiaoguang

[patch] Btrfs: remove some no-op casts

We cast 0 to a u8 but then because of type promotion, it's immediately cast to int back to int before we do a bitwise negate. The cast doesn't matter in this case, the code works as intended. It causes a static checker warning though so let's remove it. Signed-off-by: Dan Carpenter

Re: [PATCH 1/2] btrfs: try to satisfy metadata requests when every flush_space() returns

hi, On 10/07/2016 09:16 PM, Josef Bacik wrote: On 09/21/2016 02:59 AM, Wang Xiaoguang wrote: In flush_space()->shrink_delalloc(), if can_overcommit() returns true, though no bytes added to space_info, we still may satisfy some requests. Signed-off-by: Wang Xiaoguang

[PATCH 4/5] Btrfs: incremental send, add gen check in did_overwrite_ref

From: Robbie Ko There a some case similar as before. add check parent generation in the did_overwrite_ref. Signed-off-by: Robbie Ko --- fs/btrfs/send.c | 13 + 1 file changed, 13 insertions(+) diff --git a/fs/btrfs/send.c

[PATCH 2/5] Btrfs: incremental send, add gen for is_waiting_for_rm when some corner case

From: Robbie Ko There a one case for old_gen waiting_for_rm, but new_gen use get_cur_path with the same inode. Example: Parent snapshot: | dir258/ (ino 258, dir) |--- dir257/ (ino 257, dir) | dir259/ (ino 259, dir) Send snapshot: |

[PATCH 3/5] Btrfs: incremental send, add gen in waiting_dir_move for some corner case

From: Robbie Ko There a some case similar as before. add compare generation with dir items, and waiting_dir_move in the can_rmdir. Signed-off-by: Robbie Ko --- fs/btrfs/send.c | 14 +++--- 1 file changed, 11 insertions(+), 3 deletions(-)

[PATCH 5/5] Btrfs: incremental send, add gen check if has waiting_dir_move in the will_overwrite_ref

From: Robbie Ko There a some case similar as before. add check generation if has waiting_dir_move in the will_overwrite_ref Signed-off-by: Robbie Ko --- fs/btrfs/send.c | 7 ++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git

[PATCH 1/5] Btrfs: incremental send, fix don't skip root inode in overwrite_ref

From: Robbie Ko When root dir item change, don't skip will_overwrite_ref, because root inode always exist. Example: Parent snapshot: | a1/ (ino 257, dir) | a2/ (ino 258, dir) Send snapshot: | a2 (ino 257, file) ERROR: rename o257-29-0 -> a2 failed: Is a

[PATCH 0/5] Btrfs: incremental send, fix serval case for root and gen

From: Robbie Ko Patch for fix btrfs incremental send. These patches base on v4.8.0-rc8 Robbie Ko (5): Btrfs: incremental send, fix don't skip root inode in overwrite_ref Btrfs: incremental send, add gen for is_waiting_for_rm when some corner case Btrfs:

Re: RAID system with adaption to changed number of disks

On Tue, 11 Oct 2016 17:58:22 -0600 Chris Murphy wrote: > But consider the identical scenario with md or LVM raid5, or any > conventional hardware raid5. A scrub check simply reports a mismatch. > It's unknown whether data or parity is bad, so the bad data strip is >

Re: RAID system with adaption to changed number of disks

Missing device is the _only_ thing the current design handles. Right. below patches in the ML added two more device states offline and failed. It is tested with raid1. [PATCH 11/13] btrfs: introduce device dynamic state transition to offline or failed [PATCH 12/13] btrfs: check device