Device Delete Stalls - Addition

2018-08-23 Thread Stefan Malte Schumacher
Hello everybody, I think this might be useful: root@mars:~# btrfs dev usage /mnt/btrfs-raid/ /dev/sda, ID: 1 Device size: 3.64TiB Device slack: 0.00B Data,RAID1: 7.00GiB Metadata,RAID1: 1.00GiB Unallocated: 3.63TiB Yours

fs/btrfs/volumes.c:6114 suspicious rcu_dereference_check() usage!

2018-08-23 Thread David Howells
I'm seeing the attached message generated from this line: btrfs_debug_in_rcu(fs_info, "btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u", bio_op(bio), bio->bi_opf, (u64)bio->bi_iter.bi_sector, (u_long)dev->bdev->bd_dev,

Re: fs/btrfs/volumes.c:6114 suspicious rcu_dereference_check() usage!

2018-08-23 Thread David Sterba
On Thu, Aug 23, 2018 at 04:49:11PM +0100, David Howells wrote: > I'm seeing the attached message generated from this line: > > btrfs_debug_in_rcu(fs_info, > "btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u", > bio_op(bio), bio->bi_opf,

Re: fs/btrfs/volumes.c:6114 suspicious rcu_dereference_check() usage!

2018-08-23 Thread David Howells
David Sterba wrote: > The code previously had explicit rcu_lock/unlock, now it uses the > btrfs_debug_in_rcu helper which is supposed to provide that. It's > possible that the helper is missing it due to some #ifdef mess, but I > don't see it. I preprocessed the function and extracted

Re: Reproducer for "compressed data + hole data corruption bug, 2018 editiion"

2018-08-23 Thread Zygo Blaxell
On Thu, Aug 23, 2018 at 01:10:48PM +0800, Qu Wenruo wrote: > On 2018/8/23 上午11:11, Zygo Blaxell wrote: > > This is a repro script for a btrfs bug that causes corrupted data reads > > when reading a mix of compressed extents and holes. The bug is > > reproducible on at least kernels v4.1..v4.18. >

Re: Device Delete Stalls

2018-08-23 Thread Austin S. Hemmelgarn
On 2018-08-23 10:04, Stefan Malte Schumacher wrote: Hallo, I originally had RAID with six 4TB drives, which was more than 80 percent full. So now I bought a 10TB drive, added it to the Array and gave the command to remove the oldest drive in the array. btrfs device delete /dev/sda

Re: btrfs-convert missing in btrfs-tools v4.15.1

2018-08-23 Thread Nicholas D Steeves
Hi everyone, Sorry for the delay replying, I've been busy with other work. Reply follows inline. P.S. sorry about the table in this email that is 99 columns wide. On Thu, Aug 09, 2018 at 01:50:46PM +0200, David Sterba wrote: > On Sat, Jul 28, 2018 at 05:34:49PM -0400, Nicholas D Steeves wrote:

Re: btrfs-convert missing in btrfs-tools v4.15.1

2018-08-23 Thread Nicholas D Steeves
Hi Qu, On Sun, Jul 29, 2018 at 07:44:05AM +0800, Qu Wenruo wrote: > > > On 2018年07月29日 05:34, Nicholas D Steeves wrote: > > Resending because I forgot to CC list. > > > > Hi jkexcel, > > > > On 28 July 2018 at 16:50, jkexcel wrote: > >> > >> I'm an end user trying to use btrfs-convert but

Device Delete Stalls

2018-08-23 Thread Stefan Malte Schumacher
Hallo, I originally had RAID with six 4TB drives, which was more than 80 percent full. So now I bought a 10TB drive, added it to the Array and gave the command to remove the oldest drive in the array. btrfs device delete /dev/sda /mnt/btrfs-raid I kept a terminal with "watch btrfs fi show"

[PATCH] btrfs-progs: dduper - BTRFS offline deduplication tool

2018-08-23 Thread Lakshmipathi Ganapathi
dduper is an offline dedupe tool. It works by fetching checksum from BTRFS csum tree, instead of reading whole file blocks and computing checksum. This tool relies on output from 'btrfs inspect-internal dump-csum' command. Signed-off-by: Lakshmipathi.G --- dduper | 310

Re: fs/btrfs/volumes.c:6114 suspicious rcu_dereference_check() usage!

2018-08-23 Thread Misono Tomohiro
On 2018/08/24 0:49, David Howells wrote: > I'm seeing the attached message generated from this line: > > btrfs_debug_in_rcu(fs_info, > "btrfs_map_bio: rw %d 0x%x, sector=%llu, dev=%lu (%s id %llu), size=%u", > bio_op(bio), bio->bi_opf, (u64)bio->bi_iter.bi_sector, >

Re: [patch] file dedupe (and maybe clone) data corruption (was Re: [PATCH] generic: test for deduplication between different files)

2018-08-23 Thread Zygo Blaxell
On Thu, Aug 23, 2018 at 08:58:49AM -0400, Zygo Blaxell wrote: > On Mon, Aug 20, 2018 at 08:33:49AM -0700, Darrick J. Wong wrote: > > On Mon, Aug 20, 2018 at 11:09:32AM +1000, Dave Chinner wrote: > > > - should we just round down the EOF dedupe request to the > > > block before EOF so dedupe

[PATCH] btrfs: ctree.h: Fix suspicious rcu usage warning in btrfs_debug_in_rcu()

2018-08-23 Thread Misono Tomohiro
commit 672d599041c8 ("btrfs: Use wrapper macro for rcu string to remove duplicate code") replaces some open coded rcu string handling with macro. It turns out that btrfs_debug_in_rcu() is used for the first time and the macro lacks lock/unlock of rcu string for non debug case (i.e. when message

Re: Device Delete Stalls

2018-08-23 Thread Chris Murphy
And by 4.14 I actually mean 4.14.60 or 4.14.62 (based on the changelog). I don't think the single patch in 4.14.62 applies to your situation.

Re: Device Delete Stalls

2018-08-23 Thread Chris Murphy
On Thu, Aug 23, 2018 at 8:04 AM, Stefan Malte Schumacher wrote: > Hallo, > > I originally had RAID with six 4TB drives, which was more than 80 > percent full. So now I bought > a 10TB drive, added it to the Array and gave the command to remove the > oldest drive in the array. > > btrfs device

dduper - Offline btrfs deduplication tool

2018-08-23 Thread Lakshmipathi.G
Hi - dduper is an offline dedupe tool. Instead of reading whole file blocks and computing checksum, It works by fetching checksum from BTRFS csum tree. This hugely improves the performance. dduper works like: - Read csum for given two files. - Find matching location. -

Re: btrfs-convert missing in btrfs-tools v4.15.1

2018-08-23 Thread Duncan
Nicholas D Steeves posted on Thu, 23 Aug 2018 14:15:18 -0400 as excerpted: >> It's in my interest to ship all tools in distros, but there's also only >> that much what the upstream community can do. If you're going to >> reconsider the status of btrfs-convert in Debian, please let me know. > >

Re: lazytime mount option—no support in Btrfs

2018-08-23 Thread Janos Toth F.
On Tue, Aug 21, 2018 at 4:10 PM Austin S. Hemmelgarn wrote: > Also, once you've got the space cache set up by mounting once writable > with the appropriate flag and then waiting for it to initialize, you > should not ever need to specify the `space_cache` option again. True. I am not sure why I

Re: Reproducer for "compressed data + hole data corruption bug, 2018 editiion"

2018-08-23 Thread Qu Wenruo
On 2018/8/24 上午12:44, Zygo Blaxell wrote: > On Thu, Aug 23, 2018 at 01:10:48PM +0800, Qu Wenruo wrote: >> On 2018/8/23 上午11:11, Zygo Blaxell wrote: >>> This is a repro script for a btrfs bug that causes corrupted data reads >>> when reading a mix of compressed extents and holes. The bug is >>>

Device Delete Stalls - Conclusion

2018-08-23 Thread Stefan Malte Schumacher
Hello everybody, first, let me thank everybody for their advice. What I did was close the terminal with the device delete-process running in it and fired it up again. It took about 5 minutes of intensive IO-Usage and the data was redistributed and and the /dev/sda/ removed from the list of

[PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-08-23 Thread Qu Wenruo
By default dump-tree does depth-first search. For 2 level trees it's completely OK, but for 3 level trees, it would be pretty hard to locate output of middle level tree nodes. Introduce --breadth-first option to do breadth-first tree dump. This is especially handy to inspect high level trees,

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-08-23 Thread Su Yue
On 08/23/2018 03:31 PM, Qu Wenruo wrote: By default dump-tree does depth-first search. For 2 level trees it's completely OK, but for 3 level trees, it would be pretty hard to locate output of middle level tree nodes. Introduce --breadth-first option to do breadth-first tree dump. This is

[PATCH] btrfs-progs: print-tree: Skip deprecated blockptr / nodesize output

2018-08-23 Thread Qu Wenruo
When printing tree nodes, we output slots like: key (EXTENT_TREE ROOT_ITEM 0) block 73625600 (17975) gen 16 The number in the parentheses is blockptr / nodesize. However this number doesn't really do any thing useful. And in fact for unaligned metadata block group (block group start bytenr is

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-08-23 Thread Nikolay Borisov
On 23.08.2018 10:31, Qu Wenruo wrote: > Introduce --breadth-first option to do breadth-first tree dump. > This is especially handy to inspect high level trees, e.g. comparing > tree reloc tree with its source tree. Will it make sense instead of exposing another option to just have a heuristics

Re: [PATCH] btrfs-progs: dump-tree: Introduce --breadth-first option

2018-08-23 Thread Qu Wenruo
On 2018/8/23 下午3:36, Nikolay Borisov wrote: > > > On 23.08.2018 10:31, Qu Wenruo wrote: >> Introduce --breadth-first option to do breadth-first tree dump. >> This is especially handy to inspect high level trees, e.g. comparing >> tree reloc tree with its source tree. > > Will it make sense

Re: [PATCH 0/5] rb_first to rb_first_cached conversion

2018-08-23 Thread Nikolay Borisov
On 22.08.2018 22:51, Liu Bo wrote: > Several structs in btrfs are using rb_first() in a while loop, it'd be > more efficient to do this with rb_first_cached() which has the O(1) > complexity. > > This patch set updates five structs which may have a large rb tree in > practice > > Liu Bo (5):

Re: [PATCH 0/5] rb_first to rb_first_cached conversion

2018-08-23 Thread Holger Hoffstätte
On 08/22/18 21:51, Liu Bo wrote: Several structs in btrfs are using rb_first() in a while loop, it'd be more efficient to do this with rb_first_cached() which has the O(1) complexity. This patch set updates five structs which may have a large rb tree in practice Great idea, works for me with

Re: [PATCH] Btrfs: use next_state in find_first_extent_bit

2018-08-23 Thread David Sterba
On Thu, Aug 23, 2018 at 03:14:53AM +0800, Liu Bo wrote: > As next_state() is already defined to get the next state, update us to use > the helper. > > No functional changes. > > Signed-off-by: Liu Bo Added to misc-next, thanks.

Re: [patch] file dedupe (and maybe clone) data corruption (was Re: [PATCH] generic: test for deduplication between different files)

2018-08-23 Thread Zygo Blaxell
On Mon, Aug 20, 2018 at 08:33:49AM -0700, Darrick J. Wong wrote: > On Mon, Aug 20, 2018 at 11:09:32AM +1000, Dave Chinner wrote: > > - is documenting rejection on request alignment grounds > > (i.e. EINVAL) in the man page sufficient for app > > developers to understand what is

Re: [PATCH v2.1] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-23 Thread David Sterba
On Tue, Aug 21, 2018 at 09:42:03AM +0800, Qu Wenruo wrote: > [BUG] > When mounting certain crafted image, btrfs will trigger kernel BUG_ON() > when try to recover balance: > -- > [ cut here ] > kernel BUG at fs/btrfs/extent-tree.c:8956! > invalid opcode: [#1]

Re: [PATCH v2.1] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-23 Thread David Sterba
On Tue, Aug 21, 2018 at 09:42:03AM +0800, Qu Wenruo wrote: > [BUG] > When mounting certain crafted image, btrfs will trigger kernel BUG_ON() > when try to recover balance: > -- > [ cut here ] Please don't use lines starting with ---, an empty line is fine, or you can

Re: [PATCH v2.1] btrfs: Handle owner mismatch gracefully when walking up tree

2018-08-23 Thread Qu Wenruo
On 2018/8/23 下午9:24, David Sterba wrote: > On Tue, Aug 21, 2018 at 09:42:03AM +0800, Qu Wenruo wrote: >> [BUG] >> When mounting certain crafted image, btrfs will trigger kernel BUG_ON() >> when try to recover balance: >> -- >> [ cut here ] > > Please don't use lines