Send-recieve performance

Hello, we use backuppc to backup our hosting machines. I have recently migrated it to btrfs, so we can use send-recieve for offsite backups of our backups. I have several btrfs volumes, each hosts nspawn container, which runs in /system subvolume and has backuppc data in /backuppc subvolume .

[PATCH] btrfs: Update quick and auto tag for btrfs group

Update the following quick/auto tag based on their execution time 007 011 050 100 101 Two systems are used to determine their execution time. One is backed by an SATA spinning rust, whose maximum R/W speed is about 100MB/s, modern desktop performance. (VM1) Another one is a VM inside a openstack

Re: [PATCH] fstests: btrfs: Test send on heavily deduped file

At 07/21/2016 07:37 AM, Dave Chinner wrote: On Wed, Jul 20, 2016 at 03:40:29PM +0800, Qu Wenruo wrote: At 07/20/2016 03:01 PM, Eryu Guan wrote: On Tue, Jul 19, 2016 at 01:42:03PM +0800, Qu Wenruo wrote: This test uses $LOAD_FACTOR, so it should be in 'stress' group. And it hangs the latest

Re: [PATCH 0/4] update bytes_may_use timely to avoid false ENOSPC issue

hello, On 07/20/2016 04:46 PM, Holger Hoffstätte wrote: On 07/20/16 07:56, Wang Xiaoguang wrote: Currently in btrfs, for data space reservation, it does not update bytes_may_use in btrfs_update_reserved_bytes() and the decrease operation will be delayed to be done in

Re: [PATCH 1/4] btrfs: use correct offset for reloc_inode in prealloc_file_extent_cluster()

hello, On 07/20/2016 09:18 PM, Josef Bacik wrote: On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: In prealloc_file_extent_cluster(), btrfs_check_data_free_space() uses wrong file offset for reloc_inode, it uses cluster->start and cluster->end, which indeed are extent's bytenr. The correct

Re: [PATCH 4/4] btrfs: update btrfs_space_info's bytes_may_use timely

hello, On 07/20/2016 09:35 PM, Josef Bacik wrote: On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: This patch can fix some false ENOSPC errors, below test script can reproduce one false ENOSPC error: #!/bin/bash dd if=/dev/zero of=fs.img bs=$((1024*1024)) count=128 dev=$(losetup

Re: [PATCH 3/4] btrfs: introduce new EXTENT_CLEAR_DATA_RESV flag

hello, On 07/20/2016 09:22 PM, Josef Bacik wrote: On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: In next patch, btrfs_clear_bit_hook() will not call btrfs_free_reserved_data_space_noquota() to update btrfs_space_info's bytes_may_use unless it has EXTENT_DO_ACCOUNTING or EXTENT_CLEAR_DATA_RESV,

[PATCH v2] Btrfs: fix memory leak of block group cache

While processing delayed refs, we may update block group's statistics and attach it to cur_trans->dirty_bgs, and later writing dirty block groups will process the list, which happens during btrfs_commit_transaction(). For whatever reason, the transaction is aborted and dirty_bgs is not processed

[PATCH] Btrfs: fix memory leak of block group cache

While processing delayed refs, we may update block group's statistics and attach it to cur_trans->dirty_bgs, and later writing dirty block groups will process the list, which happens during btrfs_commit_transaction(). For whatever reason, the transaction is aborted and dirty_bgs is not processed

[PATCH] Btrfs: add ASSERT for block group's memory leak

This adds several ASSERT()' s to report memory leak of block group cache. Signed-off-by: Liu Bo --- fs/btrfs/extent-tree.c | 5 + 1 file changed, 5 insertions(+) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index 82b912a..50bd683 100644 ---

Re: [PATCH] fstests: btrfs: Test send on heavily deduped file

On Wed, Jul 20, 2016 at 03:40:29PM +0800, Qu Wenruo wrote: > At 07/20/2016 03:01 PM, Eryu Guan wrote: > >On Tue, Jul 19, 2016 at 01:42:03PM +0800, Qu Wenruo wrote: > >>> > >>>This test uses $LOAD_FACTOR, so it should be in 'stress' group. And it > >>>hangs the latest kernel, stop other tests from

Re: [PATCH] fstests: btrfs: Test send on heavily deduped file

On Wed, Jul 20, 2016 at 03:01:00PM +0800, Eryu Guan wrote: > For running tests, "./check -g auto -x dangerous" might fit your need. Yes, that's precisely the way the dangerous group is intended to be used: as a exclusion filter that gets applied to other test group definitions. Cheers, Dave. --

Re: Data recovery from a linear multi-disk btrfs file system

Am Thu, 21 Jul 2016 00:19:41 +0200 schrieb Kai Krakow : > Am Fri, 15 Jul 2016 20:45:32 +0200 > schrieb Matt : > > > > On 15 Jul 2016, at 14:10, Austin S. Hemmelgarn > > > wrote: > > > > > > On 2016-07-15 05:51, Matt wrote: >

Re: Data recovery from a linear multi-disk btrfs file system

Am Fri, 15 Jul 2016 20:45:32 +0200 schrieb Matt : > > On 15 Jul 2016, at 14:10, Austin S. Hemmelgarn > > wrote: > > > > On 2016-07-15 05:51, Matt wrote: > >> Hello > >> > >> I glued together 6 disks in linear lvm fashion (no RAID) to obtain > >> one

Re: Data recovery from a linear multi-disk btrfs file system

On Fri, Jul 15, 2016 at 12:52 PM, Austin S. Hemmelgarn wrote: > Your own 'btrfs fi df' output clearly says that more than 99% of your data > chunks are in a RAID0 profile, hence my statement. Somewhen in ancient Btrfs list history, there was a call to change the mkfs

Re: Status of SMR with BTRFS

On Sun, Jul 17, 2016 at 3:08 AM, Hendrik Friedel wrote: > Well, btrfs does write data very different to many other file systems. On > every write the file is copied to another place, even if just one bit is > changed. That's special and I am wondering whether that could

Re: [PATCH] btrfs: test whether fallocate(2) can preallocate half of the whole fs space

On Wed, Jul 20, 2016 at 03:34:50PM +0800, Wang Xiaoguang wrote: > Currently in btrfs, there is something wrong with data space reservation. > For example, if we try to preallocate more than haf of whole fs space, > ENOSPC will occur, but indeed fs still has free space to satisfy this > request. >

Re: A lot warnings in dmesg while running thunderbird

On 20.07.2016 15:50, Chris Mason wrote: > > > On 07/19/2016 08:11 PM, Gabriel C wrote: >> >> >> On 19.07.2016 13:05, Chris Mason wrote: >>> On Mon, Jul 11, 2016 at 11:28:01AM +0530, Chandan Rajendra wrote: Hi Chris, I am able to reproduce the issue with the 'short-write'

Re: A lot warnings in dmesg while running thunderbird

On 07/19/2016 08:11 PM, Gabriel C wrote: On 19.07.2016 13:05, Chris Mason wrote: On Mon, Jul 11, 2016 at 11:28:01AM +0530, Chandan Rajendra wrote: Hi Chris, I am able to reproduce the issue with the 'short-write' program. But before the call trace associated with btrfs_destroy_inode(), I

Re: [PATCH 4/4] btrfs: update btrfs_space_info's bytes_may_use timely

On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: This patch can fix some false ENOSPC errors, below test script can reproduce one false ENOSPC error: #!/bin/bash dd if=/dev/zero of=fs.img bs=$((1024*1024)) count=128 dev=$(losetup --show -f fs.img) mkfs.btrfs -f -M

Re: [PATCH 3/4] btrfs: introduce new EXTENT_CLEAR_DATA_RESV flag

On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: In next patch, btrfs_clear_bit_hook() will not call btrfs_free_reserved_data_space_noquota() to update btrfs_space_info's bytes_may_use unless it has EXTENT_DO_ACCOUNTING or EXTENT_CLEAR_DATA_RESV, as for the reason, please see the next patch for

Re: [PATCH 2/4] btrfs: divide btrfs_update_reserved_bytes() into two functions

On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: This patch divides btrfs_update_reserved_bytes() into btrfs_add_reserved_bytes() and btrfs_free_reserved_bytes(), and next patch will extend btrfs_add_reserved_bytes（）to fix some false ENOSPC error, please see later patch for detailed info.

Re: [PATCH 1/4] btrfs: use correct offset for reloc_inode in prealloc_file_extent_cluster()

On 07/20/2016 01:56 AM, Wang Xiaoguang wrote: In prealloc_file_extent_cluster(), btrfs_check_data_free_space() uses wrong file offset for reloc_inode, it uses cluster->start and cluster->end, which indeed are extent's bytenr. The correct value should be cluster->[start|end] minus block group's

Re: [PATCH] btrfs: test whether fallocate(2) can preallocate half of the whole fs space

On Wed, Jul 20, 2016 at 8:34 AM, Wang Xiaoguang wrote: > Currently in btrfs, there is something wrong with data space reservation. > For example, if we try to preallocate more than haf of whole fs space, > ENOSPC will occur, but indeed fs still has free space to

Re: [PATCH 0/4] update bytes_may_use timely to avoid false ENOSPC issue

On 07/20/16 07:56, Wang Xiaoguang wrote: > Currently in btrfs, for data space reservation, it does not update > bytes_may_use in btrfs_update_reserved_bytes() and the decrease operation > will be delayed to be done in extent_clear_unlock_delalloc(), for > fallocate(2), decrease operation is even

Re: [PATCH] fstests: btrfs: Test send on heavily deduped file

At 07/20/2016 03:01 PM, Eryu Guan wrote: On Tue, Jul 19, 2016 at 01:42:03PM +0800, Qu Wenruo wrote: This test uses $LOAD_FACTOR, so it should be in 'stress' group. And it hangs the latest kernel, stop other tests from running, I think we can add it to 'dangerous' group as well. Thanks for

[PATCH] btrfs: test whether fallocate(2) can preallocate half of the whole fs space

Currently in btrfs, there is something wrong with data space reservation. For example, if we try to preallocate more than haf of whole fs space, ENOSPC will occur, but indeed fs still has free space to satisfy this request. To easily reproduce this bug, this test case needs fs is mixed mode(btrfs

Re: ENOSPC / no space on very large devices

On 07/20/16 07:31, Stefan Priebe - Profihost AG wrote: > Hi list, > > while i didn't had the problem for some month i'm now getting ENOSPC on > a regular basis on one host. Well, it's getting better. :) > if i umount the volume i get traces (i already did a clear_cache 4 days > ago to

[PATCH] btrfs: backref: Fix soft lockup in __merge_refs function

When over 1000 file extents refers to one extent, find_parent_nodes() will be obviously slow, due to the O(n^2)~O(n^3) loops inside __merge_refs(). The following ftrace shows the cubic growth of execution time: 256 refs 5) + 91.768 us | __add_keyed_refs.isra.12 [btrfs](); 5) 1.447 us|

Re: [PATCH] fstests: btrfs: Test send on heavily deduped file

On Tue, Jul 19, 2016 at 01:42:03PM +0800, Qu Wenruo wrote: > > > > This test uses $LOAD_FACTOR, so it should be in 'stress' group. And it > > hangs the latest kernel, stop other tests from running, I think we can > > add it to 'dangerous' group as well. > > > > Thanks for this info. > I'm

Re: ENOSPC / no space on very large devices

hello, On 07/20/2016 01:31 PM, Stefan Priebe - Profihost AG wrote: Hi list, while i didn't had the problem for some month i'm now getting ENOSPC on a regular basis on one host. It would be great if someone can help me debugging this. Some basic informations: # touch /vmbackup/abc touch: