Re: Wrong RAID 5/6 warning when converting single -> dup

2021-03-22 Thread Holger Hoffstätte
On 2021-03-22 21:00, Holger Hoffstätte wrote: So I just did a conversion from single to dup and got the following unexpected warning: $btrfs balance start -mconvert=dup /mnt/data WARNING: RAID5/6 support has known problems and is strongly discouraged to be used besides testing or

Wrong RAID 5/6 warning when converting single -> dup

2021-03-22 Thread Holger Hoffstätte
So I just did a conversion from single to dup and got the following unexpected warning: $btrfs balance start -mconvert=dup /mnt/data WARNING: RAID5/6 support has known problems and is strongly discouraged to be used besides testing or evaluation. It is recommended that

Re: error in backport of 'btrfs: fix possible free space tree corruption with online conversion'

2021-02-19 Thread Holger Hoffstätte
On 2021-02-19 16:20, Wang Yugui wrote: Hi, On 2021-02-19 04:17, Wang Yugui wrote: Hi, Josef Bacik We noticed an error in 5.10.x backport of 'btrfs: fix possible free space tree corruption with online conversion' It is wrong in 5.10.13, but right in 5.11. 5.10.13 @@ -146,6 +146,9 @@ enum {

Re: error in backport of 'btrfs: fix possible free space tree corruption with online conversion'

2021-02-19 Thread Holger Hoffstätte
On 2021-02-19 04:17, Wang Yugui wrote: Hi, Josef Bacik We noticed an error in 5.10.x backport of 'btrfs: fix possible free space tree corruption with online conversion' It is wrong in 5.10.13, but right in 5.11. 5.10.13 @@ -146,6 +146,9 @@ enum { BTRFS_FS_STATE_DEV_REPLACING, /

Re: [BUG] 500-2000% performance regression w/ 5.10

2020-12-23 Thread Holger Hoffstätte
On 2020-12-23 20:41, Josef Bacik wrote: Could you give this a try? I'm not able to reproduce the problem, but I'm Since I wanted to rule out NVME/block layer/scheduler etc. I tried and could reproduce it immediately, see below. Didn't notice it earlier since most of btrfs is read-mostly.. :(

Re: Curious problem: btrfs device stats & unpriviliged access

2019-10-23 Thread Holger Hoffstätte
It's been a while.. :D On 10/8/18 10:02 PM, Hans van Kranenburg wrote: On 10/08/2018 06:37 PM, Holger Hoffstätte wrote: On 10/08/18 17:46, Hans van Kranenburg wrote: fs.devices() also looks for dev_items in the chunk tree: https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctr

Evaluating File System Reliability on Solid State Drives

2019-10-17 Thread Holger Hoffstätte
Since resiliency and recovery from corruption continues to be a hot topic, I figured I'd mention a paper/talk/slides from the recent Usenix ATC'19 conference: Evaluating File System Reliability on Solid State Drives https://www.usenix.org/conference/atc19/presentation/jaffer Maybe it provides

Re: [PATCH 2/2] Btrfs: keep pages dirty when using btrfs_writepage_fixup_worker

2019-10-09 Thread Holger Hoffstätte
On 6/20/18 4:56 PM, Chris Mason wrote: For COW, btrfs expects pages dirty pages to have been through a few setup steps. This includes reserving space for the new block allocations and marking the range in the state tree for delayed allocation. A few places outside btrfs will dirty pages directl

Re: Massive filesystem corruption since kernel 5.2 (ARCH)

2019-09-12 Thread Holger Hoffstätte
On 9/12/19 10:24 AM, James Harvey wrote: On Thu, Sep 12, 2019 at 3:51 AM Filipe Manana wrote: ... Until the fix gets merged to 5.2 kernels (and 5.3), I don't really recommend running 5.2 or 5.3. What is your recommendation for distributions that have been shipping 5.2.x for quite some time,

Re: Balance conversion to metadata RAID1, data RAID1 leaves some metadata as DUP

2019-09-08 Thread Holger Hoffstätte
On 9/8/19 11:18 AM, Pete wrote: On 9/8/19 8:57 AM, Holger Hoffstätte wrote: On 9/8/19 9:09 AM, Pete wrote: (snip) I presume running another balance will fix this, but surely all metadata should have been converted?  Is there a way to only balance the DUP metadata? Adding "soft" to

Re: Balance conversion to metadata RAID1, data RAID1 leaves some metadata as DUP

2019-09-08 Thread Holger Hoffstätte
On 9/8/19 9:09 AM, Pete wrote: (snip) I presume running another balance will fix this, but surely all metadata should have been converted? Is there a way to only balance the DUP metadata? Adding "soft" to -mconvert should do exactly that; it will then skip any chunks that are already in the ta

Re: docbook45 is gone

2019-09-03 Thread Holger Hoffstätte
On 9/3/19 10:17 AM, Adam Borowski wrote: Hi! I'm afraid that asciidoctor 2.0 dropped support for docbook45. The explanation given is here: https://github.com/asciidoctor/asciidoctor/issues/3005 This makes btrfs-progs fail to build unless docs are off, with: asciidoctor: FAILED: missing converte

Re: [PATCH v2 0/4] Support xxhash64 checksums

2019-08-22 Thread Holger Hoffstätte
On 8/22/19 1:40 PM, Johannes Thumshirn wrote: Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is prepared for an easy addition of new checksums, this patchset implements XXHASH64 as a second, fast but not cryptographically secure checksum hash. Question from the chea

Re: [PATCH][v2] btrfs: add an ioctl to force chunk allocation

2019-08-05 Thread Holger Hoffstätte
On 8/5/19 5:41 PM, David Sterba wrote: On Mon, Aug 05, 2019 at 04:10:39PM +0200, Holger Hoffstätte wrote: On 8/5/19 3:19 PM, Josef Bacik wrote: In testing block group removal it's sometimes handy to be able to create block groups on demand. Add an ioctl to allow us to force allocation

Re: [PATCH][v2] btrfs: add an ioctl to force chunk allocation

2019-08-05 Thread Holger Hoffstätte
his that btrfs_chunk_alloc doesn't figure out the rest of the flags needed for the type. Use btrfs_force_chunk_alloc instead so that we get the raid settings for the alloc type we're using. Tested-by: Holger Hoffstätte Now works as intended - very nice, thanks! -h

Re: [PATCH] btrfs: add an ioctl to force chunk allocation

2019-08-05 Thread Holger Hoffstätte
On 8/5/19 12:31 PM, Hans van Kranenburg wrote: On 8/5/19 12:20 PM, Holger Hoffstätte wrote: On 8/2/19 6:10 PM, Josef Bacik wrote: In testing block group removal it's sometimes handy to be able to create block groups on demand. Add an ioctl to allow us to force allocation from user

Re: [PATCH] btrfs: add an ioctl to force chunk allocation

2019-08-05 Thread Holger Hoffstätte
On 8/2/19 6:10 PM, Josef Bacik wrote: In testing block group removal it's sometimes handy to be able to create block groups on demand. Add an ioctl to allow us to force allocation from userspace. Gave this a try and it works as advertised, though I noticed that the redundancy level is ignored,

Re: Btrfs progs release 5.2.1

2019-07-30 Thread Holger Hoffstätte
On 7/30/19 1:17 PM, Baruch Siach wrote: Hi David, On Fri, Jul 26 2019, David Sterba wrote: btrfs-progs version 5.2.1 have been released. This is a bugfix release. Changes: * scrub status: fix ETA calculation after resume * check: fix crash when using -Q * restore: fix symlink owner re

Re: 5.2rc5 corruption, many "is compressed but inode flag doesn't allow"

2019-06-24 Thread Holger Hoffstätte
On 6/25/19 12:46 AM, Chris Murphy wrote: Same call trace on ro mount and ro scrub with 5.2.0-rc2+, but also an additional call trace related to zstd. As this is a zstd compressed file system, it might be related. [ 366.319583] [ 366.325036] WARNING: inconsisten

Re: updating btrfs-debugfs to python3?

2019-06-19 Thread Holger Hoffstätte
On 6/19/19 5:20 AM, Chris Murphy wrote: This is on Fedora Rawhide (which is planned to become Fedora 31 due in October), which no longer provides python2. $ ./btrfs-debugfs -b / /usr/bin/env: ‘python2’: No such file or directory $ I expect other distros are going to follow as Python 2.7 EOL is

Re: 5.1.3: unable to handle kernel NULL pointer dereference in btrfs_reloc_pre_snapshot

2019-05-18 Thread Holger Hoffstätte
On 5/19/19 1:03 AM, Zygo Blaxell wrote: There are several new things in 5.1. First a lockdep warning: [ 146.037574][C3] hrtimer: interrupt took 2002067 ns [ 365.442210][C2] [ 365.442616][C2] [ 365.443477][C2] W

Re: [PATCH] btrfs: Fix error handling in btrfs_cleanup_ordered_extents

2018-10-26 Thread Holger Hoffstätte
On 10/26/18 13:13, Nikolay Borisov wrote: + if (page_start >= offset && page_end <= (offset + bytes - 1) { fs/btrfs/inode.c: In function 'btrfs_cleanup_ordered_extents': fs/btrfs/inode.c:140:62: error: expected ')' before '{' token if (page_start >= offset && page_end <= (offset + byte

Re: Scrub aborts due to corrupt leaf

2018-10-10 Thread Holger Hoffstätte
On 10/10/18 19:44, Chris Murphy wrote: On Wed, Oct 10, 2018 at 10:04 AM, Holger Hoffstätte wrote: On 10/10/18 17:44, Larkin Lowrey wrote: (..) About once a week, or so, I'm running into the above situation where FS seems to deadlock. All IO to the FS blocks, there is no IO activity at a

Re: Scrub aborts due to corrupt leaf

2018-10-10 Thread Holger Hoffstätte
On 10/10/18 19:25, Larkin Lowrey wrote: On 10/10/2018 12:04 PM, Holger Hoffstätte wrote: On 10/10/18 17:44, Larkin Lowrey wrote: (..) About once a week, or so, I'm running into the above situation where FS seems to deadlock. All IO to the FS blocks, there is no IO activity at all. I ha

Re: Scrub aborts due to corrupt leaf

2018-10-10 Thread Holger Hoffstätte
On 10/10/18 17:44, Larkin Lowrey wrote: (..) About once a week, or so, I'm running into the above situation where FS seems to deadlock. All IO to the FS blocks, there is no IO activity at all. I have to hard reboot the system to recover. There are no error indications except for the following whi

Re: Curious problem: btrfs device stats & unpriviliged access

2018-10-08 Thread Holger Hoffstätte
On 10/08/18 17:46, Hans van Kranenburg wrote: fs.devices() also looks for dev_items in the chunk tree: https://github.com/knorrie/python-btrfs/blob/master/btrfs/ctree.py#L481 So, BOOM! you need root. Or just start a 0, ignore errors and start trying all devids until you found num_devices amou

Re: Curious problem: btrfs device stats & unpriviliged access

2018-10-08 Thread Holger Hoffstätte
On 10/08/18 16:40, Hans van Kranenburg wrote: Looking at the kernel side of things in fs/btrfs/ioctl.c I see both BTRFS_IOC_TREE_SEARCH[_V2} unconditionally require CAP_SYS_ADMIN. That's the tree search ioctl, for reading arbitrary metadata. The device stats ioctl is IOC_GET_DEV_STATS... Yea

Curious problem: btrfs device stats & unpriviliged access

2018-10-08 Thread Holger Hoffstätte
(moving the discussion here from GH [1]) Apparently there is something weird going on with the device stats ioctls. I cannot get them to work as regular user, while they work for David. A friend confirms the same issue on his system - no access as non-root. So I made a new empty fs, mounted it,

Monitoring btrfs with Prometheus (and soon OpenMonitoring)

2018-10-07 Thread Holger Hoffstätte
The Prometheus statistics collection/aggregation/monitoring/alerting system [1] is quite popular, easy to use and will probably be the basis for the upcoming OpenMetrics "standard" [2]. Prometheus collects metrics by polling host-local "exporters" that respond to http requests; many such export

Re: [PATCH 0/5] rb_first to rb_first_cached conversion

2018-08-23 Thread Holger Hoffstätte
with no bad side effects so far. So: Tested-by: Holger Hoffstätte cheers Holger

Re: [PATCH 0/2] address lock contention of btree root

2018-08-21 Thread Holger Hoffstätte
On 08/21/18 20:15, Liu Bo wrote: I just realize that patch 2 can result in softlockup as btrfs_search_slot() may return a path with all nodes being in spinning lock, and if the callers want to sleep, we're in trouble. I've removed patch 2 and am re-running the test (xfstests, fsmark and dbench).

Re: Regression with crc32c selection?

2018-07-23 Thread Holger Hoffstätte
On 07/23/18 18:50, David Sterba wrote: On Mon, Jul 23, 2018 at 04:13:26PM +0200, Holger Hoffstätte wrote: While backporting a bunch of fixes to my own 4.16.x tree (4.17 had a few too many bugs for my taste) I also ended up merging: Curious, bugs in btrfs or the whole 4.17 kernel? And if bugs

Re: Regression with crc32c selection? (solved - pilot error)

2018-07-23 Thread Holger Hoffstätte
On 07/23/18 16:39, Patrik Lundquist wrote: $ uname -a Linux nas 4.17.0-1-amd64 #1 SMP Debian 4.17.8-1 (2018-07-20) x86_64 GNU/Linux $ dmesg | grep Btrfs [8.168408] Btrfs loaded, crc32c=crc32c-intel $ lsmod | grep crc32 crc32_pclmul 16384 0 libcrc32c 16384 1 btrfs cr

python-btrfs & btrfs-heatmap ebuilds now available for Gentoo

2018-07-23 Thread Holger Hoffstätte
I wanted to migrate my collection of questionable shell scripts for btrfs maintenance/inspection to a more stable foundation and therefore created Gentoo ebuilds for Hans van Kranenburg's excellent python-btrfs [1] and btrfs-heatmap [2] packages. They can be found in my overlay at: https://git

Regression with crc32c selection?

2018-07-23 Thread Holger Hoffstätte
Hi, While backporting a bunch of fixes to my own 4.16.x tree (4.17 had a few too many bugs for my taste) I also ended up merging: df91f56adce1f: libcrc32c: Add crc32c_impl function 9678c54388b6a: btrfs: Remove custom crc32c init code ..which AFAIK went into 4.17 and seemed harmless enough; afte

Re: Issue on BTRFS/copy of really huge files

2018-07-06 Thread Holger Hoffstätte
On 07/06/18 10:37, Juergen Sauer wrote: .. Moving a virtual machine from ssd/raid1 subvolume (nocow) into the rotational big store (noocow) fails. After filling up the cachememory (ram) the data flow cuts down to zero 0 kb/sec. In fatal result the copy of an huge file hangs does not proceed any m

Re: BTRFS losing SE Linux labels on power failure or "reboot -nffd"

2018-06-04 Thread Holger Hoffstätte
On 06/04/18 15:29, Hans van Kranenburg wrote: Hi, On 06/04/2018 03:14 PM, Russell Coker wrote: The command "reboot -nffd" (kernel reboot without flushing kernel buffers or writing status) when run on a BTRFS system with SE Linux will often result in /var/log/audit/audit.log being unlabeled. T

Re: [RFC] btrfs: Speedup btrfs_read_block_groups()

2018-02-22 Thread Holger Hoffstätte
On 02/22/18 05:52, Qu Wenruo wrote: > btrfs_read_block_groups() is used to build up the block group cache for > all block groups, so it will iterate all block group items in extent > tree. > > For large filesystem (TB level), it will search for BLOCK_GROUP_ITEM > thousands times, which is the most

Re: [RFC] btrfs: Speedup btrfs_read_block_groups()

2018-02-22 Thread Holger Hoffstätte
On 02/22/18 05:52, Qu Wenruo wrote: > btrfs_read_block_groups() is used to build up the block group cache for > all block groups, so it will iterate all block group items in extent > tree. > > For large filesystem (TB level), it will search for BLOCK_GROUP_ITEM > thousands times, which is the most

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Holger Hoffstätte
On 02/13/18 13:54, Qu Wenruo wrote: > On 2018年02月13日 20:26, Holger Hoffstätte wrote: >> On 02/13/18 12:40, Qu Wenruo wrote: >>>>> The problem is not about how much space it takes, but how many extents >>>>> are here in the filesystem. >> >> I ha

Re: mount btrfs takes 30 minutes, btrfs check runs out of memory

2018-02-13 Thread Holger Hoffstätte
On 02/13/18 12:40, Qu Wenruo wrote: >>> The problem is not about how much space it takes, but how many extents >>> are here in the filesystem. I have no idea why btrfs' mount even needs to touch all block groups to get going (which seems to be the root of the problem), but here's a not so crazy id

Re: Can rsync use reflink to synchronize directories?

2018-01-17 Thread Holger Hoffstätte
On 01/17/18 18:35, MegaBrutal wrote: > Does rsync support copying files with reflink on local btrfs file > system? Of course it could only work if the necessary conditions for > reflinking are met, but it would be very useful. Not yet: https://bugzilla.samba.org/show_bug.cgi?id=10170 -h -- To uns

Re: [PATCH 2/8] blk-mq: protect completion path with RCU

2018-01-08 Thread Holger Hoffstätte
On 01/09/18 00:27, Holger Hoffstätte wrote: > On 01/08/18 23:55, Jens Axboe wrote: >> the good old >> >> int srcu_idx = srcu_idx; >> >> should get the job done. > > (Narrator: It didn't.) Narrator: we retract our previous statement and apologi

Re: [PATCH 2/8] blk-mq: protect completion path with RCU

2018-01-08 Thread Holger Hoffstätte
On 01/08/18 23:55, Jens Axboe wrote: > On 1/8/18 1:15 PM, Jens Axboe wrote: >> On 1/8/18 12:57 PM, Holger Hoffstätte wrote: >>> On 01/08/18 20:15, Tejun Heo wrote: >>>> Currently, blk-mq protects only the issue path with RCU. This patch >>>> puts the com

Re: [PATCH 2/8] blk-mq: protect completion path with RCU

2018-01-08 Thread Holger Hoffstätte
On 01/08/18 20:15, Tejun Heo wrote: > Currently, blk-mq protects only the issue path with RCU. This patch > puts the completion path under the same RCU protection. This will be > used to synchronize issue/completion against timeout by later patches, > which will also add the comments. > > Signed

Re: WARNING: CPU: 1 PID: 3016 at fs/btrfs/ctree.h:1564 btrfs_update_device+0x189/0x190 [btrfs]

2017-12-29 Thread Holger Hoffstätte
Apply the patch from https://patchwork.kernel.org/patch/9960893/ and follow the logged instructions re. device resizing (or see https://bugzilla.kernel.org/show_bug.cgi?id=196949 for examples). The patch is unfortunately not yet merged into 4.15rc, otherwise it could be sent to 4.14-stable. -h

Re: kernel hangs during balance

2017-12-20 Thread Holger Hoffstätte
On 12/20/17 20:02, Chris Murphy wrote: > I don't know if it's the sending MUA or the list server, but the line > wrapping makes this much harder to follow. I suggest putting it in a > text file and attaching the text file. It's definitely not on the > receiving side, I see it here also: > https://w

Re: btrfs-transacti hammering the system

2017-12-01 Thread Holger Hoffstätte
On 12/01/17 18:34, Matt McKinnon wrote: > Thanks, I'll give space_cache=v2 a shot. Yes, very much recommended. > My mount options are: rw,relatime,space_cache,autodefrag,subvolid=5,subvol=/ Turn autodefrag off and use noatime instead of relatime. Your filesystem also seems very full, that's bad

Re: [PATCH v2.1 2/2] btrfs: Ensure btrfs_trim_fs can trim the whole fs

2017-11-29 Thread Holger Hoffstätte
On Tue, 28 Nov 2017 15:20:39 +0800, Qu Wenruo wrote: > [BUG] > fstrim on some btrfs only trims the unallocated space, not trimming any > space in existing block groups. > > [CAUSE] > Before fstrim_range passed to btrfs_trim_fs(), it get truncated to > range [0, super->total_bytes). > So later btr

Re: Kernel 4.14 RAID5 multi disk array on bcache not mounting

2017-11-22 Thread Holger Hoffstätte
On 11/21/17 23:22, Lionel Bouton wrote: > Le 21/11/2017 à 23:04, Andy Leadbetter a écrit : >> I have a 4 disk array on top of 120GB bcache setup, arranged as follows > [...] >> Upgraded today to 4.14.1 from their PPA and the > > 4.14 and 4.14.1 have a nasty bug affecting bcache users. See for exam

Re: 4.13.12: kernel BUG at fs/btrfs/ctree.h:1802!

2017-11-17 Thread Holger Hoffstätte
On 11/17/17 18:48, Marc MERLIN wrote: > On Thu, Nov 16, 2017 at 09:53:15PM -0800, Marc MERLIN wrote: >>> I suggest that you try lvmcache instead. It's much more flexible than >>> bcache, >>> does pretty much the same job, and has much less of the "hacky" feel to it. >> >> I can read up on it, it's

Re: 4.13.12: kernel BUG at fs/btrfs/ctree.h:1802!

2017-11-16 Thread Holger Hoffstätte
On 11/16/17 22:45, Marc MERLIN wrote: (snip) >> This BUG() was recently removed and seems to be caused by some kind >> of persistent corruption, which is seen as invalid inline extent. >> See [1], [2] for details. Maybe you can backport them? >> Alternatively just give 4.14 a whirl, it's great. >>

Re: 4.13.12: kernel BUG at fs/btrfs/ctree.h:1802!

2017-11-16 Thread Holger Hoffstätte
On 11/16/17 18:07, Marc MERLIN wrote: > Sorry, was missing the kernel number in the subject, just fixed that. > > On Thu, Nov 16, 2017 at 09:04:45AM -0800, Marc MERLIN wrote: >> My server now reboots every 20mn or so, with this. >> Sadly another BUG_ON() and it won't even tell me which filesystem

Re: [PATCH] btrfs-progs: add option to only list parent subvolumes

2017-09-30 Thread Holger Hoffstätte
On 09/30/17 19:56, Holger Hoffstätte wrote: > shell hackery as alternative. Anyway, I was sure that at the time the > other letters sounded even worse/were taken, but that may just have been > in my head. ;-) > > I just rechecked and -S is still available, so that's good.

Re: [PATCH] btrfs-progs: add option to only list parent subvolumes

2017-09-30 Thread Holger Hoffstätte
On 09/30/17 18:37, Graham Cobb wrote: > On 30/09/17 14:08, Holger Hoffstätte wrote: >> A "root" subvolume is identified by a null parent UUID, so adding a new >> subvolume filter and flag -P ("Parent") does the trick. > > I don't like the naming. T

[PATCH] btrfs-progs: right-align number columns in btrfs-debugfs output

2017-09-30 Thread Holger Hoffstätte
't be hardcoded but instead derived from the filesystem size, but this seems to work for now. Signed-off-by: Holger Hoffstätte --- btrfs-debugfs | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/btrfs-debugfs b/btrfs-debugfs index dfb88539..a7ecd16d 100755 --- a/btr

[PATCH] btrfs-progs: add option to only list parent subvolumes

2017-09-30 Thread Holger Hoffstätte
raged me to send this to the list, so here it is. I'm not too proud of the -P but couldn't find a better option letter; suggestions welcome. :) cheers, Holger Signed-off-by: Holger Hoffstätte --- btrfs-list.c | 6 ++ btrfs-list.h | 1 + cmds-subvolume.c | 8 +++- 3 files

Re: kernel BUG at fs/btrfs/extent_io.c:1989

2017-09-18 Thread Holger Hoffstätte
On 09/18/17 19:09, Liu Bo wrote: > This 'mirror 0' looks fishy, (as mirror comes from > btrfs_io_bio->mirror_num, which should be at least 1 if raid1 setup is > in use.) > > Not sure if 4.13.2-gentoo made any changes on btrfs, but can you No, it did not; Gentoo always strives to be as close to ma

Re: [PATCH] Btrfs: fix unexpected result when dio reading corrupted blocks

2017-09-18 Thread Holger Hoffstätte
Hello, quick question for backporting.. On 09/15/17 23:06, Liu Bo wrote: > commit 4246a0b63bd8 ("block: add a bi_error field to struct bio") > changed the logic of how dio read endio reports errors. [snip] I've tried to merge this into my 4.9.x++ tree but have a question since the DIO APIs chang

Re: Odd fallocate behavior on BTRFS.

2017-08-01 Thread Holger Hoffstätte
On 08/01/17 20:15, Holger Hoffstätte wrote: > On 08/01/17 19:34, Austin S. Hemmelgarn wrote: > [..] >> Apparently, if you call fallocate() on a file with an offset of 0 and >> a length longer than the length of the file itself, BTRFS will >> allocate that exact amount o

Re: Odd fallocate behavior on BTRFS.

2017-08-01 Thread Holger Hoffstätte
On 08/01/17 19:34, Austin S. Hemmelgarn wrote: [..] > Apparently, if you call fallocate() on a file with an offset of 0 and > a length longer than the length of the file itself, BTRFS will > allocate that exact amount of space, instead of just filling in holes > in the file and allocating space to

Re: [PATCH] btrfs-progs: add support to sort by topid

2017-07-13 Thread Holger Hoffstätte
On 07/13/17 15:25, Jeffrey Mahoney wrote: > On 7/13/17 12:16 AM, Anand Jain wrote: >> As users generally organize the subvols and snapshots based on the subvol >> directory hierarchy. So providing an ability to sort them by topid would >> help. Thanks. > > What is a topid? I needed to look at the

Re: [PATCH] btrfs: fix integer overflow in calc_reclaim_items_nr

2017-06-23 Thread Holger Hoffstätte
On 06/23/17 16:32, Chris Mason wrote: [..] > -static inline int calc_reclaim_items_nr(struct btrfs_fs_info *fs_info, > +static inline u64 calc_reclaim_items_nr(struct btrfs_fs_info *fs_info, > u64 to_reclaim) > { > u64 bytes; > - int nr; > + u64

Re: [PATCH v3] btrfs: fiemap: Cache and merge fiemap extent before submit it to user

2017-06-18 Thread Holger Hoffstätte
On 06/18/17 13:23, Qu Wenruo wrote: > Well, still no good news. > > I created file extents which returns 0x2008 and the last extent with 0x9. > But still failed to reproduce the error message. > > BTW, I noticed that your output is a little strange. > Normally we should have "offset=%llu phys=%ll

Re: [PATCH 0/7] Btrfs: fix total_bytes_pinned counter

2017-06-07 Thread Holger Hoffstätte
> Patches 5 and 6 are ugly, I'd love any suggestions for a cleaner fix. > Finally, patch 7 adds a warning to catch similar issues in the future. Since this didn't really change the code significantly from the previous version (except for the addition of minor patches 2+7) have

Re: [PATCH] Btrfs: skip commit transaction if we don't have enough pinned bytes

2017-06-03 Thread Holger Hoffstätte
On 06/02/17 20:14, Omar Sandoval wrote: > On Fri, May 19, 2017 at 11:39:15AM -0600, Liu Bo wrote: >> We commit transaction in order to reclaim space from pinned bytes because >> it could process delayed refs, and in may_commit_transaction(), we check >> first if pinned bytes are enough for the requ

Question re. required since btrfs-progs-4.10

2017-03-11 Thread Holger Hoffstätte
Hi, I'm on Gentoo and wanted to update Docker to 17.03.0, which failed when it couldn't build the btrfs driver due to a missing . This worked fine on another machine the other day, so I dug in and found that the only difference was an intermediate update to btrfs-progs 4.10. Sure enough: since 4.1

Re: duperemove : some real world figures on BTRFS deduplication

2016-12-09 Thread Holger Hoffstätte
On 12/09/16 16:43, Chris Murphy wrote: >> If compression has nothing to do with this, then this is heavy >> fragmentation. > > It's probably not that fragmented. Due to compression, metadata > describes 128KiB extents even though the data is actually contiguous. > > And it might be the same thing

Re: resend: Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread Holger Hoffstätte
On 11/23/16 18:21, Stefan Priebe - Profihost AG wrote: > Am 04.11.2016 um 20:20 schrieb Liu Bo: >> If we have >> >> |0--hole--4095||4096--preallocate--12287| >> >> instead of using preallocated space, a 8K direct write will just >> create a new 8K extent and it'll end up with >> >> |0--new extent--

Re: [PATCH] Btrfs: deal with existing encompassing extent map in btrfs_get_extent()

2016-11-10 Thread Holger Hoffstätte
On 11/10/16 17:20, Omar Sandoval wrote: > On Thu, Nov 10, 2016 at 05:01:44PM +0100, Holger Hoffstätte wrote: >> On 11/10/16 16:37, Omar Sandoval wrote: >>> On Thu, Nov 10, 2016 at 04:11:35PM +0100, Holger Hoffstätte wrote: >>>> On 11/10/16 00:26, Omar Sandoval wr

Re: [PATCH] Btrfs: deal with existing encompassing extent map in btrfs_get_extent()

2016-11-10 Thread Holger Hoffstätte
On 11/10/16 16:37, Omar Sandoval wrote: > On Thu, Nov 10, 2016 at 04:11:35PM +0100, Holger Hoffstätte wrote: >> On 11/10/16 00:26, Omar Sandoval wrote: >>> From: Omar Sandoval >>> >>> My QEMU VM was seeing inexplicable I/O errors that I tracked down to >

Re: [PATCH] Btrfs: deal with existing encompassing extent map in btrfs_get_extent()

2016-11-10 Thread Holger Hoffstätte
On 11/10/16 16:06, David Sterba wrote: > On Wed, Nov 09, 2016 at 03:26:50PM -0800, Omar Sandoval wrote: >> From: Omar Sandoval >> [snip] >> Commit 8dff9c853410 ("Btrfs: deal with duplciates during extent_map >> insertion in btrfs_get_extent") fixed a case in btrfs_get_extent() where >> two threads

Re: [PATCH] Btrfs: deal with existing encompassing extent map in btrfs_get_extent()

2016-11-10 Thread Holger Hoffstätte
On 11/10/16 00:26, Omar Sandoval wrote: > From: Omar Sandoval > > My QEMU VM was seeing inexplicable I/O errors that I tracked down to > errors coming from the qcow2 virtual drive in the host system. The qcow2 > file is a nocow file on my Btrfs drive, which QEMU opens with O_DIRECT. > Every once

Re: [PATCH v2] btrfs: make block group flags in balance printks human-readable

2016-11-08 Thread Holger Hoffstätte
On 11/07/16 22:40, Adam Borowski wrote: > They're not even documented anywhere, letting users with no recourse but > to RTFS. It's no big burden to output the bitfield as words. > > Also, display unknown flags as hex. > > Signed-off-by: Adam Borowski [..] > > /* > + * explain bit flags, pref

Re: [PATCH] btrfs: make block group flags in balance printks human-readable

2016-11-07 Thread Holger Hoffstätte
elpful and works (for me) as advertised. Tested-by: Holger Hoffstätte > --- > fs/btrfs/relocation.c | 34 -- > 1 file changed, 32 insertions(+), 2 deletions(-) > > diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c > index 0ec8ffa..388216f 1

Re: [PATCH] btrfs: imporve delayed refs iterations

2016-10-24 Thread Holger Hoffstätte
now with no negative side effects, mostly rsync creating & deleting files with various levels of reflinking (via snapshots). No problems at all. Also tried to manually create & delete a large file with heavy CoW and hundreds of reflinked copies - no problem either and pretty fast. So.. Tested-

Re: [PATCH] btrfs: pass correct args to btrfs_async_run_delayed_refs()

2016-10-18 Thread Holger Hoffstätte
trans->delayed_ref_updates * 2, 0); > + trans->delayed_ref_updates * 2, > + trans->transid, 0); > if (be_nice) { > if (truncate_space_check(trans, root, >

Re: [PATCH 2/2] btrfs: fix false enospc for compression

2016-10-14 Thread Holger Hoffstätte
his and patch #1 seem to work fine, and they don't seem to cause any regressions; ran a couple of both full and incremental rsync backups with >100GB on a new and now compressed subvolume without problem. Also Stefan just reported that his ENOSPC seems to be gone as well, so it seem

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-10-07 Thread Holger Hoffstätte
On 10/07/16 09:17, Wang Xiaoguang wrote: > Hi, > > On 10/07/2016 03:03 PM, Stefan Priebe - Profihost AG wrote: >> Dear Wang, >> >> can't use v4.8.0 as i always get OOMs and total machine crashes. >> >> Complete traces with your patch and some more btrfs patches applied (in >> the hope in fixes the

Re: 4.8rc8 & OOM panic

2016-10-03 Thread Holger Hoffstätte
On 09/28/16 21:45, Holger Hoffstätte wrote: > On 09/28/16 20:46, E V wrote: >> I just booted my backup box with 4.8rc8 and started an rsync onto >> btrfs and it panic'd with OOM a couple hours later. I thought the OOM >> problems from 4.7 we're supposed to be

Re: [PATCH v2 0/6] Btrfs: free space tree and sanity test fixes

2016-09-29 Thread Holger Hoffstätte
On 09/29/16 14:21, Anatoly Pugachev wrote: >> ... >> >> This is fixed by patch >> >> "Btrfs: remove unnecessary btrfs_mark_buffer_dirty in split_leaf" >> >> that's in the 4.9 queue. Other than that, the self-tests seem to pass, >> thanks for the test. Would be good if you can test with the mentio

Re: 4.8rc8 & OOM panic

2016-09-28 Thread Holger Hoffstätte
On 09/28/16 20:46, E V wrote: > I just booted my backup box with 4.8rc8 and started an rsync onto > btrfs and it panic'd with OOM a couple hours later. I thought the OOM > problems from 4.7 we're supposed to be fixed in 4.8, or did I get that > wrong? No users or anything else on the system. Keep

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-09-28 Thread Holger Hoffstätte
On 09/28/16 15:06, Stefan Priebe - Profihost AG wrote: > > Yes this is 4.4.22 and no i don't have qgroups enabled so it can't help. > > # btrfs qgroup show /path/ > ERROR: can't perform the search - No such file or directory > ERROR: can't list qgroups: No such file or director > > This is the s

Re: BTRFS: space_info 4 has 18446742286429913088 free, is not full

2016-09-28 Thread Holger Hoffstätte
On 09/28/16 13:35, Wang Xiaoguang wrote: > hello, > > On 09/28/2016 07:15 PM, Stefan Priebe - Profihost AG wrote: >> Dear list, >> >> is there any chance anybody wants to work with me on the following issue? > Though I'm also somewhat new to btrfs, but I'd like to. > >> >> BTRFS: space_info 4 has

Re: [PATCH v2 3/6] Btrfs: catch invalid free space trees

2016-09-23 Thread Holger Hoffstätte
; have the kernel rebuild the free space tree. > > Cc: sta...@vger.kernel.org # 4.5+ > Signed-off-by: Omar Sandoval Tested-by: Holger Hoffstätte Initial mount with this successfully detected the absent _VALID bit; fst was rebuilt, subsequent mounts are good without rebuild. \o/ Thanks! Holger

Re: [PATCH v2 2/6] Btrfs: fix mount -o clear_cache,space_cache=v2

2016-09-23 Thread Holger Hoffstätte
t `mount -o > clear_cache,space_cache=v2` would clear the free space tree and then > recreate it. > > Fixes: 511711af91f2 ("btrfs: don't run delayed references while we are > creating the free space tree") > Cc: sta...@vger.kernel.org # 4.5+ > Signed-off-by: Omar Sa

Re: [PATCH v2 1/6] Btrfs: fix free space tree bitmaps on big-endian systems

2016-09-23 Thread Holger Hoffstätte
quot;Btrfs: implement the free space B-tree") > Cc: sta...@vger.kernel.org # 4.5+ > Signed-off-by: Omar Sandoval Tested-by: Holger Hoffstätte (only for regressions on little-endian x86-64) Thanks! Holger -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs"

Re: unable to handle kernel paging request - btrfs

2016-09-22 Thread Holger Hoffstätte
On 09/22/16 14:18, Rich Freeman wrote: > I have been getting panics consistently after doing a btrfs replace > operation on a raid1 and rebooting. I linked a photo of the panic; I > haven't been able to get a text capture of it. > > https://ibin.co/2vx0HhDeViu3.jpg > > I'm getting this error on

Re: Status of free-space-tree feature

2016-09-21 Thread Holger Hoffstätte
On 09/21/16 11:24, David Sterba wrote: > Hi, > > as you might have noticed, the [1] wiki Status page lists the > free-space-tree as 'Unstable', referencing a problem with the bitmap > endianity. This will affect only bigendian systems. > > There's one more problem that I overlooked but was pointe

Re: [PATCH] Btrfs: handle quota reserve failure properly

2016-09-16 Thread Holger Hoffstätte
On Thu, 15 Sep 2016 14:57:48 -0400, Josef Bacik wrote: > btrfs/022 was spitting a warning for the case that we exceed the quota. If we > fail to make our quota reservation we need to clean up our data space > reservation. Thanks, > > Signed-off-by: Josef Bacik > --- > fs/btrfs/extent-tree.c |

Re: [PATCH] Btrfs: fix memory leak in do_walk_down

2016-09-14 Thread Holger Hoffstätte
On 09/14/16 04:02, Liu Bo wrote: > The extent buffer 'next' needs to be free'd conditionally. > > Signed-off-by: Liu Bo > --- > fs/btrfs/extent-tree.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c > index 5a940ab..779fd72 100644 > --- a

Commit 'synchronize incompat feature bits with sysfs files' still missing in for-next?

2016-09-13 Thread Holger Hoffstätte
I've noticed that the 4.5-rc commit 14e46e04: 'btrfs: synchronize incompat feature bits with sysfs files' [1] was reverted later in [2], but despite fixes to protect sysfs with locks & exorcise GFP_NOFS in favor of GFP_KERNEL it was never reinstated - neither for 4.5-final, nor later..and it's bee

Re: [PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-12 Thread Holger Hoffstätte
On 09/12/16 09:38, Wang Xiaoguang wrote: >> Actually even that is not true; both patches seem to be wrong in subtle >> ways. Naohiro's patch seems to prevent the deletion during balance, whereas >> yours prevents the cleaner from kicking in. > Indeed in my patch, I just change "struct mutex delete

Re: [PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-09 Thread Holger Hoffstätte
On 09/09/16 12:18, Holger Hoffstätte wrote: > On Fri, 09 Sep 2016 16:17:48 +0800, Wang Xiaoguang wrote: > >> cleaner_kthread() may run at any time, in which it'll call >> btrfs_delete_unused_bgs() >> to delete unused block groups. Because this work is asynchronous,

Re: [PATCH v3] btrfs: should block unused block groups deletion work when allocating data space

2016-09-09 Thread Holger Hoffstätte
On Fri, 09 Sep 2016 16:17:48 +0800, Wang Xiaoguang wrote: > cleaner_kthread() may run at any time, in which it'll call > btrfs_delete_unused_bgs() > to delete unused block groups. Because this work is asynchronous, it may also > result > in false ENOSPC error. With this v3 I can now no longer

Re: Finding only non-snapshots via btrfs subvol list

2016-09-08 Thread Holger Hoffstätte
On 07/21/16 16:55, Holger Hoffstätte wrote: > I'm trying to find non-snapshots, i.e. 'top-level' subvolumes in a > filesystem and this seems harder than it IMHO should be. > > The fs is just like: > > /mnt/stuff > subvolA > subvolA-date1 > subvolA-da

Please disable balance auto-resume for 4.9 (or even 4.8)

2016-08-25 Thread Holger Hoffstätte
Automatically resuming an interrupted balance has repeatedly caused all sorts of problems because it creates a possible failure mode when a user can least use it: after a crash/power loss/sudden reboot (which, like it or not, is the de facto "fix random problems" approach for many people). The id

Re: [PATCH] Btrfs: check btree node's nritems

2016-08-05 Thread Holger Hoffstätte
On 08/05/16 11:24, Holger Hoffstätte wrote: > On Wed, 03 Aug 2016 12:57:28 -0700, Liu Bo wrote: > >> When btree node (level = 1) has nritems which equals to zero, >> we can end up with panic due to insert_ptr()'s >> >> BUG_ON(slot > nritems); >

Re: [PATCH] Btrfs: check btree node's nritems

2016-08-05 Thread Holger Hoffstätte
On Wed, 03 Aug 2016 12:57:28 -0700, Liu Bo wrote: > When btree node (level = 1) has nritems which equals to zero, > we can end up with panic due to insert_ptr()'s > > BUG_ON(slot > nritems); > > where slot is 1 and nritems is 0, as copy_for_split() calls > insert_ptr(.., path->slots[1] + 1, ...)

Re: memory overflow or undeflow in free space tree / space_info?

2016-07-29 Thread Holger Hoffstätte
On Fri, 29 Jul 2016 22:57:36 +, Holger Hoffstätte wrote: > The only other patch I just found missing and which looks like it > could/should (I think?) work on top of the 4.4.x pagesize-based > calculations in file.c is: > > a2af23b7 "__btrfs_buffered_write: Pass va

  1   2   3   >