[PATCH 0/5] Mkfs: Rework --rootdir to a more generic behavior

2017-09-03 Thread Qu Wenruo
mkfs.btrfs --rootdir provides user a method to generate btrfs with pre-written content while without the need of root privilege. However the code is quite old and doesn't get much review or test. This makes some strange behavior, from customized chunk allocation (which uses the reserved 0~1M devic

[PATCH 2/5] btrfs-progs: mkfs: Rework rootdir option to avoid custom chunk layout

2017-09-03 Thread Qu Wenruo
mkfs.btrfs --rootdir uses its own custom chunk layout. This provides the possibility to limit the filesystem to a minimal size. However this custom chunk allocation has several problems. The most obvious problem is that it will allocate chunk from device offset 0. Both kernel and normal mkfs will

[PATCH 1/5] btrfs-progs: Fix one-byte overlap bug in free_block_group_cache

2017-09-03 Thread Qu Wenruo
free_block_group_cache() calls clear_extent_bits() with wrong end, which is one byte larger than the correct range. This will cause the next adjacent cache state be split. And due to the split, private pointer (which points to block group cache) will be reset to NULL. This is very hard to detect

[PATCH 5/5] btrfs-progs: Doc/mkfs: Add explanation for rootdir parameter

2017-09-03 Thread Qu Wenruo
Original --rootdir parameter will shrink the filesystem to its minimal size, which is quite confusing for some users. Add extra note for this behavior. Reported-by: Goffredo Baroncelli Signed-off-by: Qu Wenruo --- Documentation/mkfs.btrfs.asciidoc | 3 +++ 1 file changed, 3 insertions(+) diff

[PATCH 4/5] btrfs-progs: mkfs: Update allocation info before verbose output

2017-09-03 Thread Qu Wenruo
Since new --rootdir can allocate chunk, it will modify the chunk allocation result. This patch will update allocation info before verbose output to reflect such info. Signed-off-by: Qu Wenruo --- mkfs/main.c | 33 + 1 file changed, 33 insertions(+) diff --git a/

[PATCH 3/5] btrfs-progs: mkfs: Shrink the image for rootdir to minimal size

2017-09-03 Thread Qu Wenruo
Follow the original rootdir behavior to shrink the device size to minimal. The shrink itself is very simple, since dev extent is allocated on demand, we just need to shrink the device size to the device extent end position. Signed-off-by: Qu Wenruo --- mkfs/main.c | 107

read-only for no good reason on 4.9.30

2017-09-03 Thread Russell Coker
I have a system with less than 50% disk space used. It just started rejecting writes due to lack of disk space. I ran "btrfs balance" and then it started working correctly again. It seems that a btrfs filesystem if left alone will eventually get fragmented enough that it rejects writes (I've

[PATCH] btrfs-progs: inspect-internal rootid: Allow a file to be specified

2017-09-03 Thread Misono, Tomohiro
Since cmd_inspect_rootid() calls btrfs_open_dir(), it rejects a file to be spcified. But as the document says, a file should be supported. This patch introduces btrfs_open_file_or_dir(), which is a counterpart of btrfs_open_dir(), to safely check and open btrfs file or directory. The original btrf

Re: block group 11778977169408 has wrong amount of free space

2017-09-03 Thread Qu Wenruo
On 2017年09月04日 09:51, Christoph Anton Mitterer wrote: Did another mount with clear_cache,rw (cause it was ro before)... now I get even more errors: # btrfs check /dev/mapper/data-a2 ; echo $? Checking filesystem on /dev/mapper/data-a2 UUID: f8acb432-7604-46ba-b3ad-0abe8e92c4db checking extents

[PATCH 1/2] btrfs-progs: mkfs: Fix wrong file type for dir items and indexes when specifying root directory

2017-09-03 Thread Qu Wenruo
[Bug] If using mkfs.btrfs with "-r" parameter and specified direct has fifo/socket/char/block special file, then created btrfs can't pass fsck: -- checking fs roots unresolved ref dir 241158 index 3 namelen 9 name S.dirmngr filetype 0 errors 80, filetype mismatch ERROR: errors found i

[PATCH 2/2] btrfs-progs: test/mkfs: Add test case for rootdir parameter

2017-09-03 Thread Qu Wenruo
Add test case which checks if -r|--rootdir mount option can handle softlink/char/block/fifo files. Signed-off-by: Qu Wenruo --- .../009-special-files-for-rootdir/test.sh | 40 ++ 1 file changed, 40 insertions(+) create mode 100755 tests/mkfs-tests/009-special-files-

Re: How to disable/revoke 'compression'?

2017-09-03 Thread Qu Wenruo
On 2017年09月04日 08:14, Adam Borowski wrote: On Mon, Sep 04, 2017 at 07:55:27AM +0800, Qu Wenruo wrote: On 2017年09月04日 02:06, Adam Borowski wrote: I've once written a tool which does this, but 1. it's extremely slow, 2. insane, 3. so insane a certain member of this list would kill me had I dist

Re: block group 11778977169408 has wrong amount of free space

2017-09-03 Thread Christoph Anton Mitterer
Did another mount with clear_cache,rw (cause it was ro before)... now I get even more errors: # btrfs check /dev/mapper/data-a2 ; echo $? Checking filesystem on /dev/mapper/data-a2 UUID: f8acb432-7604-46ba-b3ad-0abe8e92c4db checking extents checking free space cache block group 9857516175360 has w

Re: block group 11778977169408 has wrong amount of free space

2017-09-03 Thread Christoph Anton Mitterer
Just checked, and mounting with clear_cache, and then re-fscking doesn't even fix the problem... Output stays the same. Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Josef Bacik
Ok this output looked fishy and so I went and tested it on my box again. It looks like I wasn't testing modifying a snapshot with an existing fs so I never saw these errors, but I see them as well. I definitely fucked the building of the initial ref tree. It's too late tonight for me to rewor

block group 11778977169408 has wrong amount of free space

2017-09-03 Thread Christoph Anton Mitterer
Hey. Just got the following: $ uname -a Linux heisenberg 4.12.0-1-amd64 #1 SMP Debian 4.12.6-1 (2017-08-12) x86_64 GNU/Linux $ btrfs version btrfs-progs v4.12 on a filesystem: # btrfs check /dev/mapper/data-a2 ; echo $? Checking filesystem on /dev/mapper/data-a2 UUID: f8acb432-7604-46ba-b3ad-0

Re: How to disable/revoke 'compression'?

2017-09-03 Thread Adam Borowski
On Mon, Sep 04, 2017 at 07:55:27AM +0800, Qu Wenruo wrote: > On 2017年09月04日 02:06, Adam Borowski wrote: > > I've once written a tool which does this, but 1. it's extremely slow, 2. > > insane, 3. so insane a certain member of this list would kill me had I > > distributed the tool. Thus, I'd need t

Re: How to disable/revoke 'compression'?

2017-09-03 Thread Qu Wenruo
On 2017年09月04日 02:06, Adam Borowski wrote: On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote: Hi, I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume. Is it enough

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Marc MERLIN
On Sun, Sep 03, 2017 at 05:33:33PM +, Josef Bacik wrote: > Alright pushed, sorry about that. I'm reasonably sure I'm running the new code, but still got this: [ 2104.336513] Dropping a ref for a root that doesn't have a ref on the block [ 2104.358226] Dumping block entry [115253923840 155648]

Re: speed up big btrfs volumes with ssds

2017-09-03 Thread Peter Grandi
> [ ... ] - needed volume size is 60TB I wonder how long that takes to 'scrub', 'balance', 'check', 'subvolume delete', 'find', etc. > [ ... ] 4x HW Raid 5 with 1GB controller memory of 4TB 3,5" > devices and using btrfs as raid 0 for data and metadata on top > of those 4 raid 5. [ ... ] the wri

Re: How to disable/revoke 'compression'?

2017-09-03 Thread Hans van Kranenburg
On 09/03/2017 08:06 PM, Adam Borowski wrote: > On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote: >> Hi, >> I used the mount option 'compression' on some mounted sub volumes. How >> can I revoke the compression? Means to delete the option and get all >> data uncompressed on this volume. >

speed up big btrfs volumes with ssds

2017-09-03 Thread Stefan Priebe - Profihost AG
Hello, i'm trying to speed up big btrfs volumes. Some facts: - Kernel will be 4.13-rc7 - needed volume size is 60TB Currently without any ssds i get the best speed with: - 4x HW Raid 5 with 1GB controller memory of 4TB 3,5" devices and using btrfs as raid 0 for data and metadata on top of those

Re: How to disable/revoke 'compression'?

2017-09-03 Thread Adam Borowski
On Sun, Sep 03, 2017 at 07:32:01PM +0200, Cloud Admin wrote: > Hi, > I used the mount option 'compression' on some mounted sub volumes. How > can I revoke the compression? Means to delete the option and get all > data uncompressed on this volume. > Is it enough to remount the sub volume without thi

How to disable/revoke 'compression'?

2017-09-03 Thread Cloud Admin
Hi, I used the mount option 'compression' on some mounted sub volumes. How can I revoke the compression? Means to delete the option and get all data uncompressed on this volume. Is it enough to remount the sub volume without this option? Or is it necessary to do some addional step (balancing?) to g

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Josef Bacik
Alright pushed, sorry about that. Josef Sent from my iPhone > On Sep 3, 2017, at 10:42 AM, Marc MERLIN wrote: > >> On Sun, Sep 03, 2017 at 02:38:57PM +, Josef Bacik wrote: >> Oh yeah you need CONFIG_STACKTRACE turned on, otherwise this is going to be >> difficult ;). Thanks, > > Right,

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Josef Bacik
Jesus Christ I misspelled it, I'll fix it up when I get home. Thanks, Josef Sent from my iPhone > On Sep 3, 2017, at 10:42 AM, Marc MERLIN wrote: > >> On Sun, Sep 03, 2017 at 02:38:57PM +, Josef Bacik wrote: >> Oh yeah you need CONFIG_STACKTRACE turned on, otherwise this is going to be >

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Marc MERLIN
On Sun, Sep 03, 2017 at 02:38:57PM +, Josef Bacik wrote: > Oh yeah you need CONFIG_STACKTRACE turned on, otherwise this is going to be > difficult ;). Thanks, Right, except that I thought I did: saruman:/usr/src/linux-btrfs/btrfs-next# grep STACKTRACE .config CONFIG_STACKTRACE_SUPPORT=y CO

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Josef Bacik
Oh yeah you need CONFIG_STACKTRACE turned on, otherwise this is going to be difficult ;). Thanks, Josef Sent from my iPhone > On Sep 3, 2017, at 10:31 AM, Marc MERLIN wrote: > >> On Sun, Sep 03, 2017 at 03:26:34AM +, Josef Bacik wrote: >> I was looking through the code for other ways to

Re: BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists (since 3.4 / 2012)

2017-09-03 Thread Marc MERLIN
On Sun, Sep 03, 2017 at 03:26:34AM +, Josef Bacik wrote: > I was looking through the code for other ways to cut down memory usage when I > noticed we only catch improper re-allocations, not adding another ref for > metadata which is what I suspect your problem is. I added another patch and

Re: joining to contribute

2017-09-03 Thread Piotr Pawłow
Hello, > >Alongside this, there's also a requirement for being able to do > round-trip send/receive while preserving the ability to do incremental > sends. This is likely to be related to the above bug-fix. I did a > complete write-up of what's happening, and what needs to happen, here: > > htt