Re: btrfs send extremely slow (almost stuck)

At 08/31/2016 09:35 AM, Jeff Mahoney wrote: On 8/28/16 10:12 PM, Qu Wenruo wrote: At 08/29/2016 10:11 AM, Qu Wenruo wrote: At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to

Re: btrfs send extremely slow (almost stuck)

On 8/28/16 10:12 PM, Qu Wenruo wrote: > > > At 08/29/2016 10:11 AM, Qu Wenruo wrote: >> >> >> At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: >>> Dear btrfs experts, >>> >>> I just tried to make use of btrfs send / receive for incremental >>> backups (using btrbk to simplify the process). >>>

[PATCH] btrfsprogs: only install udev rules for udev >= 190

Prior to udev v190, there was no btrfs builtin helper. Installing it on systems with an older udev will cause problems. Signed-off-by: Jeff Mahoney --- configure.ac |8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index

Re: [PATCH] btrfs-progs: fuzz-test: Add image for wrong chunk item in root tree

At 08/30/2016 10:17 PM, David Sterba wrote: On Tue, Aug 30, 2016 at 10:15:50AM +0800, Qu Wenruo wrote: From: Lukas Lueg Reported by Lukas and the same image from him. DATA_RELOC tree's key type is modifed to CHUNK_ITEM, causing btrfsck interpret it as CHUNK_ITEM and

Re: Recommendation on raid5 drive error resolution

On Tue, Aug 30, 2016 at 3:23 PM, Gareth Pye wrote: > On Wed, Aug 31, 2016 at 4:28 AM, Chris Murphy wrote: >> But I'd try a newer kernel before you >> give up on it. > > > Any recommendations on liveCDs that have recent kernels & btrfs tools? > For

Re: Recommendation on raid5 drive error resolution

Or I could just once again select the right boot device in the bios. I think I want some new hardware :) On Wed, Aug 31, 2016 at 7:23 AM, Gareth Pye wrote: > On Wed, Aug 31, 2016 at 4:28 AM, Chris Murphy wrote: >> But I'd try a newer kernel before

Re: Recommendation on raid5 drive error resolution

On Wed, Aug 31, 2016 at 4:28 AM, Chris Murphy wrote: > But I'd try a newer kernel before you > give up on it. Any recommendations on liveCDs that have recent kernels & btrfs tools? For no apparent reason system isn't booting normally either, and I'm reluctant to fix

Re: `btrfs dev del` fails with `No space left on device`

On Tue, Aug 30, 2016 at 5:13 PM, Chris Murphy wrote: > On Tue, Aug 30, 2016 at 4:22 AM, ojab // wrote: >> On Mon, Aug 29, 2016 at 9:05 PM, Chris Murphy >> wrote: >>> On Mon, Aug 29, 2016 at 10:04 AM, ojab // wrote:

Re: [PATCH 3/3] ioctl_xfs_ioc_getfsmap.2: document XFS_IOC_GETFSMAP ioctl

[add a few more relevant lists to cc] On Mon, Aug 29, 2016 at 03:34:11PM -0600, Andreas Dilger wrote: > On Aug 25, 2016, at 5:26 PM, Darrick J. Wong wrote: > > > > Document the new XFS_IOC_GETFSMAP ioctl that returns the physical > > layout of a (disk-based) filesystem.

Re: Recommendation on raid5 drive error resolution

On Tue, Aug 30, 2016 at 12:04 PM, Chris Murphy wrote: > One of us would have to go look in source to see what causes "[ > 163.612313] BTRFS: failed to read the system array on sdd" to appear

Re: Recommendation on raid5 drive error resolution

On Tue, Aug 30, 2016 at 3:58 AM, Gareth Pye wrote: > Okay, things aren't looking good. The FS wont mount for me: > http://pastebin.com/sEEdRxsN Try to mount with -o ro,degraded. I have no idea which device it'll end up dropping, but it might at least get you a read only

Re: Raid 5 to raid 1: balance hangs and scrub aborts. Is this salvageable?

Well it looks like metadata corruption, if it were just a case of data corruption there'd be a file name path associated with the checksum mismatch error; and in that case you'd be able to just delete the file (or first extract a copy of it it with btrfs restore) and then it wouldn't trigger csum

Re: [PATCH 0/5] Fuzzer test fix

>> And special notes for the BUG_ON fix: >> The fix just fixes a small corner, while tons of BUG_ON()/abort() are >> still here and there. >> We need quite a lot of boring work to handle them later. > > Yeah yeah, that's been neglected for a very long time. The kernel has > the abort_transaction

Re: does btrfs-receive use/compare the checksums from the btrfs-send side?

On Sun, Aug 28, 2016 at 10:25:32PM +0200, Christoph Anton Mitterer wrote: > On Sun, 2016-08-28 at 22:19 +0200, Adam Borowski wrote: > > Transports over which you're likely to send a filesystem stream > > already > > protect against corruption. > Well... in some cases,... but not always... just

Re: `btrfs dev del` fails with `No space left on device`

On Tue, Aug 30, 2016 at 4:22 AM, ojab // wrote: > On Mon, Aug 29, 2016 at 9:05 PM, Chris Murphy wrote: >> On Mon, Aug 29, 2016 at 10:04 AM, ojab // wrote: >> What do you get for 'btrfs fi us ' > > $ sudo btrfs fi us /mnt/xxx/ > Overall: >

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

Em Ter, 2016-08-30 às 10:44 -0600, Chris Murphy escreveu: > It sounds related to read-only snapshots to me. I wonder if this > system has something busy that's writing to a file, database, even > maybe something just spamming journald, and then there's a read-only > snapshot during the write,

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

On Tue, Aug 30, 2016 at 6:50 AM, Ronan Arraes Jardim Chagas wrote: > Hi! > > Em Ter, 2016-08-30 às 10:12 +0800, Wang Xiaoguang escreveu: >> For metadata, "bytes_may_use" is about 80GB, it's very big, >> I think this value is very abnormal. >> >> So this explains why you have

Re: [PATCH 0/5] Fuzzer test fix

On Tue, Aug 30, 2016 at 03:22:12PM +0800, Qu Wenruo wrote: > Cc: Lukas Lueg > > Thanks for the fuzz test from Lukas, quite a lot of bugs are exposed. > > The full fixes can be fetched from my github: > https://github.com/adam900710/btrfs-progs/tree/fuzz_fix_160830 > > The

Re: [PATCH 2/2] btrfs-progs: fuzz-test: Add image for unaligned tree block ptr

On Tue, Aug 30, 2016 at 11:29:33AM +0800, Qu Wenruo wrote: > From: Lukas Lueg > > Add test case image for unaligned tree block ptr. > It should lead to BUG_ON in free_extent_buffer(). > > Signed-off-by: Lukas Lueg > Signed-off-by: Qu Wenruo

Re: [PATCH] btrfs-progs: fuzz-test: Add image for wrong chunk item in root tree

On Tue, Aug 30, 2016 at 10:15:50AM +0800, Qu Wenruo wrote: > From: Lukas Lueg > > Reported by Lukas and the same image from him. > > DATA_RELOC tree's key type is modifed to CHUNK_ITEM, causing btrfsck > interpret it as CHUNK_ITEM and cause 0 num_stripes. > > Add the

Re: Raid 5 to raid 1: balance hangs and scrub aborts. Is this salvageable?

The problem with that strategy in my case is that I can't get a handle on what the inode that triggers the problem is. As a result I don't know which files/directories I would need to delete and restore from backup later on to make the scrub/balance succeed. From the first post >>>

Re: [PATCH] btrfs-progs: fuzz-test: Add image for wrong chunk item in root tree

On Tue, Aug 30, 2016 at 10:15:50AM +0800, Qu Wenruo wrote: > From: Lukas Lueg > > Reported by Lukas and the same image from him. > > DATA_RELOC tree's key type is modifed to CHUNK_ITEM, causing btrfsck > interpret it as CHUNK_ITEM and cause 0 num_stripes. > > Add the

Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

Hi! Em Ter, 2016-08-30 às 10:12 +0800, Wang Xiaoguang escreveu: > For metadata, "bytes_may_use" is about 80GB, it's very big, > I think this value is very abnormal. > > So this explains why you have huge unallocated space, you still > get ENOSPC error. In kernel btrfs, there is a function >

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

On Tue, Aug 30, 2016 at 07:44:17PM +0800, Wang Xiaoguang wrote: > Hi, > > On 08/30/2016 07:32 PM, David Sterba wrote: > > On Tue, Aug 30, 2016 at 09:50:20AM +0800, Qu Wenruo wrote: > >>> Are they not? The low-memory patchset has been released in 4.7.1, the > >>> devel branch is always on top of

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

Hi, On 08/30/2016 07:32 PM, David Sterba wrote: On Tue, Aug 30, 2016 at 09:50:20AM +0800, Qu Wenruo wrote: Are they not? The low-memory patchset has been released in 4.7.1, the devel branch is always on top of master branch. I see both branches pushed to the public git repos so I don't see

Re: Raid 5 to raid 1: balance hangs and scrub aborts. Is this salvageable?

I ran this all off my personal machine, so whenever it locked up I just forced a power cycle, I did this probably more than a dozen times. I think the link below is the one I used to translate innodes into file names for me to delete and restore from backups (in addition to the files that where

Re: [PATCH] btrfs-progs: check: skip shared node or leaf check for low_memory mode

On Tue, Aug 30, 2016 at 09:50:20AM +0800, Qu Wenruo wrote: > > Are they not? The low-memory patchset has been released in 4.7.1, the > > devel branch is always on top of master branch. I see both branches > > pushed to the public git repos so I don't see what you mean. > > Unfortunately, the low

Re: Multiple bugs found by fuzzing BTRFS

On Mon, Aug 29, 2016 at 08:47:10PM +0200, Lukas Lueg wrote: > I'll report new issues to bz as they turn up from the current round > only if they represent a yet unreported kind of problem (e.g. there > are stack-based buffer over- and underruns lurking, I lost them due to > a bug in my setup,

Re: `btrfs dev del` fails with `No space left on device`

On Tue, Aug 30, 2016 at 10:22:24AM +, ojab // wrote: > On Mon, Aug 29, 2016 at 9:05 PM, Chris Murphy wrote: > > On Mon, Aug 29, 2016 at 10:04 AM, ojab // wrote: > > What do you get for 'btrfs fi us ' > > $ sudo btrfs fi us /mnt/xxx/ > Overall: >

Re: `btrfs dev del` fails with `No space left on device`

On Mon, Aug 29, 2016 at 9:05 PM, Chris Murphy wrote: > On Mon, Aug 29, 2016 at 10:04 AM, ojab // wrote: > What do you get for 'btrfs fi us ' $ sudo btrfs fi us /mnt/xxx/ Overall: Device size: 3.64TiB Device allocated:

Re: Recommendation on raid5 drive error resolution

Okay, things aren't looking good. The FS wont mount for me: http://pastebin.com/sEEdRxsN On Tue, Aug 30, 2016 at 9:01 AM, Gareth Pye wrote: > When I can get this stupid box to boot from an external drive I'll > have some idea of what is going on -- Gareth Pye -

Re: btrfs and systemd

Am 29.08.2016 um 13:33 schrieb Timofey Titovets: > Do you try: nofail,noauto,x-systemd.automount ? sure this fails too as it has the same timeout in systemd. Mr. Poettering has recommanded me todo the following: # mkdir -p /etc/systemd/system/$(systemd-escape --suffix=mount -p /foo/bar/baz).d/ #

[PATCH 4/5] btrfs-progs: fsck: Avoid abort and BUG_ON in add_tree_backref

Add_tree_backref() can cause BUG_ON() and abort() in quite a lot of cases, from the ENOMEM to existing tree backref records. Change all these BUG_ON() and abort() to return proper values. And modify all callers to handle such problems. Reported-by: Lukas Lueg

[PATCH 3/5] btrfs-progs: fsck: Check bytenr alignment for extent item

Check bytenr alignment for extent item to filter invalid items early. Signed-off-by: Qu Wenruo --- cmds-check.c | 15 +++ 1 file changed, 15 insertions(+) diff --git a/cmds-check.c b/cmds-check.c index 2aa0a7b..c56b176 100644 --- a/cmds-check.c +++

[PATCH 2/5] btrfs-progs: fuzz-test: Add test case for invalid drop level

From: Lukas Lueg Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- tests/fuzz-tests/images/invalid-drop-level.raw.txt | 30 + tests/fuzz-tests/images/invalid-drop-level.raw.xz | Bin 0 -> 3788 bytes

[PATCH 5/5] btrfs-progs: fuzz-test: Add test case for unaligned extent item

From: Lukas Lueg Signed-off-by: Lukas Lueg Signed-off-by: Qu Wenruo --- tests/fuzz-tests/images/unaligned-extent-item.raw.txt | 8 tests/fuzz-tests/images/unaligned-extent-item.raw.xz | Bin 0 -> 3684 bytes 2

[PATCH 1/5] btrfs-progs: fsck: Check drop level before walking through fs tree

Exposed by fuzzed image from Lukas, which contains invalid drop level (16), causing segfault when accessing path->nodes[drop_level]. This patch will check drop level against fs tree level and BTRFS_MAX_LEVEL to avoid such problem. Reported-by: Lukas Lueg Signed-off-by: Qu

[PATCH 0/5] Fuzzer test fix

Cc: Lukas Lueg Thanks for the fuzz test from Lukas, quite a lot of bugs are exposed. The full fixes can be fetched from my github: https://github.com/adam900710/btrfs-progs/tree/fuzz_fix_160830 The branch has go through fuzz and mkfs tests. For full low-memory mode

Re: Raid 5 to raid 1: balance hangs and scrub aborts. Is this salvageable?

Thanks for the response Justin. This is exactly what I tried before posting to the list, but it doesn't seem to get me anywhere. The moment I hit the logical address that is flagged up in btrfs check as problematic the balancing operation just sits there and does nothing, but the operation also