Re: getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

Am 2017-06-07 um 15:52 schrieb Adam Borowski:
> On Wed, Jun 07, 2017 at 06:48:48PM +0500, Roman Mamedov wrote:
>> On Wed, 7 Jun 2017 15:09:02 +0200
>> Adam Borowski  wrote:
 Yes, because btrfs will return -EIO
 So try dd_rescue
>>>
>>> Or even plain dd conv=noerror.  Both will do a faithful analogue of a
>>> physical disk with a silent data corruption on the affected sectors.
>>
>> Yeah, except "plain dd conv=noerror" will produce a useless corrupted image,
>> because it will be shifted forward by the number of unreadable bytes after 
>> the
>> first error.
>>
>> You also need the "sync" flag in there.
> 
> Doh, yeah.
> 
>> Or just stick with dd_rescue and not try to correct people's perfectly good
>> suggestions with completely wrong and harmful ones.
> 
> On the other hand, you have dd everywhere, while dd_rescue is not available
> on most bootable media not specifically made for rescue purposes.  And
> installing extra software when your disk is already unsound but mostly
> working might be risky.

I had ddrescue (without "_") on the given server (gentoo linux) and was
able to copy the 2 files via

ddrescue -n -e1 $filename /other_fs/$filename

I then cp-ied the file back via plain "cp" and the VM started again.
Looks good, now also btrfs scr is happy!

Now it's interesting if there is an underlying corruption happening in
the hardware (on disks).

thanks so far, Stefan


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

Am 2017-06-07 um 11:37 schrieb Timofey Titovets:

> btrfs scrub start /mnt_path do this trick
> 
> After, you can find info with paths in dmesg

thank you, I think I have the file, it's a qemu-img-file.
I try cp-ing it to another fs first, but assume this will fail, right?



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

getting rid of "csum failed" on a hw raid

[Stefan G. Weichinger]

greets

I have a customer server that was set up with a hardware raid and on top
of that /dev/sda exists a btrfs filesystem.

Yes, I know, that is stupid (can't change that easily).

In dmesg I get flooding:

[2329426.792480] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500
[2329427.844757] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500
[2329428.929354] BTRFS warning (device sda): csum failed ino 7454384 off
81708052480 csum 3746940842 expected csum 3305376500

...

I googled how to spot the file containing that problematic block(?):

find / -type f -exec cp {} /dev/null \;

I have that running within tmux now, are there any other ways to find
that file and somehow fix it?

That btrfs contains the root-fs of the server ... so I would maybe have
to driver there, boot from stick or so and btrfsck the unmounted fs?

thanks, Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

corrupt leaf on kernel 3.17.2, w/ btrfs-progs 3.17

[Stefan G. Weichinger]

My rootfs on /dev/sda2 (an SSD) shows this:

Nov 06 15:37:26 hiro.oops.intern kernel: BTRFS critical (device sda2):
corrupt leaf, slot offset bad: block=100235100160,root=1, slot=142

Scrub runs through fine:

# btrfs scr stat /dev/sda2
scrub status for ea95dbd1-ef4e-48a4-9732-54e6c80b31df
scrub started at Thu Nov  6 15:35:00 2014, running for 172 seconds
total bytes scrubbed: 68.46GiB with 0 errors

Do I have to panic? ;-)

Pls advise how to proceed.

Should I do some repair or btrfsck from a live-medium?

I found this:

http://comments.gmane.org/gmane.comp.file-systems.btrfs/34407

but haven't yet figured out what that means to me.


# btrfs version
Btrfs v3.17

# cat /proc/version
Linux version 3.17.2-gentoo (r...@hiro.oops.intern) (gcc version 4.8.3
(Gentoo 4.8.3 p1.1, pie-0.5.9) ) #1 SMP Thu Nov 6 00:53:58 CET 2014


Stefan
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt leaf on kernel 3.17.2, w/ btrfs-progs 3.17

[Stefan G. Weichinger]

Am 06.11.2014 um 15:45 schrieb Stefan G. Weichinger:

 Pls advise how to proceed.
 
 Should I do some repair or btrfsck from a live-medium?

... had some hefty kernel crashes ...

Now I am on latest sysresccd.

btrfsck v 3.16 ... fails as well

Backing up.

Seems I have to recreate the btrfs (with 3.16?) and all its subvolumes
... *sigh*

Maybe the SSD dies?

S
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt leaf on kernel 3.17.2, w/ btrfs-progs 3.17

[Stefan G. Weichinger]

Am 06.11.2014 um 17:11 schrieb Stefan G. Weichinger:
 Am 06.11.2014 um 15:45 schrieb Stefan G. Weichinger:
 
 Pls advise how to proceed.

 Should I do some repair or btrfsck from a live-medium?
 
 ... had some hefty kernel crashes ...
 
 Now I am on latest sysresccd.
 
 btrfsck v 3.16 ... fails as well
 
 Backing up.

While I am rsyncing I get this again in dmesg:

btrfs: corrupt leaf, slot offset bad: block ...
btrfs no csum found for inode ...

Can I determine which subvol this relates to?

As far as I understand it is likely the one currently synced?

Maybe I could only recreate that one subvolume and restore?

Or do I have to recreate the whole btrfs?

S

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt leaf on kernel 3.17.2, w/ btrfs-progs 3.17

[Stefan G. Weichinger]

Copied a tarball of btrfs-progs v 3.17-r1 over and ran:

btrfs check --repair /dev/sda2

it tells me

incorrect offsets 12499 blabla
items overlap, can't fix
cmds-check.c:2918 fix_item_offset: Assertion `ret` failed

... and crashes

Any help appreciated!

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: corrupt leaf on kernel 3.17.2, w/ btrfs-progs 3.17

[Stefan G. Weichinger]

restoring from backup now
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: general thoughts and questions + general and RAID5/6 stability?

[Stefan G. Weichinger]

Am 23.09.2014 um 15:38 schrieb Austin S Hemmelgarn:

 What features for example?
 Well, running 'mkfs.btrfs -O list-all' with 3.16 btrfs-progs gives the
 following list of features:
 mixed-bg- mixed data and metadata block groups
 extref- increased hard-link limit per file to 65536
 raid56- raid56 extended format
 skinny-metadata- reduced size metadata extent refs
 no-holes- no explicit hole extents for files
 
 mixed-bg is something that you generally wouldn't want to change after
 mkfs.
 extref can be enabled online, and the filesystem metadata gets updated
 as-needed, and dosen't provide any real performance improvement (but is
 needed for some mail servers that have HUGE mail-queues)
 I don't know anything about the raid56 option, but there isn't any way
 to change it after mkfs.
 skinyy-metadata can be changed online, and the format gets updated on
 rewrite of each metadata block.  This one does provide a performance
 improvement (stat() in particular runs noticeably faster).  You should
 probably enable this if it isn't already enabled, even if you don't
 recreate your filesystem.
 no-holes cannot currently be changed online, and is a very recent
 addition (post v3.14 btrfs-progs I believe) that provides improved
 performance for sparse files (which is particularly useful if you are
 doing things with fixed size virtual machine disk images).

Recreating or at least btrfstune -rx for my rootfs would mean that I
have to boot from a live medium bringing recent btrfs-progs, right?

sysresccd brings btrfs-progs-3.14.2 ... that should be enough, ok?

aside from that, the rootfs on my thinkpad shows these features:

# ls /sys/fs/btrfs/bec7dff9-8749-4db4-9a1b-fa844cfcc36a/features/
big_metadata  compress_lzo  extended_iref  mixed_backref

So I only miss the skinny extents ... and no-holes.

Stefan
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: general thoughts and questions + general and RAID5/6 stability?

[Stefan G. Weichinger]

Am 23.09.2014 um 14:08 schrieb Austin S Hemmelgarn:
 On 2014-09-22 16:51, Stefan G. Weichinger wrote:
 Is re-creating btrfs-filesystems *recommended* in any way?

 Does that actually make a difference in the fs-structure?

 I would recommend it, there are some newer features that you can only
 set at mkfs time.  Quite often, when a new feature is implemented, it is
 some time before things are such that it can be enabled online, and even
 then that doesn't convert anything until it is rewritten.

What features for example?

I created my main btrfs a few months ago and would like to avoid
recreating it as this would mean restoring my root-fs on my main
workstation.

Although I would do it if it is worth it ;-)

I assume I could read some kind of version number out of the superblock
or so?

btrfs-show-super ?

S



--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: general thoughts and questions + general and RAID5/6 stability?

[Stefan G. Weichinger]

Am 23.09.2014 um 15:38 schrieb Austin S Hemmelgarn:
 On 2014-09-23 09:06, Stefan G. Weichinger wrote:
 What features for example?
 Well, running 'mkfs.btrfs -O list-all' with 3.16 btrfs-progs gives the
 following list of features:
 mixed-bg- mixed data and metadata block groups
 extref- increased hard-link limit per file to 65536
 raid56- raid56 extended format
 skinny-metadata- reduced size metadata extent refs
 no-holes- no explicit hole extents for files
 
 mixed-bg is something that you generally wouldn't want to change after
 mkfs.
 extref can be enabled online, and the filesystem metadata gets updated
 as-needed, and dosen't provide any real performance improvement (but is
 needed for some mail servers that have HUGE mail-queues)

ok, not needed here

 I don't know anything about the raid56 option, but there isn't any way
 to change it after mkfs.

not needed in my systems.

 skinyy-metadata can be changed online, and the format gets updated on
 rewrite of each metadata block.  This one does provide a performance
 improvement (stat() in particular runs noticeably faster).  You should
 probably enable this if it isn't already enabled, even if you don't
 recreate your filesystem.

So this is done via btrfstune, right?

I will give that a try, for my rootfs it doesn't allow me right now as
it is obviously mounted (live-cd, right?).

 no-holes cannot currently be changed online, and is a very recent
 addition (post v3.14 btrfs-progs I believe) that provides improved
 performance for sparse files (which is particularly useful if you are
 doing things with fixed size virtual machine disk images).

Yes, I have some of those!

 AFAIK there isn't really any 'version number' that has any meaning in
 the superblock (except for telling the kernel that it uses the stable
 disk layout), however, there are flag bits that you can look for
 (compat_flags, compat_ro_flags, and incompat_flags).  I'm not 100%
 certain what each bit means, but on my system with a only 1 month old
 BTRFS filesystem, with extref, skinny-metadata, and no-holes turned on,
 i have compat_flags: 0x0, compat_ro_flags: 0x0, and incompat_flags: 0x16b.
 
 The other potentially significant thing is that the default
 nodesize/leafsize has changed recently from 4096 to 16384, as that gives
 somewhat better performance for most use cases.

I have the 16k for both already.

Thanks for your explanations, I will dig into it as soon as I find the
time. Seems I have to backup/restore quite some stuff ;-)

Stefan

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Unable to mount multiple subvolumes of a single disk

[Stefan G. Weichinger]

Am 17.09.2014 um 15:21 schrieb Chris Mason:

 No problem, the original patch looked right to me too.  We're getting
 closer to rc6, I think at this point I'll revert the original patch
 until the next merge window.  Then we can step back and nail down
 exactly what is going on.

running git-sources 3.17-rc6 now and didn't need to apply the patch.
So you fixed it already?

btrfs fi show also displays devices correctly again.

Thanks, Stefan

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: general thoughts and questions + general and RAID5/6 stability?

[Stefan G. Weichinger]

Am 20.09.2014 um 11:32 schrieb Duncan:

 What I do as part of my regular backup regime, is every few kernel cycles 
 I wipe the (first level) backup and do a fresh mkfs.btrfs, activating new 
 optional features as I believe appropriate.  Then I boot to the new 
 backup and run a bit to test it, then wipe the normal working copy and do 
 a fresh mkfs.btrfs on it, again with the new optional features enabled 
 that I want.

Is re-creating btrfs-filesystems *recommended* in any way?

Does that actually make a difference in the fs-structure?

So far I assumed it was enough to keep the kernel up2date, use current
(stable) btrfs-progs and run some scrub every week or so (not to mention
backups .. if it ain't backed up, it was/isn't important).

Stefan




--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and mount in gentoo linux

[Stefan G. Weichinger]

Am 09.07.2014 03:57, schrieb Qu Wenruo:

 The [/rootfs] is something in the right direction ...
 The bug is that, if you don't use 'subvol=' mount option but use default
 subvolume or 'subvolid=' mount option,
 findmnt will not give the output.

Yes, that sounds true as I switched to mostly using 'subvolid=' over
times. I also use that in my fstab.

 Since 'subvol=' mount option differs from default subvolume mount or
 'subvolid=' mount option in calling behavior,
 'subvol=' uses mount_subtree() vfs call, which records subtree mount info.
 On the other hand, default subvolume mount or 'subvolid=' mount does not
 go through the vfs subtree mount
 but use btrfs's implement, which does not report vfs submount.

OK, good explanation ;-)

 I'll try to investigate further and find whether we can fix it to show
 more info.

Thanks a lot, I think this would be a more expectable behavior in the end.

Stefan
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

btrfs and mount in gentoo linux

[Stefan G. Weichinger]

Hello, btrfs-mailing-list,

I already tried to contact Chris Mason via pm but so far I didn't get a
reply so I taking this approach now:

I am a happy btrfs-user with gentoo linux for some time now and noticed
that the command mount does not show me which subvolid is mounted where.

eg.

# mount | grep sda2

/dev/sda2 on /mnt/btrfs_pool1 type btrfs
(rw,noatime,compress=lzo,ssd,space_cache)

/dev/sda2 on /mnt/oopsfiles type btrfs
(rw,noatime,compress=lzo,ssd,space_cache)

/dev/sda2 on /home type btrfs (rw,noatime,compress=lzo,ssd,space_cache)

See?

I filed a bug against util-linux at bugs.gentoo.org:

https://bugs.gentoo.org/show_bug.cgi?id=510148

but as you see from Comment 5 it isn't resolved yet due to don't know ;-)


Can you tell me where the problem/solution might be located?
Is it a known behavior in a way ... ?

Thanks for looking into it, I appreciate it ... I will then see to
report things back at the gentoo bugzilla to help improving btrfs
support in gentoo!

Regards, Stefan
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and mount in gentoo linux

[Stefan G. Weichinger]

Am 08.07.2014 22:20, schrieb Chris Murphy:
 
 On Jul 8, 2014, at 1:07 PM, Stefan G. Weichinger li...@xunil.at wrote:

 Can you tell me where the problem/solution might be located?
 Is it a known behavior in a way … ?
 
 It's known. Ubuntu has a patch so that mount shows subvolume names. 
 This information is probably available on Gentoo with cat 
 /proc/self/mountinfo.


Unfortunately no.

See example from my laptop:

# grep btrfs /proc/self/mountinfo
14 0 0:14 /rootfs / rw,noatime shared:1 - btrfs /dev/sda3
rw,compress=lzo,ssd,space_cache

51 14 0:14 / /mnt/uncow rw,noatime shared:19 - btrfs /dev/sda3
rw,compress=lzo,ssd,space_cache

156 14 0:35 / /home/sgw rw,noatime shared:117 - btrfs
/dev/mapper/_dev_sda4 rw,compress=lzo,ssd,space_cache


for reference: the subvols are:

# btrfs su list /
ID 256 gen 7982 top level 5 path rootfs
ID 275 gen 3972 top level 5 path __snapshots
ID 283 gen 3274 top level 275 path __snapshots/rootfs_2014_06_14
ID 288 gen 7465 top level 5 path uncow


What does the Ubuntu patch apply to? util-linux?

Thanks! Stefan

--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and mount in gentoo linux

[Stefan G. Weichinger]

Am 08.07.2014 22:19, schrieb Holger Hoffstätte:
 
 On Tue, 08 Jul 2014 21:07:30 +0200, Stefan G. Weichinger wrote:
 
 I am a happy btrfs-user with gentoo linux for some time now and noticed
 that the command mount does not show me which subvolid is mounted where.
 
 Interestingly I just found that this came up in Fedora some time ago:
 https://bugzilla.redhat.com/show_bug.cgi?id=743118
 
 findmnt seems to do the trick, so the underlying functionality in
 libmnt seems to work. Maybe Debian's mount does something differently?

findmnt does not fully show the subvolids or names:


# findmnt  | grep btrfs
//dev/sda3[/rootfs]btrfs
rw,noatime,compress=lzo,ssd,space_cache
├─/mnt/uncow /dev/sda3 btrfs
rw,noatime,compress=lzo,ssd,space_cache
└─/home/sgw  /dev/mapper/_dev_sda4 btrfs
rw,noatime,compress=lzo,ssd,space_cache

The [/rootfs] is something in the right direction ...

S
--
To unsubscribe from this list: send the line unsubscribe linux-btrfs in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html