Re: bad metadata crossing stripe boundary

[Chris Murphy]

On Sun, Apr 3, 2016 at 2:51 AM, Marc Haber  wrote:
> On Sat, Apr 02, 2016 at 01:41:53PM -0600, Chris Murphy wrote:

>> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
>> > That should do a much better job than wipefsing it, shouldnt it?
>>
>> Not really. The first btrfs super is at 64K. The second at 64M. The
>> third at 256G. While wipefs will remove the magic only on the first,
>> mkfs.btrfs will take care of all three. And luksFormat only overwrites
>> the first 132K of a block device. There's a scant chance of bugs
>> related to previous filesystems not being erased, I think this is more
>> likely when mixing and matching filesystems just because the
>> superblocks for each filesystem aren't in the same location.
>
> If I do:
>
> umount /dev/mapper/foo
> cryptsetup close /dev/mapper/foo
> cryptsetup luksFormat /dev/mapper/pv-c_foo
> cryptsetup open /dev/mapper/pv-c_foo foo
>
> and the contents of /dev/mapper/foo would randomly resemble its
> previous contents afterwards, I would be _very_ disturbed.


You wrote "luksFormat the partition, then mkfs.btrfs it" and then also
"wipefs it" where in each case "it" sounded to me like you're
referring to the same partition. But your example makes it more clear
you're referring to mkfs.btrfs being done not on the partition, but on
the dmcrypt device.



> During the
> luksFormat process, a new random symmetric key is created, and
> overwrites the old random symmetric key in the LUKS header. Therefore,
> the following crypto operations are _very_ unlikely to produce
> something that resembles an ext4 fileystem.

All the primates on the planet have a better chance of writing
Shakespeare, this year.

>
> Even if I did:
>
> umount /dev/mapper/foo
> cryptsetup close /dev/mapper/foo
> mkfs.btrfs /dev/mapper/pv-c_foo
>
> (assuming I previously did cryptsetup open /dev/mapper/pv-c_foo foo)
>
> I would be _very_ surprised if the kernel would find something
> resembling and ext4 file system on /dev/mapper/pv-c_foo.

I don't follow that at all, really. It's a sequence, vaguely modified
by a parenthetical in a way that I do not understand, with an outcome
that can't be confirmed or denied because mkfs.ext4 isn't in your
sequence anywhere.

If you  were to see (encrypted) writes to the LV, you could identify
the file system being used based on the write pattern. While the
writes themselves are ciphertext and therefore you don't know what is
being written, the write pattern is not obfuscated by dmcrypt so you
could infer what is doing the writing by that pattern (probably both
application and the filesystem could be identified).


-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 01:41:53PM -0600, Chris Murphy wrote:
> On Thu, Mar 31, 2016 at 11:57 PM, Marc Haber
>  wrote:
> > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> >> Am Thu, 31 Mar 2016 23:00:04 +0200
> >> schrieb Marc Haber :
> >> > I find it somewhere between funny and disturbing that the first call
> >> > of btrfs check made my kernel log the following:
> >> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> >> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> >> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> >> > 67526 /dev/dm-31
> >> >
> >> > No, the filesystem was not converted, it was directly created as
> >> > btrfs, and no, I didn't try mounting it.
> >>
> >> I suggest that your partition contained ext4 before, and you didn't run
> >> wipefs before running mkfs.btrfs.
> >
> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> > That should do a much better job than wipefsing it, shouldnt it?
> 
> Not really. The first btrfs super is at 64K. The second at 64M. The
> third at 256G. While wipefs will remove the magic only on the first,
> mkfs.btrfs will take care of all three. And luksFormat only overwrites
> the first 132K of a block device. There's a scant chance of bugs
> related to previous filesystems not being erased, I think this is more
> likely when mixing and matching filesystems just because the
> superblocks for each filesystem aren't in the same location.

If I do:

umount /dev/mapper/foo
cryptsetup close /dev/mapper/foo
cryptsetup luksFormat /dev/mapper/pv-c_foo
cryptsetup open /dev/mapper/pv-c_foo foo

and the contents of /dev/mapper/foo would randomly resemble its
previous contents afterwards, I would be _very_ disturbed. During the
luksFormat process, a new random symmetric key is created, and
overwrites the old random symmetric key in the LUKS header. Therefore,
the following crypto operations are _very_ unlikely to produce
something that resembles an ext4 fileystem.

Even if I did:

umount /dev/mapper/foo
cryptsetup close /dev/mapper/foo
mkfs.btrfs /dev/mapper/pv-c_foo

(assuming I previously did cryptsetup open /dev/mapper/pv-c_foo foo)

I would be _very_ surprised if the kernel would find something
resembling and ext4 file system on /dev/mapper/pv-c_foo.

> If you're concerned about traces of previous file systems, then use
> the dmcrypt device itself, rather than merely using the original block
> device where merely 132K at the beginning has been overwritten.
> Everytime you format a device, the resulting dmcrypt logical device is
> in effect full of completely random data. A new random key is
> generated each time you use luksFormat, even if you're using the same
> passphrase.

That's what I am saying.

I must be missing something.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 08:31:17PM +0200, Kai Krakow wrote:
> Am Sat, 2 Apr 2016 11:44:32 +0200
> schrieb Marc Haber :
> 
> > On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> > > Am Fri, 1 Apr 2016 07:57:25 +0200
> > > schrieb Marc Haber :  
> > > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:  
> >  [...]  
> >  [...]  
> >  [...]  
> > > > 
> > > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
> > > > it. That should do a much better job than wipefsing it, shouldnt
> > > > it?  
> > > 
> > > Not sure how luksFormat works. If it encrypts what is already on the
> > > device, it would also encrypt orphan superblocks.  
> > 
> > It overwrites the LUKS metadata including the symmetric key that was
> > used to encrypt the existing data. Short of Shor's Algorithm and
> > Quantum Computers, after that operation it is no longer possible to
> > even guess what was on the disk before.
> 
> If it was encrypted before... ;-)

First, it was.

Second, cleartext found on the block device is quite unlikely to be
readable from the unlocked crypto device. I would be very worried if
that were the case.

I must be missing something here.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Chris Murphy]

On Thu, Mar 31, 2016 at 11:57 PM, Marc Haber
 wrote:
> On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
>> Am Thu, 31 Mar 2016 23:00:04 +0200
>> schrieb Marc Haber :
>> > I find it somewhere between funny and disturbing that the first call
>> > of btrfs check made my kernel log the following:
>> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
>> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
>> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
>> > 67526 /dev/dm-31
>> >
>> > No, the filesystem was not converted, it was directly created as
>> > btrfs, and no, I didn't try mounting it.
>>
>> I suggest that your partition contained ext4 before, and you didn't run
>> wipefs before running mkfs.btrfs.
>
> I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> That should do a much better job than wipefsing it, shouldnt it?

Not really. The first btrfs super is at 64K. The second at 64M. The
third at 256G. While wipefs will remove the magic only on the first,
mkfs.btrfs will take care of all three. And luksFormat only overwrites
the first 132K of a block device. There's a scant chance of bugs
related to previous filesystems not being erased, I think this is more
likely when mixing and matching filesystems just because the
superblocks for each filesystem aren't in the same location.

If you're concerned about traces of previous file systems, then use
the dmcrypt device itself, rather than merely using the original block
device where merely 132K at the beginning has been overwritten.
Everytime you format a device, the resulting dmcrypt logical device is
in effect full of completely random data. A new random key is
generated each time you use luksFormat, even if you're using the same
passphrase.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Patrik Lundquist]

On 2 April 2016 at 20:31, Kai Krakow  wrote:
> Am Sat, 2 Apr 2016 11:44:32 +0200
> schrieb Marc Haber :
>
>> On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
>> > Am Fri, 1 Apr 2016 07:57:25 +0200
>> > schrieb Marc Haber :
>> > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
>>  [...]
>>  [...]
>>  [...]
>> > >
>> > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
>> > > it. That should do a much better job than wipefsing it, shouldnt
>> > > it?
>> >
>> > Not sure how luksFormat works. If it encrypts what is already on the
>> > device, it would also encrypt orphan superblocks.
>>
>> It overwrites the LUKS metadata including the symmetric key that was
>> used to encrypt the existing data. Short of Shor's Algorithm and
>> Quantum Computers, after that operation it is no longer possible to
>> even guess what was on the disk before.
>
> If it was encrypted before... ;-)

What does wipefs -n find?
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Sat, 2 Apr 2016 11:44:32 +0200
schrieb Marc Haber :

> On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> > Am Fri, 1 Apr 2016 07:57:25 +0200
> > schrieb Marc Haber :  
> > > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:  
>  [...]  
>  [...]  
>  [...]  
> > > 
> > > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed
> > > it. That should do a much better job than wipefsing it, shouldnt
> > > it?  
> > 
> > Not sure how luksFormat works. If it encrypts what is already on the
> > device, it would also encrypt orphan superblocks.  
> 
> It overwrites the LUKS metadata including the symmetric key that was
> used to encrypt the existing data. Short of Shor's Algorithm and
> Quantum Computers, after that operation it is no longer possible to
> even guess what was on the disk before.

If it was encrypted before... ;-)

-- 
Regards,
Kai

Replies to list-only preferred.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Sat, Apr 02, 2016 at 11:03:53AM +0200, Kai Krakow wrote:
> Am Fri, 1 Apr 2016 07:57:25 +0200
> schrieb Marc Haber :
> > On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> > > Am Thu, 31 Mar 2016 23:00:04 +0200
> > > schrieb Marc Haber :  
> > > > I find it somewhere between funny and disturbing that the first
> > > > call of btrfs check made my kernel log the following:
> > > > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31):
> > > > mounted filesystem with ordered data mode. Opts: (null) Mar 31
> > > > 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr
> > > > devid 1 transid 67526 /dev/dm-31
> > > > 
> > > > No, the filesystem was not converted, it was directly created as
> > > > btrfs, and no, I didn't try mounting it.  
> > > 
> > > I suggest that your partition contained ext4 before, and you didn't
> > > run wipefs before running mkfs.btrfs.  
> > 
> > I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> > That should do a much better job than wipefsing it, shouldnt it?
> 
> Not sure how luksFormat works. If it encrypts what is already on the
> device, it would also encrypt orphan superblocks.

It overwrites the LUKS metadata including the symmetric key that was
used to encrypt the existing data. Short of Shor's Algorithm and
Quantum Computers, after that operation it is no longer possible to
even guess what was on the disk before.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Fri, 1 Apr 2016 07:57:25 +0200
schrieb Marc Haber :

> On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> > Am Thu, 31 Mar 2016 23:00:04 +0200
> > schrieb Marc Haber :  
> > > I find it somewhere between funny and disturbing that the first
> > > call of btrfs check made my kernel log the following:
> > > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31):
> > > mounted filesystem with ordered data mode. Opts: (null) Mar 31
> > > 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr
> > > devid 1 transid 67526 /dev/dm-31
> > > 
> > > No, the filesystem was not converted, it was directly created as
> > > btrfs, and no, I didn't try mounting it.  
> > 
> > I suggest that your partition contained ext4 before, and you didn't
> > run wipefs before running mkfs.btrfs.  
> 
> I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
> That should do a much better job than wipefsing it, shouldnt it?

Not sure how luksFormat works. If it encrypts what is already on the
device, it would also encrypt orphan superblocks.

If it actually wipes the device, the orphan superblock should be gone.

This suggests it does not wipe the device:

https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption:

> Encryption options for LUKS mode
> The cryptsetup action to set up a new dm-crypt device in LUKS
> encryption mode is luksFormat. Unlike the name implies, it does not
> format the device, but sets up the LUKS device header and encrypts
> the master-key with the desired cryptographic options.

Thus, I may very well have an orphan superblock lying around.

-- 
Regards,
Kai

Replies to list-only preferred.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Thu, Mar 31, 2016 at 11:16:30PM +0200, Kai Krakow wrote:
> Am Thu, 31 Mar 2016 23:00:04 +0200
> schrieb Marc Haber :
> > I find it somewhere between funny and disturbing that the first call
> > of btrfs check made my kernel log the following:
> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> > 67526 /dev/dm-31
> > 
> > No, the filesystem was not converted, it was directly created as
> > btrfs, and no, I didn't try mounting it.
> 
> I suggest that your partition contained ext4 before, and you didn't run
> wipefs before running mkfs.btrfs.

I cryptsetup luksFormat'ted the partition before I mkfs.btrfs'ed it.
That should do a much better job than wipefsing it, shouldnt it?

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 23:16:30 +0200
schrieb Kai Krakow <hurikha...@gmail.com>:

> Am Thu, 31 Mar 2016 23:00:04 +0200
> schrieb Marc Haber <mh+linux-bt...@zugschlus.de>:
> 
> > On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:  
> > > Would you please try the following patch based on v4.5
> > > btrfs-progs? https://patchwork.kernel.org/patch/8706891/    
> > 
> > This also fixes the "bad metadata crossing stripe boundary" on my
> > pet patient.
> > 
> > I find it somewhere between funny and disturbing that the first call
> > of btrfs check made my kernel log the following:
> > Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> > filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> > kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> > 67526 /dev/dm-31
> > 
> > No, the filesystem was not converted, it was directly created as
> > btrfs, and no, I didn't try mounting it.  
> 
> I suggest that your partition contained ext4 before, and you didn't
> run wipefs before running mkfs.btrfs. Now, the ext4 superblock is
> still detected because no btrfs structure or block did overwrite it.
> I had a similar problem when I first tried btrfs.
> 
> I think there is some magic dd-fu to damage the ext4 superblock
> without hurting the btrfs itself. But I leave this to the fs devs,
> they could properly tell you.

Tho, you could also try to force detecting btrfs before ext4 by
modifying /etc/filesystems.

-- 
Regards,
Kai

Replies to list-only preferred.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 23:00:04 +0200
schrieb Marc Haber <mh+linux-bt...@zugschlus.de>:

> On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:
> > Would you please try the following patch based on v4.5 btrfs-progs?
> > https://patchwork.kernel.org/patch/8706891/  
> 
> This also fixes the "bad metadata crossing stripe boundary" on my pet
> patient.
> 
> I find it somewhere between funny and disturbing that the first call
> of btrfs check made my kernel log the following:
> Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted
> filesystem with ordered data mode. Opts: (null) Mar 31 22:45:38 fan
> kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 transid
> 67526 /dev/dm-31
> 
> No, the filesystem was not converted, it was directly created as
> btrfs, and no, I didn't try mounting it.

I suggest that your partition contained ext4 before, and you didn't run
wipefs before running mkfs.btrfs. Now, the ext4 superblock is still
detected because no btrfs structure or block did overwrite it. I had a
similar problem when I first tried btrfs.

I think there is some magic dd-fu to damage the ext4 superblock without
hurting the btrfs itself. But I leave this to the fs devs, they could
properly tell you.

-- 
Regards,
Kai

Replies to list-only preferred.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Marc Haber]

On Thu, Mar 31, 2016 at 10:31:49AM +0800, Qu Wenruo wrote:
> Would you please try the following patch based on v4.5 btrfs-progs?
> https://patchwork.kernel.org/patch/8706891/

This also fixes the "bad metadata crossing stripe boundary" on my pet
patient.

I find it somewhere between funny and disturbing that the first call
of btrfs check made my kernel log the following:
Mar 31 22:45:36 fan kernel: [ 6253.178264] EXT4-fs (dm-31): mounted filesystem 
with ordered data mode. Opts: (null)
Mar 31 22:45:38 fan kernel: [ 6255.361328] BTRFS: device label fanbtr devid 1 
transid 67526 /dev/dm-31

No, the filesystem was not converted, it was directly created as
btrfs, and no, I didn't try mounting it.

Greetings
Marc

-- 
-
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany|  lose things."Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Henk Slager]

>> Would you please try the following patch based on v4.5 btrfs-progs?
>> https://patchwork.kernel.org/patch/8706891/
>>
>> According to your output, all the output is false alert.
>> All the extent starting bytenr can be divided by 64K, and I think at
>> initial time, its 'max_size' may be set to 0, causing "start + 0 - 1"
>> to be inside previous 64K range.
>>
>> The patch would update cross_stripe every time the extent is updated,
>> so such temporary false alert should disappear.
>
> Applied and no more reports of crossing stripe boundary - thanks.
>
> Will this go into 4.5.1 or 4.5.2?

It is not in 4.5.1
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bad metadata crossing stripe boundary

[Kai Krakow]

Am Thu, 31 Mar 2016 10:31:49 +0800
schrieb Qu Wenruo :

> Qu Wenruo wrote on 2016/03/31 09:33 +0800:
> >
> >
> > Kai Krakow wrote on 2016/03/28 12:02 +0200:  
> >> Changing subject to reflect the current topic...
> >>
> >> Am Sun, 27 Mar 2016 21:55:40 +0800
> >> schrieb Qu Wenruo :
> >>  
>  [...]  
>  [...]  
> >>
> >> No, btrfs-progs 4.5 reports those, too (as far as I understood,
> >> this includes the fixes for bogus "bad metadata" errors, tho I
> >> thought this has already been fixed in 4.2.1, I used 4.4.1). There
> >> were some nbytes wrong errors before which I already repaired
> >> using "--repair". I think that's okay, I had those in the past and
> >> it looks like btrfsck can repair those now (and I don't have to
> >> delete and recreate the files). It caused problems with "du" and
> >> "df" in the past, a problem that I'm currently facing too. So I
> >> better fixed them.
> >>
> >> With that done, the backup fs now only reports "bad metadata" which
> >> have been there before space cache v2. Full output below.
> >>  
>  [...]  
>  [...]  
> >>
> >> Copy and paste problem. Claws mail pretends to be smarter than me
> >> - I missed to fix that one. ;-)  
> >
> > I was searching for the missing '\n' and hopes to find any chance to
> > submit a new patch.
> > What a pity. :(
> >  
> >>  
>  [...]  
> >>
> >> Helped, it automatically reverted the FS back to space cache v1
> >> with incompat flag cleared. (I wouldn't have enabled v2 if it
> >> wasn't documented that this is possible)
> >>  
>  [...]  
>  [...]  
> >>
> >> It's gone now, ignore that. It's back to the situation before space
> >> cache v2. Minus some "nbytes wrong" errors I had and fixed.  
> >
> > Nice to see it works.
> >  
> >>
> >> Nevertheless, I'm now using btrfs-progs 4.5. Here's the full
> >> output: (the lines seem to be partly out of order, probably due to
> >> the redirection)
> >>
> >> $ sudo btrfsck /dev/sde1 2>&1 | tee btrfsck-label-usb-backup.txt
> >> checking extents
> >> bad metadata [156041216, 156057600) crossing stripe boundary
> >> bad metadata [181403648, 181420032) crossing stripe boundary
> >> bad metadata [392167424, 392183808) crossing stripe boundary
> >> bad metadata [783482880, 783499264) crossing stripe boundary
> >> bad metadata [784924672, 784941056) crossing stripe boundary
> >> bad metadata [130151612416, 130151628800) crossing stripe boundary
> >> bad metadata [162826813440, 162826829824) crossing stripe boundary
> >> bad metadata [162927083520, 162927099904) crossing stripe boundary
> >> bad metadata [619740659712, 619740676096) crossing stripe boundary
> >> bad metadata [619781947392, 619781963776) crossing stripe boundary
> >> bad metadata [619795644416, 619795660800) crossing stripe boundary
> >> bad metadata [619816091648, 619816108032) crossing stripe boundary
> >> bad metadata [620011388928, 620011405312) crossing stripe boundary
> >> bad metadata [890992459776, 890992476160) crossing stripe boundary
> >> bad metadata [891022737408, 891022753792) crossing stripe boundary
> >> bad metadata [891101773824, 891101790208) crossing stripe boundary
> >> bad metadata [891301199872, 891301216256) crossing stripe boundary
> >> bad metadata [1012219314176, 1012219330560) crossing stripe
> >> boundary bad metadata [1017202409472, 1017202425856) crossing
> >> stripe boundary bad metadata [1017365397504, 1017365413888)
> >> crossing stripe boundary bad metadata [1020764422144,
> >> 1020764438528) crossing stripe boundary bad metadata
> >> [1251103342592, 1251103358976) crossing stripe boundary bad
> >> metadata [1251145809920, 1251145826304) crossing stripe boundary
> >> bad metadata [1251147055104, 1251147071488) crossing stripe
> >> boundary bad metadata [1259271225344, 1259271241728) crossing
> >> stripe boundary bad metadata [1266223611904, 1266223628288)
> >> crossing stripe boundary bad metadata [1304750063616,
> >> 130475008) crossing stripe boundary bad metadata
> >> [1304790106112, 1304790122496) crossing stripe boundary bad
> >> metadata [1304850792448, 1304850808832) crossing stripe boundary
> >> bad metadata [1304869928960, 1304869945344) crossing stripe
> >> boundary bad metadata [1305089540096, 1305089556480) crossing
> >> stripe boundary bad metadata [1309581443072, 1309581459456)
> >> crossing stripe boundary bad metadata [1309583671296,
> >> 1309583687680) crossing stripe boundary bad metadata
> >> [1309942808576, 1309942824960) crossing stripe boundary bad
> >> metadata [1310050549760, 1310050566144) crossing stripe boundary
> >> bad metadata [1313031585792, 1313031602176) crossing stripe
> >> boundary bad metadata [1313232912384, 1313232928768) crossing
> >> stripe boundary bad metadata [1555210764288, 1555210780672)
> >> crossing stripe boundary bad metadata [1555395182592,
> >> 1555395198976) crossing stripe boundary bad metadata
> >> [205057678, 2050576760832) crossing stripe boundary bad
> >> metadata [2050803957760, 

Re: bad metadata crossing stripe boundary

[Qu Wenruo]

Qu Wenruo wrote on 2016/03/31 09:33 +0800:



Kai Krakow wrote on 2016/03/28 12:02 +0200:

Changing subject to reflect the current topic...

Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo :


I finally got copy data:

# before mounting let's check the FS:

$ sudo btrfsck /dev/disk/by-label/usb-backup
Checking filesystem on /dev/disk/by-label/usb-backup
UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
checking extents
bad metadata [156041216, 156057600) crossing stripe boundary
bad metadata [181403648, 181420032) crossing stripe boundary
bad metadata [392167424, 392183808) crossing stripe boundary
bad metadata [783482880, 783499264) crossing stripe boundary
bad metadata [784924672, 784941056) crossing stripe boundary
bad metadata [130151612416, 130151628800) crossing stripe boundary
bad metadata [162826813440, 162826829824) crossing stripe boundary
bad metadata [162927083520, 162927099904) crossing stripe boundary
bad metadata [619740659712, 619740676096) crossing stripe boundary
bad metadata [619781947392, 619781963776) crossing stripe boundary
bad metadata [619795644416, 619795660800) crossing stripe boundary
bad metadata [619816091648, 619816108032) crossing stripe boundary
bad metadata [620011388928, 620011405312) crossing stripe boundary
bad metadata [890992459776, 890992476160) crossing stripe boundary
bad metadata [891022737408, 891022753792) crossing stripe boundary
bad metadata [891101773824, 891101790208) crossing stripe boundary
bad metadata [891301199872, 891301216256) crossing stripe boundary
bad metadata [1012219314176, 1012219330560) crossing stripe boundary
bad metadata [1017202409472, 1017202425856) crossing stripe boundary
bad metadata [1017365397504, 1017365413888) crossing stripe boundary
bad metadata [1020764422144, 1020764438528) crossing stripe boundary
bad metadata [1251103342592, 1251103358976) crossing stripe boundary
bad metadata [1251144695808, 1251144712192) crossing stripe boundary
bad metadata [1251147055104, 1251147071488) crossing stripe boundary
bad metadata [1259271225344, 1259271241728) crossing stripe boundary
bad metadata [1266223611904, 1266223628288) crossing stripe boundary
bad metadata [1304750063616, 130475008) crossing stripe boundary
bad metadata [1304790106112, 1304790122496) crossing stripe boundary
bad metadata [1304850792448, 1304850808832) crossing stripe boundary
bad metadata [1304869928960, 1304869945344) crossing stripe boundary
bad metadata [1305089540096, 1305089556480) crossing stripe boundary
bad metadata [1309561651200, 1309561667584) crossing stripe boundary
bad metadata [1309581443072, 1309581459456) crossing stripe boundary
bad metadata [1309583671296, 1309583687680) crossing stripe boundary
bad metadata [1309942808576, 1309942824960) crossing stripe boundary
bad metadata [1310050549760, 1310050566144) crossing stripe boundary
bad metadata [1313031585792, 1313031602176) crossing stripe boundary
bad metadata [1313232912384, 1313232928768) crossing stripe boundary
bad metadata [1555210764288, 1555210780672) crossing stripe boundary
bad metadata [1555395182592, 1555395198976) crossing stripe boundary
bad metadata [205057678, 2050576760832) crossing stripe boundary
bad metadata [2050803957760, 2050803974144) crossing stripe boundary
bad metadata [2050969108480, 2050969124864) crossing stripe
boundary


Already mentioned in another reply, this *seems* to be false alert.
Latest btrfs-progs would help.


No, btrfs-progs 4.5 reports those, too (as far as I understood, this
includes the fixes for bogus "bad metadata" errors, tho I thought this
has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
wrong errors before which I already repaired using "--repair". I think
that's okay, I had those in the past and it looks like btrfsck can
repair those now (and I don't have to delete and recreate the files).
It caused problems with "du" and "df" in the past, a problem that I'm
currently facing too. So I better fixed them.

With that done, the backup fs now only reports "bad metadata" which
have been there before space cache v2. Full output below.


checking free space tree cache and super generation don't match,
space cache will be invalidated checking fs roots

Err, I found a missing '\n' before "checking fs roots".


Copy and paste problem. Claws mail pretends to be smarter than me
- I missed to fix that one. ;-)


I was searching for the missing '\n' and hopes to find any chance to
submit a new patch.
What a pity. :(




And it seems that fs roots and extent tree are all OK.

Quite surprising.
The only possible problem seems to be outdated space cache.

Maybe mount with "-o clear_cache" will help, but I don't think that's
the cause.


Helped, it automatically reverted the FS back to space cache v1 with
incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
documented that this is possible)


checking csums
checking root refs
found 1860217443214 bytes used err is 0
total csum bytes: 1805105116
total 

Re: bad metadata crossing stripe boundary

[Qu Wenruo]

Kai Krakow wrote on 2016/03/28 12:02 +0200:

Changing subject to reflect the current topic...

Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo :


I finally got copy data:

# before mounting let's check the FS:

$ sudo btrfsck /dev/disk/by-label/usb-backup
Checking filesystem on /dev/disk/by-label/usb-backup
UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
checking extents
bad metadata [156041216, 156057600) crossing stripe boundary
bad metadata [181403648, 181420032) crossing stripe boundary
bad metadata [392167424, 392183808) crossing stripe boundary
bad metadata [783482880, 783499264) crossing stripe boundary
bad metadata [784924672, 784941056) crossing stripe boundary
bad metadata [130151612416, 130151628800) crossing stripe boundary
bad metadata [162826813440, 162826829824) crossing stripe boundary
bad metadata [162927083520, 162927099904) crossing stripe boundary
bad metadata [619740659712, 619740676096) crossing stripe boundary
bad metadata [619781947392, 619781963776) crossing stripe boundary
bad metadata [619795644416, 619795660800) crossing stripe boundary
bad metadata [619816091648, 619816108032) crossing stripe boundary
bad metadata [620011388928, 620011405312) crossing stripe boundary
bad metadata [890992459776, 890992476160) crossing stripe boundary
bad metadata [891022737408, 891022753792) crossing stripe boundary
bad metadata [891101773824, 891101790208) crossing stripe boundary
bad metadata [891301199872, 891301216256) crossing stripe boundary
bad metadata [1012219314176, 1012219330560) crossing stripe boundary
bad metadata [1017202409472, 1017202425856) crossing stripe boundary
bad metadata [1017365397504, 1017365413888) crossing stripe boundary
bad metadata [1020764422144, 1020764438528) crossing stripe boundary
bad metadata [1251103342592, 1251103358976) crossing stripe boundary
bad metadata [1251144695808, 1251144712192) crossing stripe boundary
bad metadata [1251147055104, 1251147071488) crossing stripe boundary
bad metadata [1259271225344, 1259271241728) crossing stripe boundary
bad metadata [1266223611904, 1266223628288) crossing stripe boundary
bad metadata [1304750063616, 130475008) crossing stripe boundary
bad metadata [1304790106112, 1304790122496) crossing stripe boundary
bad metadata [1304850792448, 1304850808832) crossing stripe boundary
bad metadata [1304869928960, 1304869945344) crossing stripe boundary
bad metadata [1305089540096, 1305089556480) crossing stripe boundary
bad metadata [1309561651200, 1309561667584) crossing stripe boundary
bad metadata [1309581443072, 1309581459456) crossing stripe boundary
bad metadata [1309583671296, 1309583687680) crossing stripe boundary
bad metadata [1309942808576, 1309942824960) crossing stripe boundary
bad metadata [1310050549760, 1310050566144) crossing stripe boundary
bad metadata [1313031585792, 1313031602176) crossing stripe boundary
bad metadata [1313232912384, 1313232928768) crossing stripe boundary
bad metadata [1555210764288, 1555210780672) crossing stripe boundary
bad metadata [1555395182592, 1555395198976) crossing stripe boundary
bad metadata [205057678, 2050576760832) crossing stripe boundary
bad metadata [2050803957760, 2050803974144) crossing stripe boundary
bad metadata [2050969108480, 2050969124864) crossing stripe
boundary


Already mentioned in another reply, this *seems* to be false alert.
Latest btrfs-progs would help.


No, btrfs-progs 4.5 reports those, too (as far as I understood, this
includes the fixes for bogus "bad metadata" errors, tho I thought this
has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
wrong errors before which I already repaired using "--repair". I think
that's okay, I had those in the past and it looks like btrfsck can
repair those now (and I don't have to delete and recreate the files).
It caused problems with "du" and "df" in the past, a problem that I'm
currently facing too. So I better fixed them.

With that done, the backup fs now only reports "bad metadata" which
have been there before space cache v2. Full output below.


checking free space tree cache and super generation don't match,
space cache will be invalidated checking fs roots

Err, I found a missing '\n' before "checking fs roots".


Copy and paste problem. Claws mail pretends to be smarter than me
- I missed to fix that one. ;-)


I was searching for the missing '\n' and hopes to find any chance to 
submit a new patch.

What a pity. :(




And it seems that fs roots and extent tree are all OK.

Quite surprising.
The only possible problem seems to be outdated space cache.

Maybe mount with "-o clear_cache" will help, but I don't think that's
the cause.


Helped, it automatically reverted the FS back to space cache v1 with
incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
documented that this is possible)


checking csums
checking root refs
found 1860217443214 bytes used err is 0
total csum bytes: 1805105116
total tree bytes: 11793776640
total fs tree bytes: 

bad metadata crossing stripe boundary (was: csum errors in VirtualBox VDI files)

[Kai Krakow]

Changing subject to reflect the current topic...

Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo :

> > I finally got copy data:
> >
> > # before mounting let's check the FS:
> >
> > $ sudo btrfsck /dev/disk/by-label/usb-backup
> > Checking filesystem on /dev/disk/by-label/usb-backup
> > UUID: 1318ec21-c421-4e36-a44a-7be3d41f9c3f
> > checking extents
> > bad metadata [156041216, 156057600) crossing stripe boundary
> > bad metadata [181403648, 181420032) crossing stripe boundary
> > bad metadata [392167424, 392183808) crossing stripe boundary
> > bad metadata [783482880, 783499264) crossing stripe boundary
> > bad metadata [784924672, 784941056) crossing stripe boundary
> > bad metadata [130151612416, 130151628800) crossing stripe boundary
> > bad metadata [162826813440, 162826829824) crossing stripe boundary
> > bad metadata [162927083520, 162927099904) crossing stripe boundary
> > bad metadata [619740659712, 619740676096) crossing stripe boundary
> > bad metadata [619781947392, 619781963776) crossing stripe boundary
> > bad metadata [619795644416, 619795660800) crossing stripe boundary
> > bad metadata [619816091648, 619816108032) crossing stripe boundary
> > bad metadata [620011388928, 620011405312) crossing stripe boundary
> > bad metadata [890992459776, 890992476160) crossing stripe boundary
> > bad metadata [891022737408, 891022753792) crossing stripe boundary
> > bad metadata [891101773824, 891101790208) crossing stripe boundary
> > bad metadata [891301199872, 891301216256) crossing stripe boundary
> > bad metadata [1012219314176, 1012219330560) crossing stripe boundary
> > bad metadata [1017202409472, 1017202425856) crossing stripe boundary
> > bad metadata [1017365397504, 1017365413888) crossing stripe boundary
> > bad metadata [1020764422144, 1020764438528) crossing stripe boundary
> > bad metadata [1251103342592, 1251103358976) crossing stripe boundary
> > bad metadata [1251144695808, 1251144712192) crossing stripe boundary
> > bad metadata [1251147055104, 1251147071488) crossing stripe boundary
> > bad metadata [1259271225344, 1259271241728) crossing stripe boundary
> > bad metadata [1266223611904, 1266223628288) crossing stripe boundary
> > bad metadata [1304750063616, 130475008) crossing stripe boundary
> > bad metadata [1304790106112, 1304790122496) crossing stripe boundary
> > bad metadata [1304850792448, 1304850808832) crossing stripe boundary
> > bad metadata [1304869928960, 1304869945344) crossing stripe boundary
> > bad metadata [1305089540096, 1305089556480) crossing stripe boundary
> > bad metadata [1309561651200, 1309561667584) crossing stripe boundary
> > bad metadata [1309581443072, 1309581459456) crossing stripe boundary
> > bad metadata [1309583671296, 1309583687680) crossing stripe boundary
> > bad metadata [1309942808576, 1309942824960) crossing stripe boundary
> > bad metadata [1310050549760, 1310050566144) crossing stripe boundary
> > bad metadata [1313031585792, 1313031602176) crossing stripe boundary
> > bad metadata [1313232912384, 1313232928768) crossing stripe boundary
> > bad metadata [1555210764288, 1555210780672) crossing stripe boundary
> > bad metadata [1555395182592, 1555395198976) crossing stripe boundary
> > bad metadata [205057678, 2050576760832) crossing stripe boundary
> > bad metadata [2050803957760, 2050803974144) crossing stripe boundary
> > bad metadata [2050969108480, 2050969124864) crossing stripe
> > boundary  
> 
> Already mentioned in another reply, this *seems* to be false alert.
> Latest btrfs-progs would help.

No, btrfs-progs 4.5 reports those, too (as far as I understood, this
includes the fixes for bogus "bad metadata" errors, tho I thought this
has already been fixed in 4.2.1, I used 4.4.1). There were some nbytes
wrong errors before which I already repaired using "--repair". I think
that's okay, I had those in the past and it looks like btrfsck can
repair those now (and I don't have to delete and recreate the files).
It caused problems with "du" and "df" in the past, a problem that I'm
currently facing too. So I better fixed them.

With that done, the backup fs now only reports "bad metadata" which
have been there before space cache v2. Full output below.

> > checking free space tree cache and super generation don't match,
> > space cache will be invalidated checking fs roots  
> Err, I found a missing '\n' before "checking fs roots".

Copy and paste problem. Claws mail pretends to be smarter than me
- I missed to fix that one. ;-)

> And it seems that fs roots and extent tree are all OK.
> 
> Quite surprising.
> The only possible problem seems to be outdated space cache.
> 
> Maybe mount with "-o clear_cache" will help, but I don't think that's 
> the cause.

Helped, it automatically reverted the FS back to space cache v1 with
incompat flag cleared. (I wouldn't have enabled v2 if it wasn't
documented that this is possible)

> > checking csums
> > checking root refs
> > found 1860217443214 bytes used err