Re: spurious full btrfs corruption
Hey Qu. Some update on the corruption issue on my Fujitsu notebook: Finally got around running some memtest on it... and few seconds after it started I already got this: https://paste.pics/1ff8b13b94f31082bc7410acfb1c6693 So plenty of bad memory... I'd say it's probably not so unlikely that *this* was the actual reason for btrfs-metadata corruption. It would perfectly fit to the symptom that I saw shortly before the fs was completely destroyed: The spurious csum errors on reads that went away when I read the file again. I'd guess you also found no further issue with the v1 space cache and/or the tree log in the meantime? So it's probably safe to turn them on again? We(aka you + me testing fixes) can still look in the issue that newer btrfsprogs no longer recover anything from the broken fs, while older to. I can keep the image around, so no reason to hurry from your side. Cheers, Chris. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: spurious full btrfs corruption
Just some addition on this: On Fri, 2018-03-16 at 01:03 +0100, Christoph Anton Mitterer wrote: > The issue that newer btrfs-progs/kernel don't restore anything at all > from my corrupted fs: 4.13.3 seems to be already buggy... 4.7.3 works, but interestingly btrfs-find-super seems to hang on it forever with 100% CPU but apparently no disc IO (works in later versions, where it finishes in a few seconds). Cheers, Chris. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: spurious full btrfs corruption
Hey. Found some time to move on with this: Frist, I think from my side (i.e. restoring as much as possible) I'm basically done now, so everything left over here is looking for possible bugs/etc. I have from my side no indication that my corruptions were actually a bug in btrfs... the new notebook used to be unstable for some time and it might be just that. Also that second occurrence of csum errors (when I made a image from the broken fs to external HDD) kinda hints that it may be a memory issue (though I haven't found time to run memtest86+ yet). So let's just suppose that btrfs code is as rocksolid as its raid56 is ;-P and assume the issues were cause by some unlucky memory corruption just happening at the wrong (important) meta-data. The issue that newer btrfs-progs/kernel don't restore anything at all from my corrupted fs: On Fri, 2018-03-09 at 07:48 +0800, Qu Wenruo wrote: > > So something changed after 4.14, which makes the tools no longer > > being > > able to restore at least that what they could restore at 4.14. > > This seems to be a regression. > But I'm not sure if it's the kernel to blame or the btrfs-progs. > > > > > > > => Some bug recently introduced in btrfs-progs? > > Is the "block mapping error" message from kernel or btrfs-progs? All progs messages unless otherwise noticed. /dev/mapper/restore being the image from the broken SSD fs. Everything below was on the OLD laptop (which has probably no memory or whichever issues) under kernel 4.15.4 and progs 4.15.1. # btrfs-find-root /dev/mapper/restore Couldn't map the block 4503658729209856 No mapping for 4503658729209856-4503658729226240 Couldn't map the block 4503658729209856 Superblock thinks the generation is 2083143 Superblock thinks the level is 1 Found tree root at 58572800 gen 2083143 level 1 Well block 27820032(gen: 2083133 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 Well block 25526272(gen: 2083132 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 Well block 21807104(gen: 2083131 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 Well block 11829248(gen: 2083130 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 Well block 8716288(gen: 2083129 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 Well block 6209536(gen: 2083128 level: 1) seems good, but generation/level doesn't match, want gen: 2083143 level: 1 # btrfs-debug-tree -b 27820032 /dev/mapper/restore btrfs-progs v4.15.1 Couldn't map the block 4503658729209856 No mapping for 4503658729209856-4503658729226240 Couldn't map the block 4503658729209856 bytenr mismatch, want=4503658729209856, have=0 node 27820032 level 1 items 2 free 491 generation 2083133 owner 1 fs uuid b6050e38-716a-40c3-a8df-fcf1dd7e655d chunk uuid ae6b0cc6-bbc5-4131-b3f3-41b748f5a775 key (EXTENT_TREE ROOT_ITEM 0) block 27836416 (1699) gen 2083133 key (1853 INODE_ITEM 0) block 28000256 (1709) gen 2083133 => I *think* (but not 100% sure - would need to double check if it's important for you to know), that the older progs/kernel showed me much more here # btrfs-debug-tree /dev/mapper/restore btrfs-progs v4.15.1 Couldn't map the block 4503658729209856 No mapping for 4503658729209856-4503658729226240 Couldn't map the block 4503658729209856 bytenr mismatch, want=4503658729209856, have=0 ERROR: unable to open /dev/mapper/restore => same here: I *think* (but not 100% sure - would need to double check if it's important for you to know), that the older progs/kernel showed me much more here # btrfs-debug-tree -b 27836416 /dev/mapper/restore btrfs-progs v4.15.1 Couldn't map the block 4503658729209856 No mapping for 4503658729209856-4503658729226240 Couldn't map the block 4503658729209856 bytenr mismatch, want=4503658729209856, have=0 leaf 27836416 items 63 free space 6131 generation 2083133 owner 1 leaf 27836416 flags 0x1(WRITTEN) backref revision 1 fs uuid b6050e38-716a-40c3-a8df-fcf1dd7e655d chunk uuid ae6b0cc6-bbc5-4131-b3f3-41b748f5a775 item 0 key (EXTENT_TREE ROOT_ITEM 0) itemoff 15844 itemsize 439 generation 2083133 root_dirid 0 bytenr 27328512 level 2 refs 1 lastsnap 0 byte_limit 0 bytes_used 182190080 flags 0x0(none) uuid ---- drop key (0 UNKNOWN.0 0) level 0 item 1 key (DEV_TREE ROOT_ITEM 0) itemoff 15405 itemsize 439 generation 2083129 root_dirid 0 bytenr 9502720 level 1 refs 1 lastsnap 0 byte_limit 0 bytes_used 114688 flags 0x0(none) uuid ---- drop key (0 UNKNOWN.0 0) level 0 item 2 key (FS_TREE INODE_REF 6) itemoff 15388 itemsize 17 index 0 namelen 7 name: default item 3 key (FS_TREE ROOT_ITEM 0) itemoff 14949 itemsize 439
Re: spurious full btrfs corruption
On 2018年03月08日 22:38, Christoph Anton Mitterer wrote: > Hey. > > > On Tue, 2018-03-06 at 09:50 +0800, Qu Wenruo wrote: >>> These were the two files: >>> -rw-r--r-- 1 calestyo calestyo 90112 Feb 22 16:46 'Lady In The >>> Water/05.mp3' >>> -rw-r--r-- 1 calestyo calestyo 4892407 Feb 27 23:28 >>> '/home/calestyo/share/music/Lady In The Water/05.mp3' >>> >>> >>> -rw-r--r-- 1 calestyo calestyo 1904640 Feb 22 16:47 'The Hunt For >>> Red October [Intrada]/21.mp3' >>> -rw-r--r-- 1 calestyo calestyo 2968128 Feb 27 23:28 >>> '/home/calestyo/share/music/The Hunt For Red October >>> [Intrada]/21.mp3' >>> >>> with the former (smaller one) being the corrupted one (i.e. the one >>> returned by btrfs-restore). >>> >>> Both are (in terms of filesize) multiples of 4096... what does that >>> mean now? >> >> That means either we lost some file extents or inode items. >> >> Btrfs-restore only found EXTENT_DATA, which contains the pointer to >> the >> real data, and inode number. >> But no INODE_ITEM is found, which records the real inode size, so it >> can >> only use EXTENT_DATA to rebuild as much data as possible. >> That why all recovered one is aligned to 4K. >> >> So some metadata is also corrupted. > > But that can also happen to just some files? Yep, one tree leaf corruption would leads to corruption to several files. > Anyway... still strange that it hit just those two (which weren't > touched for long). > > >>> However, all the qcow2 files from the restore are more or less >>> garbage. >>> During the btrfs-restore it already complained on them, that it >>> would >>> loop too often on them and whether I want to continue or not (I >>> choose >>> n and on another full run I choose y). >>> >>> Some still contain a partition table, some partitions even >>> filesystems >>> (btrfs again)... but I cannot mount them. >> >> I think the same problem happens on them too. >> >> Some data is lost while some are good. >> Anyway, they would be garbage. > > Again, still strange... that so many files (of those that I really > checked) were fully okay,... while those 4 were all broken. One leaf contains some extent data is corrupted here. > > When it only uses EXTENT_DATA, would that mean that it basically breaks > on every border where the file is split up into multiple extents (which > is of course likely for the (CoWed) images that I had. This depends on the leaf boundary. But normally one corrupted leaf can only lead to one or two corruption. For 4 files, we have at least 2 leaf corruption. > > > >>> Would you please try to restore the fs on another system with good memory? >>> >>> Which one? The originally broken fs from the SSD? >> >> Yep. >> >>> And what should I try to find out here? >> >> During restore, if the csum error happens again on the newly created >> destination btrfs. >> (And I recommend use mount option nospace_cache,notreelog on the >> destination fs) > > So an update on this (everything on the OLD notebook with likely good > memory): > > I booted again from USBstick (with 4.15 kernel/progs), > luksOpened+losetup+luksOpened (yes two dm-crypt, the one from the > external restore HDD, then the image file of the SSD which again > contained dmcrypt+LUKS, of which one was the broken btrfs). > > As I've mentioned before... btrfs-restore (and the other tools for > trying to find the bytenr) immediately fail here. > They bring some "block mapping error" and produce no output. > > This worked on my first rescue attempt (where I had 4.12 kernel/progs). > > Since I had no 4.12 kernel/progs at hand anymore, I went to an even > older rescue stick, wich has 4.7 kernel/progs (if I'm not wrong). > There it worked again (on the same image file). > > So something changed after 4.14, which makes the tools no longer being > able to restore at least that what they could restore at 4.14. This seems to be a regression. But I'm not sure if it's the kernel to blame or the btrfs-progs. > > > => Some bug recently introduced in btrfs-progs? Is the "block mapping error" message from kernel or btrfs-progs? > > > > > I finished the dump then (from OLD notebook/good RAM) with 4.7 > kernel/progs,... to the very same external HDD I've used before. > > And afterwards I: > diff -qr --no-dereference restoreFromNEWnotebook/ restoreFromOLDnotebook/ > > => No differences were found, except one further file that was in the > new restoreFromOLDnotebook. Could be that this was a file wich I > deleted on the old restore because of csum errors, but I don't really > remember (actually I thought to remember that there were a few which I > deleted). > > Since all other files were equal (that is at least in terms of file > contents and symlink targets - I didn't compare the metadata like > permissions, dates and owners)... the qcow2 images are garbage as well. > > => No csum errors were recorded in the kernel log during the diff, and > since both, the (remaining) restore results from the NEW notebook and >
Re: spurious full btrfs corruption
Hey. On Tue, 2018-03-06 at 09:50 +0800, Qu Wenruo wrote: > > These were the two files: > > -rw-r--r-- 1 calestyo calestyo 90112 Feb 22 16:46 'Lady In The > > Water/05.mp3' > > -rw-r--r-- 1 calestyo calestyo 4892407 Feb 27 23:28 > > '/home/calestyo/share/music/Lady In The Water/05.mp3' > > > > > > -rw-r--r-- 1 calestyo calestyo 1904640 Feb 22 16:47 'The Hunt For > > Red October [Intrada]/21.mp3' > > -rw-r--r-- 1 calestyo calestyo 2968128 Feb 27 23:28 > > '/home/calestyo/share/music/The Hunt For Red October > > [Intrada]/21.mp3' > > > > with the former (smaller one) being the corrupted one (i.e. the one > > returned by btrfs-restore). > > > > Both are (in terms of filesize) multiples of 4096... what does that > > mean now? > > That means either we lost some file extents or inode items. > > Btrfs-restore only found EXTENT_DATA, which contains the pointer to > the > real data, and inode number. > But no INODE_ITEM is found, which records the real inode size, so it > can > only use EXTENT_DATA to rebuild as much data as possible. > That why all recovered one is aligned to 4K. > > So some metadata is also corrupted. But that can also happen to just some files? Anyway... still strange that it hit just those two (which weren't touched for long). > > However, all the qcow2 files from the restore are more or less > > garbage. > > During the btrfs-restore it already complained on them, that it > > would > > loop too often on them and whether I want to continue or not (I > > choose > > n and on another full run I choose y). > > > > Some still contain a partition table, some partitions even > > filesystems > > (btrfs again)... but I cannot mount them. > > I think the same problem happens on them too. > > Some data is lost while some are good. > Anyway, they would be garbage. Again, still strange... that so many files (of those that I really checked) were fully okay,... while those 4 were all broken. When it only uses EXTENT_DATA, would that mean that it basically breaks on every border where the file is split up into multiple extents (which is of course likely for the (CoWed) images that I had. > > > > > Would you please try to restore the fs on another system with > > > good > > > memory? > > > > Which one? The originally broken fs from the SSD? > > Yep. > > > And what should I try to find out here? > > During restore, if the csum error happens again on the newly created > destination btrfs. > (And I recommend use mount option nospace_cache,notreelog on the > destination fs) So an update on this (everything on the OLD notebook with likely good memory): I booted again from USBstick (with 4.15 kernel/progs), luksOpened+losetup+luksOpened (yes two dm-crypt, the one from the external restore HDD, then the image file of the SSD which again contained dmcrypt+LUKS, of which one was the broken btrfs). As I've mentioned before... btrfs-restore (and the other tools for trying to find the bytenr) immediately fail here. They bring some "block mapping error" and produce no output. This worked on my first rescue attempt (where I had 4.12 kernel/progs). Since I had no 4.12 kernel/progs at hand anymore, I went to an even older rescue stick, wich has 4.7 kernel/progs (if I'm not wrong). There it worked again (on the same image file). So something changed after 4.14, which makes the tools no longer being able to restore at least that what they could restore at 4.14. => Some bug recently introduced in btrfs-progs? I finished the dump then (from OLD notebook/good RAM) with 4.7 kernel/progs,... to the very same external HDD I've used before. And afterwards I: diff -qr --no-dereference restoreFromNEWnotebook/ restoreFromOLDnotebook/ => No differences were found, except one further file that was in the new restoreFromOLDnotebook. Could be that this was a file wich I deleted on the old restore because of csum errors, but I don't really remember (actually I thought to remember that there were a few which I deleted). Since all other files were equal (that is at least in terms of file contents and symlink targets - I didn't compare the metadata like permissions, dates and owners)... the qcow2 images are garbage as well. => No csum errors were recorded in the kernel log during the diff, and since both, the (remaining) restore results from the NEW notebook and the ones just made on the OLD one were read because of the diff,... I'd guess that no further corruption happened in the recent btrfs-restore. On to the next working site: > > > This -28 (ENOSPC) seems to show that the extent tree of the new > > > btrfs > > > is > > > corrupted. > > > > "new" here is dm-1, right? Which is the fresh btrfs I've created on > > some 8TB HDD for my recovery works. > > While that FS shows me: > > [26017.690417] BTRFS info (device dm-2): disk space caching is > > enabled > > [26017.690421] BTRFS info (device dm-2): has skinny extents > > [26017.798959] BTRFS info (device dm-2): bdev /dev/mapper/data-a4 > > errs: >
Re: spurious full btrfs corruption
Christoph Anton Mitterer posted on Tue, 06 Mar 2018 01:57:58 +0100 as excerpted: > In the meantime I had a look of the remaining files that I got from the > btrfs-restore (haven't run it again so far, from the OLD notebook, so > only the results from the NEW notebook here:): > > The remaining ones were multi-GB qcow2 images for some qemu VMs. > I think I had non of these files open (i.e. VMs running) while in the > final corruption phase... but at least I'm sure that not *all* of them > were running. > > However, all the qcow2 files from the restore are more or less garbage. > During the btrfs-restore it already complained on them, that it would > loop too often on them and whether I want to continue or not (I choose n > and on another full run I choose y). > > Some still contain a partition table, some partitions even filesystems > (btrfs again)... but I cannot mount them. Just a note on format choices FWIW, nothing at all to do with your current problem... As my own use-case doesn't involve VMs I'm /far/ from an expert here, but if I'm screwing things up I'm sure someone will correct me and I'll learn something too, but it does /sound/ reasonable, so assuming I'm remembering correctly from a discussion here... Tip: Btrfs and qcow2 are both copy-on-write/COW (it's in the qcow2 name, after all), and doing multiple layers of COW is both inefficient and a good candidate to test for corner-case bugs that wouldn't show up in more normal use-cases. Assuming bug-free it /should/ work properly, of course, but equally of course, bug-free isn't an entirely realistic assumption. =8^0 ... And you're putting btrfs on qcow2 on btrfs... THREE layers of COW! The recommendation was thus to pick what layer you wish to COW at, and use something that's not COW-based at the other layers. Apparently, qemu has raw-format as a choice as well as qcow2, and that was recommended as preferred for use with btrfs (and IIRC what the recommender was using himself). But of course that still leaves cow-based btrfs on both the top and the bottom layers. I suppose which of those is best to remain btrfs, while making the other say ext4 as widest used and hopefully safest general purpose non-COW alternative, depends on the use-case. Of course keeping btrfs at both levels but nocowing the image files on the host btrfs is a possibility as well, but nocow on btrfs has enough limits and caveats that I consider it a second-class "really should have used a different filesystem for this but didn't want to bother setting up a dedicated one" choice, and as such, don't consider it a viable option here. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: spurious full btrfs corruption
On 2018年03月06日 08:57, Christoph Anton Mitterer wrote: > Hey Qu. > > On Thu, 2018-03-01 at 09:25 +0800, Qu Wenruo wrote: >>> - For my personal data, I have one[0] Seagate 8 TB SMR HDD, which I >>> backup (send/receive) on two further such HDDs (all these are >>> btrfs), and (rsync) on one further with ext4. >>> These files have all their SHA512 sums attached as XATTRs, which >>> I >>> regularly test. So I think I can be pretty sure, that there was >>> never >>> a case of silent data corruption and the RAM on the E782 is fine. >> >> Good backup practice can't be even better. > > Well I still would want to add something tape and/or optical based > solution... > But having this depends a bit on having a good way to do incremental > backups, i.e. I wouldn't want to write full copies of everything to > tape/BluRay over and over again, but just the actually added data and > records of metadata changes. > The former (adding just added files is rather easy), but still > recording any changes in metadata (moved/renamed/deleted files, changes > in file dates, permissions, XATTRS etc.). > Also I would always want to backup complete files, so not just changes > to a file, even if just one byte changed of a 4 GiB file... and not > want to have files split over mediums. > > send/receive sounds like a candidate for this (except it works only on > changes, not full files), but I would prefer to have everything in a > standard format like tar which one can rather easily recover manually > if there are failures in the backups. > > > Another missing piece is a tool which (at my manual order) adds hash > sums to the files, and which can verify them > Actually I wrote such a tool already, but as shell script and it simply > forks so often, that it became extremely slow at millions of small > files. > I often found it so useful to have that kind of checksumming in > addition to the kind of checksumming e.g. btrfs does which is not at > the level of whole files. > So if something goes wrong like now, I cannot only verify whether > single extents are valid, but also the chain of them that comprises a > file.. and that just for the point where I defined "now, as it is, the > file is valid",.. and automatically on any writes, as it would be done > at file system level checksumming. > In the current case,... for many files where I had such whole-file- > csums, verifying whether what btrfs-restore gave me was valid or not, > was very easy because of them. > > >> Normally I won't blame memory unless strange behavior happens, from >> unexpected freeze to strange kernel panic. > Me neither... I think bad RAM happens rather rarely these days but > my case may actually be one. > > >> Netconsole would help here, especially when U757 has an RJ45. >> As long as you have another system which is able to run nc, it should >> catch any kernel message, and help us to analyse if it's really a >> memory >> corruption. > Ah thanks... I wasn't even aware of that ^^ > I'll have a look at it when I start inspecting the U757 again in the > next weeks. > > >>> - The notebooks SSD is a Samsung SSD 850 PRO 1TB, the same which I >>> already used with the old notebook. >>> A long SMART check after the corruption, brought no errors. >> >> Also using that SSD with smaller capacity, it's less possible for the >> SSD. > Sorry, what do you mean? :) I'm using the same SSD (with smaller size). So unless some strange thing happened, I won't blame the SSD. > > >> Normally I won't blame memory, but even newly created btrfs, without >> any >> powerloss, it still reports csum error, then it maybe the problem. > That was also my idea... > I may mix up things, but I think I even found a csum error later on the > rescue USB stick (which is also btrfs)... would need to double check > that, though. > >>> - So far, in the data I checked (which as I've said, excludes a >>> lot,.. >>> especially the QEMU images) >>> I found only few cases, where the data I got from btrfs restore >>> was >>> really bad. >>> Namely, two MP3 files. Which were equal to their backup >>> counterparts, >>> but just up to some offset... and the rest of the files were just >>> missing. >> >> Offset? Is that offset aligned to 4K? >> Or some strange offset? > > These were the two files: > -rw-r--r-- 1 calestyo calestyo 90112 Feb 22 16:46 'Lady In The Water/05.mp3' > -rw-r--r-- 1 calestyo calestyo 4892407 Feb 27 23:28 > '/home/calestyo/share/music/Lady In The Water/05.mp3' > > > -rw-r--r-- 1 calestyo calestyo 1904640 Feb 22 16:47 'The Hunt For Red October > [Intrada]/21.mp3' > -rw-r--r-- 1 calestyo calestyo 2968128 Feb 27 23:28 > '/home/calestyo/share/music/The Hunt For Red October [Intrada]/21.mp3' > > with the former (smaller one) being the corrupted one (i.e. the one > returned by btrfs-restore). > > Both are (in terms of filesize) multiples of 4096... what does that > mean now? That means either we lost some file extents or inode items.
Re: spurious full btrfs corruption
Hey Qu. On Thu, 2018-03-01 at 09:25 +0800, Qu Wenruo wrote: > > - For my personal data, I have one[0] Seagate 8 TB SMR HDD, which I > > backup (send/receive) on two further such HDDs (all these are > > btrfs), and (rsync) on one further with ext4. > > These files have all their SHA512 sums attached as XATTRs, which > > I > > regularly test. So I think I can be pretty sure, that there was > > never > > a case of silent data corruption and the RAM on the E782 is fine. > > Good backup practice can't be even better. Well I still would want to add something tape and/or optical based solution... But having this depends a bit on having a good way to do incremental backups, i.e. I wouldn't want to write full copies of everything to tape/BluRay over and over again, but just the actually added data and records of metadata changes. The former (adding just added files is rather easy), but still recording any changes in metadata (moved/renamed/deleted files, changes in file dates, permissions, XATTRS etc.). Also I would always want to backup complete files, so not just changes to a file, even if just one byte changed of a 4 GiB file... and not want to have files split over mediums. send/receive sounds like a candidate for this (except it works only on changes, not full files), but I would prefer to have everything in a standard format like tar which one can rather easily recover manually if there are failures in the backups. Another missing piece is a tool which (at my manual order) adds hash sums to the files, and which can verify them Actually I wrote such a tool already, but as shell script and it simply forks so often, that it became extremely slow at millions of small files. I often found it so useful to have that kind of checksumming in addition to the kind of checksumming e.g. btrfs does which is not at the level of whole files. So if something goes wrong like now, I cannot only verify whether single extents are valid, but also the chain of them that comprises a file.. and that just for the point where I defined "now, as it is, the file is valid",.. and automatically on any writes, as it would be done at file system level checksumming. In the current case,... for many files where I had such whole-file- csums, verifying whether what btrfs-restore gave me was valid or not, was very easy because of them. > Normally I won't blame memory unless strange behavior happens, from > unexpected freeze to strange kernel panic. Me neither... I think bad RAM happens rather rarely these days but my case may actually be one. > Netconsole would help here, especially when U757 has an RJ45. > As long as you have another system which is able to run nc, it should > catch any kernel message, and help us to analyse if it's really a > memory > corruption. Ah thanks... I wasn't even aware of that ^^ I'll have a look at it when I start inspecting the U757 again in the next weeks. > > - The notebooks SSD is a Samsung SSD 850 PRO 1TB, the same which I > > already used with the old notebook. > > A long SMART check after the corruption, brought no errors. > > Also using that SSD with smaller capacity, it's less possible for the > SSD. Sorry, what do you mean? :) > Normally I won't blame memory, but even newly created btrfs, without > any > powerloss, it still reports csum error, then it maybe the problem. That was also my idea... I may mix up things, but I think I even found a csum error later on the rescue USB stick (which is also btrfs)... would need to double check that, though. > > - So far, in the data I checked (which as I've said, excludes a > > lot,.. > > especially the QEMU images) > > I found only few cases, where the data I got from btrfs restore > > was > > really bad. > > Namely, two MP3 files. Which were equal to their backup > > counterparts, > > but just up to some offset... and the rest of the files were just > > missing. > > Offset? Is that offset aligned to 4K? > Or some strange offset? These were the two files: -rw-r--r-- 1 calestyo calestyo 90112 Feb 22 16:46 'Lady In The Water/05.mp3' -rw-r--r-- 1 calestyo calestyo 4892407 Feb 27 23:28 '/home/calestyo/share/music/Lady In The Water/05.mp3' -rw-r--r-- 1 calestyo calestyo 1904640 Feb 22 16:47 'The Hunt For Red October [Intrada]/21.mp3' -rw-r--r-- 1 calestyo calestyo 2968128 Feb 27 23:28 '/home/calestyo/share/music/The Hunt For Red October [Intrada]/21.mp3' with the former (smaller one) being the corrupted one (i.e. the one returned by btrfs-restore). Both are (in terms of filesize) multiples of 4096... what does that mean now? > > - Especially recovering the VM images will take up some longer > > time... > > (I think I cannot really trust what came out from the btrfs restore > > here, since these already brought csum errs before) In the meantime I had a look of the remaining files that I got from the btrfs-restore (haven't run it again so far, from the OLD notebook, so only the results from the NEW notebook
Re: spurious full btrfs corruption
On 2018年02月28日 23:50, Christoph Anton Mitterer wrote: > Hey Qu > > Thanks for still looking into this. > I'm still in the recovery process (and there are other troubles at the > university where I work, so everything will take me some time), but I > have made a dd image of the broken fs, before I put a backup on the > SSD, so that still exist in the case we need to do further debugging. > > To thoroughly describe what has happened, let me go a bit back. > > - Until last ~ September, I was using some Fujitsu E782, for at least > 4 years, with no signs of data corruptions. That's pretty good. > - For my personal data, I have one[0] Seagate 8 TB SMR HDD, which I > backup (send/receive) on two further such HDDs (all these are > btrfs), and (rsync) on one further with ext4. > These files have all their SHA512 sums attached as XATTRs, which I > regularly test. So I think I can be pretty sure, that there was never > a case of silent data corruption and the RAM on the E782 is fine. Good backup practice can't be even better. > - In October I got a new notebook from the university... brand new > Fujitsu U757 in basically the best possible configuration. > I ran memtest86+ in it's normal (non-SMP) mode for roughly a day, > with no errors. > In SMP mode (which is considered experimental, I think) it crashes > reproducible on the same position. Many people seem to have this > (with exactly the same test, address range where it freezes) so I > considered it a bug in memtest86+ SMP mode, which it likely is. > A patch[1], didn't help me. Normally I won't blame memory unless strange behavior happens, from unexpected freeze to strange kernel panic. But when this happens, a lot of things can go wrong. > - Unfortunately from the beginning on that notebook showed many further > issues. > - CPU overheating[2] > - boot freezes, when the initramfs tool of Debian isn't configured > to > blindly add all modules to the initramfs[3]. > - spurious freezes, which I couldn't really debug any further since > there is no serial port... Netconsole would help here, especially when U757 has an RJ45. As long as you have another system which is able to run nc, it should catch any kernel message, and help us to analyse if it's really a memory corruption. > in that cases neither Magic-SysRq nor > even NumLock LEDs and so worked anymore. > These freezes caused me some troubles with dpkg[4]. > The issue I describe there, could also shed some light on the whole > situation, since it resulted out of the freezes. > - The dealer replaced the thermal paste on the CPU and when the CPU > overheating and the freezes didn't go away, they sent the notebook > for one week to Fujitsu in Germany, who allegedly thoroughly tested > it with Windows, and found no errors. That's unfortunately very common for consumer electronics, as few people and cooperation really care about Linux user on consumer laptops. And since there are problems with the system (either hardware or software), I already see a much higher possibility to hard reset. > > - The notebooks SSD is a Samsung SSD 850 PRO 1TB, the same which I > already used with the old notebook. > A long SMART check after the corruption, brought no errors. Also using that SSD with smaller capacity, it's less possible for the SSD. > > > - Just before the corruption on the btrfs happened, I decided it's > time > for a backup of the notebooks SSD (what an irony, I know), so I made > a snapshot of my one and only subvol, removed and non-precious data > from that snapshot, made anotjer ro-snapshot of that and removed the > rw snapshot. > - The kernel was some 4.14. > > - More or less after that, I saw the "BUG: unable to handle kernel > paging request at 9fb75f827100" which I reported here. > I'm not sure whether this had to do with btrfs at all, and even if > whether it was the fs on the SSD, or another one on an external HDD It could be Btrfs, and it would block btrfs module to continue, which is almost a hard reset. > I've had mounted at that time. > sync/umount/remount,rw/shutdown all didn't work, and I had to power > off the node. > - After that things went on basically as I described in my previous > mails to the list already. > - There were some csum erros.> - Checking these files with debsums > (Debian stores MD5s of the > package's files) found no errors. > - A scrub brought no errors. > - Shortly after the scrub, further csum errors as well as: > BTRFS critical (device dm-0): unable to find logical 4503658729209856 > length 4096 > - Then I booted from a rescue USB stick with kernel/btrfs-progs 4.12. > - fsck in normal/lowmem mode were okay except: > Couldn't find free space inode 1 > - I cleared the v1 free space cache > - a scrub failed with "ret=-1, errno=5 (Input/output error)" > - Things like these in the kernel log: > Feb 21 17:43:09 heisenberg
