Re: How to fix these btrfs errors
One more while running a scrub dmesg logs showed this [98761.912449] scrub_handle_errored_block: 22 callbacks suppressed [98761.912472] BTRFS warning (device sda): checksum error at logical 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0) in tree 7 [98761.912476] BTRFS warning (device sda): checksum error at logical 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0) in tree 7 [98761.912479] btrfs_dev_stat_print_on_error: 22 callbacks suppressed [98761.912484] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 270, gen 22 [98761.921028] scrub_handle_errored_block: 22 callbacks suppressed [98761.921034] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914584576 on dev /dev/sda [98761.925274] BTRFS warning (device sda): checksum error at logical 5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0) in tree 7 [98761.925278] BTRFS warning (device sda): checksum error at logical 5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0) in tree 7 [98761.925284] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 271, gen 22 [98761.929502] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914600960 on dev /dev/sda [98761.933715] BTRFS warning (device sda): checksum error at logical 5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0) in tree 7 [98761.933717] BTRFS warning (device sda): checksum error at logical 5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0) in tree 7 [98761.933722] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 272, gen 22 [98761.937951] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914617344 on dev /dev/sda [98763.775452] BTRFS warning (device sda): checksum/header error at logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf (level 0) in tree 7 [98763.775457] BTRFS warning (device sda): checksum/header error at logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf (level 0) in tree 7 [98763.775463] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 272, gen 23 [98763.784019] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914584576 on dev /dev/sda [98763.788323] BTRFS warning (device sda): checksum/header error at logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf (level 0) in tree 7 [98763.788328] BTRFS warning (device sda): checksum/header error at logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf (level 0) in tree 7 [98763.788334] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 272, gen 24 [98763.792722] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914600960 on dev /dev/sda [98763.797106] BTRFS warning (device sda): checksum/header error at logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf (level 0) in tree 7 [98763.797111] BTRFS warning (device sda): checksum/header error at logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf (level 0) in tree 7 [98763.797116] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 272, gen 25 [98763.801401] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914617344 on dev /dev/sda [100173.104082] BTRFS error (device sda): bad tree block start 65536 5614914584576 [100173.833202] BTRFS error (device sda): bad tree block start 65536 5614914584576 [100173.845121] BTRFS error (device sda): bad tree block start 499762348032 5614914584576 And then it was aborted scrub status for bdd89c26-038d-49fd-b895-52b8deb989cc scrub started at Sun Apr 30 16:27:12 2017 and was aborted after 03:29:08 total bytes scrubbed: 5.79TiB with 55 errors error details: verify=3 csum=52 corrected errors: 0, uncorrectable errors: 55, unverified errors: 0 I also tried to run this with no luck ./btrfs inspect-internal logical-resolve 5614914584576 /mnt/btrfs/ ERROR: logical ino ioctl: No such file or directory /btrfs inspect-internal logical-resolve 5614914600960 /mnt/btrfs/ ERROR: logical ino ioctl: No such file or directory On 4/30/17, 5:20 PM, "Zach Aller" <zal...@iteris.com> wrote: >Output of one more command. >--- >./btrfs inspect-internal dump-tree /dev/sda > dump.txt >parent transid verify failed on 15732050345984 wanted 73879 found 73881 >parent transid verify failed on 15732050345984 wanted 73879 found 73881 >parent transid verify failed on 15732050345984 wanted 73879 found 73881 >parent transid verify failed on 15732050345984 wanted 73879 found 73881 >Ignoring transid failure >WARNING: eb corrupted: item 0 eb level 3 next level 3, skipping the rest >--- > > > >Here is the end of dump.txt after the above command
Re: How to fix these btrfs errors
Output of one more command. --- ./btrfs inspect-internal dump-tree /dev/sda > dump.txt parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 parent transid verify failed on 15732050345984 wanted 73879 found 73881 Ignoring transid failure WARNING: eb corrupted: item 0 eb level 3 next level 3, skipping the rest --- Here is the end of dump.txt after the above command failed. --- checksum tree key (CSUM_TREE ROOT_ITEM 0) node 15732050329600 level 3 items 15 free 478 generation 73879 owner 7 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33 key (EXTENT_CSUM EXTENT_CSUM 12582912) block 15732050345984 (960208151) gen 73879 key (EXTENT_CSUM EXTENT_CSUM 1036259938304) block 15732062109696 (960208869) gen 73879 key (EXTENT_CSUM EXTENT_CSUM 2055219736576) block 22061012254720 (1346497330) gen 73796 key (EXTENT_CSUM EXTENT_CSUM 3067616186368) block 15732273971200 (960221800) gen 72465 key (EXTENT_CSUM EXTENT_CSUM 4125595508736) block 19670583918592 (1200597163) gen 71747 key (EXTENT_CSUM EXTENT_CSUM 5160724217856) block 22060767379456 (1346482384) gen 73047 key (EXTENT_CSUM EXTENT_CSUM 6225035022336) block 18786519695360 (1146638165) gen 71390 key (EXTENT_CSUM EXTENT_CSUM 7317037834240) block 22060823707648 (1346485822) gen 73791 key (EXTENT_CSUM EXTENT_CSUM 8520881180672) block 22061229129728 (1346510567) gen 72304 key (EXTENT_CSUM EXTENT_CSUM 955614720) block 21773927448576 (1328975064) gen 72764 key (EXTENT_CSUM EXTENT_CSUM 10643353645056) block 22061034897408 (1346498712) gen 73797 key (EXTENT_CSUM EXTENT_CSUM 11886318727168) block 21773509066752 (1328949528) gen 73741 key (EXTENT_CSUM EXTENT_CSUM 13041959669760) block 2532038983680 (154543395) gen 71958 key (EXTENT_CSUM EXTENT_CSUM 14315004522496) block 21773773946880 (1328965695) gen 57898 key (EXTENT_CSUM EXTENT_CSUM 16297194516480) block 22061036748800 (1346498825) gen 72864 uuid tree key (UUID_TREE ROOT_ITEM 0) leaf 29376512 items 0 free space 16283 generation 6 owner 9 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33 data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 0) leaf 29442048 items 2 free space 16061 generation 4 owner 18446744073709551607 fs uuid bdd89c26-038d-49fd-b895-52b8deb989cc chunk uuid 2cf80f39-d64e-489f-acec-8411f5c1bb33 item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160 inode generation 3 transid 0 size 0 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 0 flags 0x0(none) atime 1490407706.0 (2017-03-24 21:08:26) ctime 1490407706.0 (2017-03-24 21:08:26) mtime 1490407706.0 (2017-03-24 21:08:26) otime 1490407706.0 (2017-03-24 21:08:26) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 inode ref index 0 namelen 2 name: .. total bytes 24004303781888 bytes used 18737354412032 uuid bdd89c26-038d-49fd-b895-52b8deb989cc --- On 4/30/17, 4:39 PM, "linux-btrfs-ow...@vger.kernel.org on behalf of Zach Aller" <linux-btrfs-ow...@vger.kernel.org on behalf of zal...@iteris.com> wrote: >It is a recent filesystem the data was written with kernel 4.10, today I >upgraded to 4.11rc8 to see if it helped anything which it did not. > >On 4/30/17, 4:35 PM, "ch...@colorremedies.com on behalf of Chris Murphy" ><ch...@colorremedies.com on behalf of li...@colorremedies.com> wrote: > >>On Sun, Apr 30, 2017 at 3:08 PM, Zach Aller <zal...@iteris.com> wrote: >> >>> uname -a >>> Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 >>> 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux >>> >>> ./btrfs --version >>> btrfs-progs v4.10 >>> >>> ./btrfs fi show >>> Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc >>> Total devices 1 FS bytes used 17.04TiB >>> devid1 size 21.83TiB used 17.28TiB path /dev/sda >> >> >>How old is the file system? Is this a recent problem with just >>4.11rc8? Is most of the 17TB written with a particular kernel version, >>which? >> >> >>> >>> Here is a dmesg snippet >>> >>> >>> [3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc >>> devid 1 transid 72387 /dev/sda >>> [ 12.907658] BTRFS info (device sda): disk space caching is enabled >>> [ 12.907659] BTRFS info (device sda): has skinny extents >>> [ 13.129140] BTRFS info (device sd
Re: How to fix these btrfs errors
It is a recent filesystem the data was written with kernel 4.10, today I upgraded to 4.11rc8 to see if it helped anything which it did not. On 4/30/17, 4:35 PM, "ch...@colorremedies.com on behalf of Chris Murphy" <ch...@colorremedies.com on behalf of li...@colorremedies.com> wrote: >On Sun, Apr 30, 2017 at 3:08 PM, Zach Aller <zal...@iteris.com> wrote: > >> uname -a >> Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 >> 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux >> >> ./btrfs --version >> btrfs-progs v4.10 >> >> ./btrfs fi show >> Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc >> Total devices 1 FS bytes used 17.04TiB >> devid1 size 21.83TiB used 17.28TiB path /dev/sda > > >How old is the file system? Is this a recent problem with just >4.11rc8? Is most of the 17TB written with a particular kernel version, >which? > > >> >> Here is a dmesg snippet >> >> >> [3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc >> devid 1 transid 72387 /dev/sda >> [ 12.907658] BTRFS info (device sda): disk space caching is enabled >> [ 12.907659] BTRFS info (device sda): has skinny extents >> [ 13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0, >> flush 0, corrupt 217, gen 19 >> [20956.415076] BTRFS info (device sda): The free space cache file >> (9804365955072) is invalid. skip it >> [36292.358558] BTRFS warning (device sda): checksum error at logical >> 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level >>0) >> in tree 7 >> [36292.358563] BTRFS warning (device sda): checksum error at logical >> 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level >>0) >> in tree 7 >> [36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, >> flush 0, corrupt 218, gen 19 >> [36292.364717] BTRFS error (device sda): unable to fixup (regular) error >> at logical 5614914584576 on dev /dev/sda > > >Both copies of metadata are failing checksum, so it can't be fixed. It >suggests there's a hardware problem (memory or storage), or maybe a >new bug. > >Have there been any crashes while writing to the file system? >What is the storage stack configuration? 22TB for a single block >device means it's built up from something else. > >I'd dig around for any non-btrfs storage stack related errors in the >meantime, maybe a dev will have some idea what's going on from the >call traces, I'm not sure what they mean. > >-- >Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
How to fix these btrfs errors
So I have been getting a lot of errors with a btrfs filesystem. I would like to figure out how to fix them without losing data or having to redo the data. Below I have the output of some of the requested command. I have ran a btrfs check without the ‹repair option I wanted to check here first to see if it was safe to do so based off what I have read it seems to be unsafe is this true for this case or is there some other thing I should do to try and fix these errors? uname -a Linux server 4.11.0-041100rc8-generic #201704232131 SMP Mon Apr 24 01:32:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux ./btrfs --version btrfs-progs v4.10 ./btrfs fi show Label: none uuid: bdd89c26-038d-49fd-b895-52b8deb989cc Total devices 1 FS bytes used 17.04TiB devid1 size 21.83TiB used 17.28TiB path /dev/sda ./btrfs fi df /mnt/btrfs/ Data, single: total=17.24TiB, used=17.02TiB System, DUP: total=8.00MiB, used=1.81MiB Metadata, DUP: total=19.50GiB, used=17.97GiB GlobalReserve, single: total=512.00MiB, used=0.00B Here is a dmesg snippet [3.633295] BTRFS: device fsid bdd89c26-038d-49fd-b895-52b8deb989cc devid 1 transid 72387 /dev/sda [ 12.907658] BTRFS info (device sda): disk space caching is enabled [ 12.907659] BTRFS info (device sda): has skinny extents [ 13.129140] BTRFS info (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 217, gen 19 [20956.415076] BTRFS info (device sda): The free space cache file (9804365955072) is invalid. skip it [36292.358558] BTRFS warning (device sda): checksum error at logical 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0) in tree 7 [36292.358563] BTRFS warning (device sda): checksum error at logical 5614914584576 on dev /dev/sda, sector 10979229344: metadata leaf (level 0) in tree 7 [36292.358569] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 218, gen 19 [36292.364717] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914584576 on dev /dev/sda [36292.368359] BTRFS warning (device sda): checksum error at logical 5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0) in tree 7 [36292.368363] BTRFS warning (device sda): checksum error at logical 5614914600960 on dev /dev/sda, sector 10979229376: metadata leaf (level 0) in tree 7 [36292.368369] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 219, gen 19 [36292.372004] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914600960 on dev /dev/sda [36292.375825] BTRFS warning (device sda): checksum error at logical 5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0) in tree 7 [36292.375830] BTRFS warning (device sda): checksum error at logical 5614914617344 on dev /dev/sda, sector 10979229408: metadata leaf (level 0) in tree 7 [36292.375836] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 220, gen 19 [36292.379664] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914617344 on dev /dev/sda [36294.785933] BTRFS warning (device sda): checksum/header error at logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf (level 0) in tree 7 [36294.785938] BTRFS warning (device sda): checksum/header error at logical 5614914584576 on dev /dev/sda, sector 10980277920: metadata leaf (level 0) in tree 7 [36294.785945] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 220, gen 20 [36294.811292] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914584576 on dev /dev/sda [36294.815711] BTRFS warning (device sda): checksum/header error at logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf (level 0) in tree 7 [36294.815715] BTRFS warning (device sda): checksum/header error at logical 5614914600960 on dev /dev/sda, sector 10980277952: metadata leaf (level 0) in tree 7 [36294.815721] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 220, gen 21 [36294.819616] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914600960 on dev /dev/sda [36294.823458] BTRFS warning (device sda): checksum/header error at logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf (level 0) in tree 7 [36294.823463] BTRFS warning (device sda): checksum/header error at logical 5614914617344 on dev /dev/sda, sector 10980277984: metadata leaf (level 0) in tree 7 [36294.823469] BTRFS error (device sda): bdev /dev/sda errs: wr 0, rd 0, flush 0, corrupt 220, gen 22 [36294.827285] BTRFS error (device sda): unable to fixup (regular) error at logical 5614914617344 on dev /dev/sda [38085.152105] BTRFS error (device sda): bad tree block start 65536 5614914584576 [38085.920817] BTRFS error (device sda): bad tree block start 65536 5614914584576 [38085.934875] BTRFS error (device sda): bad tree block start 499762348032 5614914584576 [60948.070835] BTRFS info (device sda): space_info 4 has 1641807872
