Probably safer to just copy (cp/scp) files without using hammer's mirroring, provided that your file contents are still safe.
2016-06-04 6:50 GMT+09:00 Antony T Curtis <[email protected]>: > Would it be fixable by using mirror-copy to a different volume and then > mirror-copy back after newfs? > > > On Fri, Jun 3, 2016, 14:24 Tomohiro Kusumi <[email protected]> > wrote: >> >> Using a 5TB disk, I filled up the first layer1 (layer#0 for the first >> 4TB of the filesystem) to see if the statistics gets broken by a bug >> of statistics itself. >> As a result, it's not broken, so I guess your raid issue did break >> your filesystem metadata. >> >> >> [root@]~# df -T /HAMMER >> Filesystem Type 1K-blocks Used Avail Capacity Mounted on >> TEST hammer 4881580032 4310634400 570945632 88% /HAMMER >> [root@]~# umount /HAMMER >> [root@]~# hammer -vvf /dev/da4 blockmap > out >> [root@]~# grep "layer1 " out -A 5 >> layer1 4000000000000000 @2000000000800000 blocks-free 0 >> 4000000000000000 zone=4 vol=0 L1#=0 L2#=0 L1=0 >> L2=0 app=8388608 free=0 fill=100.0 >> crc=9d65da9d-6663e9c8 >> 4000000000800000 zone=4 vol=0 L1#=0 L2#=1 L1=0 >> L2=16 app=8388608 free=0 fill=100.0 >> crc=9d65da9d-6663e9c8 >> 4000000001000000 zone=4 vol=0 L1#=0 L2#=2 L1=0 >> L2=32 app=8388608 free=0 fill=100.0 >> crc=9d65da9d-6663e9c8 >> 4000000001800000 zone=3 vol=0 L1#=0 L2#=3 L1=0 >> L2=48 app=8388608 free=0 fill=100.0 >> crc=9d65da9d-12fb0047 >> 4000000002000000 zone=3 vol=0 L1#=0 L2#=4 L1=0 >> L2=64 app=8388608 free=0 fill=100.0 >> crc=9d65da9d-12fb0047 >> -- >> layer1 4000040000000000 @2000000001000000 blocks-free 69741 >> 4000040000000000 zone=10 vol=0 L1#=1 L2#=0 L1=32 >> L2=0 app=8388608 free=0 fill=100.0 >> crc=5cccfffc-8f523ad6 >> 4000040000800000 zone=10 vol=0 L1#=1 L2#=1 L1=32 >> L2=16 app=8388608 free=0 fill=100.0 >> crc=5cccfffc-8f523ad6 >> 4000040001000000 zone=10 vol=0 L1#=1 L2#=2 L1=32 >> L2=32 app=8388608 free=0 fill=100.0 >> crc=5cccfffc-8f523ad6 >> 4000040001800000 zone=10 vol=0 L1#=1 L2#=3 L1=32 >> L2=48 app=8388608 free=0 fill=100.0 >> crc=5cccfffc-8f523ad6 >> 4000040002000000 zone=10 vol=0 L1#=1 L2#=4 L1=32 >> L2=64 app=8388608 free=0 fill=100.0 >> crc=5cccfffc-8f523ad6 >> [root@]~# tail -22 ./out >> HAMMER zone statistics >> zone # blocks items used[B] >> used[%] >> zone 0 69741 0 0 0 >> zone 1 0 0 0 0 >> zone 2 0 0 0 0 >> zone 3 128 0 1073741824 100 >> zone 4 3 0 25165824 100 >> zone 5 0 0 0 0 >> zone 6 0 0 0 0 >> zone 7 0 0 0 0 >> zone 8 1075 0 8795586560 >> 97.5363 >> zone 9 2 0 5610928 >> 33.4437 >> zone 10 525078 0 4404626128896 >> 99.9989 >> zone 11 0 0 0 0 >> zone 12 0 0 0 0 >> zone 13 0 0 0 0 >> zone 14 0 0 0 0 >> zone 15 452549 0 3796256161792 100 >> >> ---------------------------------------------------------------------- >> total 1048576 0 8210782395824 >> 93.3458 >> 0 bad layer1 >> 0 bad layer2 >> >> >> 2016-06-03 3:27 GMT+09:00 Tomohiro Kusumi <[email protected]>: >> > Your total block count matches your disk size (12TB), >> > so it's either layer2's bytes_free field is broken or this statistics >> > has a bug. >> > >> > >> >> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496 >> >> >> >> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824 >> >> >> >> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360 >> >> >> >> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536 >> >> >> >> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464 >> >> >> >> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928 >> >> >> >> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536 >> >> >> >> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576 >> > >> > These errors come from the reason I mentioned above. >> > >> > Technically speaking, these offsets says the bad data are stored in >> > the second layer1 entry, where 1 layer1 entry is 4TB. >> > In other words these are beyond 4TB of the filesystem. >> > I'll check if hammer blockmap/checkmap have a bug in this case >> > tomorrow, before I say your metadata is broken. >> > >> > >> > 2016-06-03 2:54 GMT+09:00 Antony T Curtis <[email protected]>: >> >> Overall size of the disk is around 12 Tb. >> >> >> >> slightly different result from checkmap, likely because it is a mounted >> >> volume... >> >> >> >>> M block=200004b1f4800000 zone=11 calc 5386240 free, got 5470496 >> >>> >> >>> BM block=200005e72e000000 zone=9 calc 5999696 free, got 5999824 >> >>> >> >>> BM block=200006400c800000 zone=10 calc 7995392 free, got 8001360 >> >>> >> >>> BM block=2000064225000000 zone=10 calc 8192000 free, got 8257536 >> >>> >> >>> BM block=2000064225800000 zone=10 calc 8192000 free, got 8126464 >> >>> >> >>> BM block=200006445d800000 zone=10 calc 8126464 free, got 8060928 >> >>> >> >>> BM block=200006445f000000 zone=10 calc 8323072 free, got 8257536 >> >>> >> >>> BM block=2000077674000000 zone=10 calc 6864896 free, got 7192576 >> >>> >> >>> HAMMER zone statistics >> >>> >> >>> zone # blocks items used[B] used[%] >> >>> >> >>> zone 0 0 0 0 0 >> >>> >> >>> zone 1 0 0 0 0 >> >>> >> >>> zone 2 0 0 0 0 >> >>> >> >>> zone 3 128 0 1073741824 100 >> >>> >> >>> zone 4 4 0 33554432 100 >> >>> >> >>> zone 5 0 0 0 0 >> >>> >> >>> zone 6 0 0 0 0 >> >>> >> >>> zone 7 0 0 0 0 >> >>> >> >>> zone 8 1029 0 7988363264 92.5449 >> >>> >> >>> zone 9 1443 0 2183889824 18.0416 >> >>> >> >>> zone 10 330736 0 2911421278384 104.938 >> >>> >> >>> zone 11 1154 0 14288724672 147.604 >> >>> >> >>> zone 12 0 0 0 0 >> >>> >> >>> zone 13 0 0 0 0 >> >>> >> >>> zone 14 0 0 0 0 >> >>> >> >>> zone 15 0 0 0 0 >> >>> >> >>> ---------------------------------------------------------------------- >> >>> >> >>> total 334494 0 2936989552400 104.67 >> >>> >> >>> 0 bad nodes >> >>> >> >>> 8 errors >> >> >> >> >> >> >> >> On 2 June 2016 at 10:42, Tomohiro Kusumi <[email protected]> >> >> wrote: >> >>> >> >>> 1. What's your approximate disk size (or sum of disks) ? >> >>> >> >>> 2. If your total block counts from blockmap output matches your disk >> >>> size, >> >>> > zone 0 1095221 0 18446744073709486080 >> >>> > 2.00784e+06 >> >>> this line from blockmap output shows your layer2 metadata probably had >> >>> bad (broken) free_bytes. >> >>> Can't tell how it was broken from this output though. >> >>> >> >>> 3. What are the 10 errors for checkmap ? >> >>> You have output before statistics that indicates those errors. >> >>> >> >>> >> >>> 2016-06-03 2:32 GMT+09:00 Antony T Curtis <[email protected]>: >> >>> > Hammer checkmap shows this instead: >> >>> >> >> >>> >> HAMMER zone statistics >> >>> >> >> >>> >> zone # blocks items used[B] used[%] >> >>> >> >> >>> >> zone 0 0 0 0 0 >> >>> >> >> >>> >> zone 1 0 0 0 0 >> >>> >> >> >>> >> zone 2 0 0 0 0 >> >>> >> >> >>> >> zone 3 128 0 1073741824 100 >> >>> >> >> >>> >> zone 4 4 0 33554432 100 >> >>> >> >> >>> >> zone 5 0 0 0 0 >> >>> >> >> >>> >> zone 6 0 0 0 0 >> >>> >> >> >>> >> zone 7 0 0 0 0 >> >>> >> >> >>> >> zone 8 1029 0 7988092928 92.5418 >> >>> >> >> >>> >> zone 9 1443 0 2183767648 18.0406 >> >>> >> >> >>> >> zone 10 330734 0 2911406172336 104.938 >> >>> >> >> >>> >> zone 11 1154 0 14287486768 147.591 >> >>> >> >> >>> >> zone 12 0 0 0 0 >> >>> >> >> >>> >> zone 13 0 0 0 0 >> >>> >> >> >>> >> zone 14 0 0 0 0 >> >>> >> >> >>> >> zone 15 0 0 0 0 >> >>> >> >> >>> >> >> >>> >> ---------------------------------------------------------------------- >> >>> >> >> >>> >> total 334492 0 2936972815936 104.67 >> >>> >> >> >>> >> 0 bad nodes >> >>> >> >> >>> >> 10 errors >> >>> > >> >>> > >> >>> > >> >>> > On 2 June 2016 at 10:14, Antony T Curtis <[email protected]> wrote: >> >>> >> >> >>> >> This is from a hammer blockmap command. >> >>> >> >> >>> >> On 2 June 2016 at 10:13, Tomohiro Kusumi >> >>> >> <[email protected]> >> >>> >> wrote: >> >>> >>> >> >>> >>> From which command is this from ? >> >>> >>> It's not hammer show (because hammer show has non 0 items), so I >> >>> >>> assume either blockmap or checkmap. >> >>> >>> >> >>> >>> 2016-06-03 2:05 GMT+09:00 Antony T Curtis <[email protected]>: >> >>> >>> > Is there any info as to how to repair a Hammer volume or should >> >>> >>> > I >> >>> >>> > simply try >> >>> >>> > to backup and restore? >> >>> >>> > >> >>> >>> >> HAMMER zone statistics >> >>> >>> >> >> >>> >>> >> zone # blocks items used[B] >> >>> >>> >> used[%] >> >>> >>> >> >> >>> >>> >> zone 0 1095221 0 18446744073709486080 >> >>> >>> >> 2.00784e+06 >> >>> >>> >> >> >>> >>> >> zone 1 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 2 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 3 128 0 1073741824 100 >> >>> >>> >> >> >>> >>> >> zone 4 4 0 33554432 100 >> >>> >>> >> >> >>> >>> >> zone 5 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 6 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 7 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 8 1029 0 7986941952 >> >>> >>> >> 92.5284 >> >>> >>> >> >> >>> >>> >> zone 9 1443 0 2183756816 >> >>> >>> >> 18.0405 >> >>> >>> >> >> >>> >>> >> zone 10 330680 0 2910941826784 >> >>> >>> >> 104.939 >> >>> >>> >> >> >>> >>> >> zone 11 1154 0 14287415824 >> >>> >>> >> 147.59 >> >>> >>> >> >> >>> >>> >> zone 12 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 13 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 14 0 0 0 0 >> >>> >>> >> >> >>> >>> >> zone 15 143205 0 1201290608640 100 >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> >> >>> >>> >> ---------------------------------------------------------------------- >> >>> >>> >> >> >>> >>> >> total 1572864 0 4137797780736 >> >>> >>> >> 31.3609 >> >>> >>> >> >> >>> >>> >> 0 bad layer1 >> >>> >>> >> >> >>> >>> >> 0 bad layer2 >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > >> >>> >>> > -- >> >>> >>> > Antony T Curtis >> >>> >>> > >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> -- >> >>> >> Antony T Curtis >> >>> >> 0523 C487 9187 6972 6894 >> >>> >> AEC7 3087 F819 B477 B687 >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > -- >> >>> > Antony T Curtis >> >>> > 0523 C487 9187 6972 6894 >> >>> > AEC7 3087 F819 B477 B687 >> >> >> >> >> >> >> >> >> >> -- >> >> Antony T Curtis >> >> 0523 C487 9187 6972 6894 >> >> AEC7 3087 F819 B477 B687
