Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 03.10.2017 16:39 (localtime):
> Bezüglich Andriy Gapon's Nachricht vom 03.10.2017 16:28 (localtime):
>> On 03/10/2017 17:19, Harry Schmalzbauer wrote:
>>> Have tried several different txg IDs, but the latest 5 or so lead to the
>>> panic and some other random picked all claim missing devices...
>>> Doh, if I only knew about -T some days ago, when I had all 4 devices
>>> available.
>> I don't think that the error is really about the missing devices.
>> Most likely the real problem is that you are going too far back in history 
>> where
>> the data required to import the pool is not present.  It's just that there 
>> is no
>> special error code to report that condition distinctly, so it gets 
>> interpreted
>> as a missing device condition.
> Sounds reasonable.
> When the RAM-corruption happened, a live update was started, where
> several pool availability checks were done. No data write.
> Last data write were view KBytes some minutes before the corruption, and
> the last significant ammount written to that pool was long time before that.
> So I still have hope to find an importable txg ID.
>
> Are they strictly serialized?

Seems so.
Just for the records, I couldn't recover any data yet, but in general,
if a pool isn't damaged that much, the following promising steps were
the ones I got closest:

I have attached dumps of the physical disks as md2 and md3.
'zpool import' offers
cetusPsysDEGRADED
  mirror-0   DEGRADED
8178308212021996317  UNAVAIL  cannot open
md3  ONLINE
  mirror-1   DEGRADED
md2p5ONLINE
4036286347185017167  UNAVAIL  cannot open

Which is ḱnown to be corrupt.
This time I also attached zdb(8) dumps (sparse files) of the remaining
two disks, resp. partition.
Now import offers this:
   pool: cetusPsys
 id: 13207378952432032998
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

cetusPsys   ONLINE
  mirror-0  ONLINE
md5 ONLINE
md3 ONLINE
  mirror-1  ONLINE
md2p5   ONLINE
md4 ONLINE

'zdb -ue cetusPsys' showed me the latest txg ID (3757573 in my case).

So I decremented the txg ID by one and repeated until the following
fatal panicing indicator vanished:
loading space map for vdev 1 of 2, metaslab 108 of 109 ...
WARNING: blkptr at 0x80e0ead00 has invalid CHECKSUM 1
WARNING: blkptr at 0x80e0ead00 has invalid COMPRESS 0
WARNING: blkptr at 0x80e0ead00 DVA 0 has invalid VDEV 2337865727
WARNING: blkptr at 0x80e0ead00 DVA 1 has invalid VDEV 289407040
WARNING: blkptr at 0x80e0ead00 DVA 2 has invalid VDEV 3959586324

Which was 'zdb -c -t 3757569 -AAA -e cetusPsys':

Traversing all blocks to verify metadata checksums and verify nothing
leaked ...

loading space map for vdev 1 of 2, metaslab 108 of 109 ...
89.0M completed (   6MB/s) estimated time remaining: 3hr 34min 47sec
zdb_blkptr_cb: Got error 122 reading <69, 0, 0, c>  -- skipping
86.8G completed ( 588MB/s) estimated time remaining: 0hr 00min 00sec   
Error counts:

errno  count
  122  1
leaked space: vdev 0, offset 0xa01084200, size 512
leaked space: vdev 0, offset 0xd0dc23c00, size 512
leaked space: vdev 0, offset 0x2380182200, size 3072
leaked space: vdev 0, offset 0x2380189a00, size 1536
leaked space: vdev 0, offset 0x2380183000, size 1536
leaked space: vdev 0, offset 0x238039a200, size 2560
leaked space: vdev 0, offset 0x238039be00, size 18944
leaked space: vdev 0, offset 0x23801b3200, size 9216
leaked space: vdev 0, offset 0x33122a8800, size 512
leaked space: vdev 1, offset 0x2808f1600, size 512
leaked space: vdev 1, offset 0x2808f1e00, size 512
leaked space: vdev 1, offset 0x2808f2e00, size 4096
leaked space: vdev 1, offset 0x2808f1a00, size 512
leaked space: vdev 1, offset 0x9010e6c00, size 512
leaked space: vdev 1, offset 0x23c5ad9c00, size 512
leaked space: vdev 1, offset 0x2e00ad4800, size 512
leaked space: vdev 1, offset 0x2f0030b200, size 50176
leaked space: vdev 1, offset 0x2f000ca800, size 512
leaked space: vdev 1, offset 0x2f003a9800, size 15360
leaked space: vdev 1, offset 0x2f003af600, size 13312
leaked space: vdev 1, offset 0x2f00715c00, size 1024
leaked space: vdev 1, offset 0x2f003adc00, size 6144
leaked space: vdev 1, offset 0x2f00363600, size 38912
block traversal size 93540302336 != alloc 93540473344 (leaked 171008)

bp count: 3670624
ganged count:   0
bp logical:96083156992  avg:  26176
bp physical:   93308853248  avg:  25420 compression:   1.03
bp allocated:  93540302336  avg:  25483 compression:   1.03
bp deduped: 0ref>1:  0   deduplication:   1.00
SPA allocated: 93540473344 used: 19.98%

additional, non-pointer bps of type 0:  48879
Dittoed blocks on same vdev: 23422


In my case, import didn't work with the highest non-panicing txg ID:
zpool 

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

Bezüglich Andriy Gapon's Nachricht vom 03.10.2017 16:28 (localtime):
> On 03/10/2017 17:19, Harry Schmalzbauer wrote:
>> Have tried several different txg IDs, but the latest 5 or so lead to the
>> panic and some other random picked all claim missing devices...
>> Doh, if I only knew about -T some days ago, when I had all 4 devices
>> available.
> 
> I don't think that the error is really about the missing devices.
> Most likely the real problem is that you are going too far back in history 
> where
> the data required to import the pool is not present.  It's just that there is 
> no
> special error code to report that condition distinctly, so it gets interpreted
> as a missing device condition.

Sounds reasonable.
When the RAM-corruption happened, a live update was started, where
several pool availability checks were done. No data write.
Last data write were view KBytes some minutes before the corruption, and
the last significant ammount written to that pool was long time before that.
So I still have hope to find an importable txg ID.

Are they strictly serialized?

Dou you know any other possibility to get data from a dataset (by
objetct id) without importing the whole pool?

zdz successfully checks the one datset (object ID) I'm interested in;
the rest of the pool isnt much an issue...

Thank you very much for your help!

-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Andriy Gapon]

On 03/10/2017 17:19, Harry Schmalzbauer wrote:
> Have tried several different txg IDs, but the latest 5 or so lead to the
> panic and some other random picked all claim missing devices...
> Doh, if I only knew about -T some days ago, when I had all 4 devices
> available.

I don't think that the error is really about the missing devices.
Most likely the real problem is that you are going too far back in history where
the data required to import the pool is not present.  It's just that there is no
special error code to report that condition distinctly, so it gets interpreted
as a missing device condition.
-F/-X/-T is not guaranteed to work as the old (freed, overwritten) data is not
kept indefinitely.


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

 Bezüglich Andriy Gapon's Nachricht vom 03.10.2017 11:20 (localtime):
> On 03/10/2017 11:43, Harald Schmalzbauer wrote:
> ...
>>  action: The pool can be imported despite missing or damaged devices.  The
>> fault tolerance of the pool may be compromised if imported.
> ...
>> Is it impossible to import degraded pools in general, or only together> with 
>> "-X -T"?
> It should be possible to import degraded pools...
> Perhaps the pool originally had more devices?  Like log devices.
> Or maybe there is some issue with the txg you picked.
>
> By the way, I think that you didn't have to provide -T option for -F or -X.
> It's either -F or -X or -T , the first two try to figure out txg
> automatically.  But I could be wrong.

You're right that -T works without F[X] flag, but -X needs -F.

Unfortunately not specifying a distinct txg leads to panic.
Specifying one leads to "device is missing".
Which is true, but only redundant data...

It's abosultely sure that this pool never had any log or cache or other
device than the two mirror vdevs.
zdb -l confirms that.

Have tried several different txg IDs, but the latest 5 or so lead to the
panic and some other random picked all claim missing devices...
Doh, if I only knew about -T some days ago, when I had all 4 devices
available.
I haven't expected problems due to missing redundant mirrors.

Can anybody imagine why degraded import doesn't work in my case and how
to work arround?

Will try to provide the sparse zdb dumps in addition, maybe that changes
anything. But I'm sure these don't have much data., dump time was within
3 seconds at most.

Thanks,

-harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Andriy Gapon]

On 03/10/2017 11:43, Harald Schmalzbauer wrote:
...
>  action: The pool can be imported despite missing or damaged devices.  The
> fault tolerance of the pool may be compromised if imported.
...
> Is it impossible to import degraded pools in general, or only together> with 
> "-X -T"?
It should be possible to import degraded pools...
Perhaps the pool originally had more devices?  Like log devices.
Or maybe there is some issue with the txg you picked.

By the way, I think that you didn't have to provide -T option for -F or -X.
It's either -F or -X or -T , the first two try to figure out txg
automatically.  But I could be wrong.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harald Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 02.10.2017 20:28 (localtime):
> Bezüglich Andriy Gapon's Nachricht vom 02.10.2017 13:49 (localtime):
>> On 01/10/2017 00:38, Harry Schmalzbauer wrote:
>>> Now my striped mirror has all 4 devices healthy available, but all
>>> datasets seem to be lost.
>>> No problem for 450G (99,9_%), but there's a 80M dataset which I'm really
>>> missing :-(
>> If it's not too late now, you may try to experiment with an "unwind" / 
>> "extreme
>> unwind" import using -F -n / -X -n.  Or manually specifying a txg number for
>> import (in read-only mode).
> Thanks for your reply!
>
> I had dumped one of each mirror's drive and attaching it as memory disk
> works as intended.
> So "zfs import" offers me the corrupt backup (on the host with a already
> recreated pool).
>
> Unfortunately my knowledge about ZFS internals (transaction group number
> relations to (ü)uberblocks) doesn't allow me to follow your hint.
>
> How can I determine the last txg#, resp. the ones before the last?
> I guess 'zpool import -t' is the tool/parameter to use.
> ZFS has wonderful documentation, but although this was a perfect reason
> to start learning the details about my beloved ZFS, I don't have the
> time to.
>
> Is there a zdb(8) aequivalent of 'zpool import -t', so I can issue the
> zdb check, wich doesn't crash the kernel but only zdb(8)?
>
> For regular 'zpool import', 'zdb -ce' seems to be such a synonym. At
> least the crash report is identical, see my reply to Scott Bennett's post..

Made some progress.
Unfortunately the zpool(8) man page doesn't mention some available
options for the 'import' command.
-T was the important one for me some days ago...
Utilizing internet search engines would have discovered -T, but I
haven't tried...

This post describes the -T (-t for zdb) option very well:
https://lists.freebsd.org/pipermail/freebsd-hackers/2013-July/043131.html


After attaching the dumps from two of the 4 drives as memory disk,
'zpool import' offers me:
   pool: cetusPsys
 id: 13207378952432032998
  state: DEGRADED
 status: The pool was last accessed by another system.
 action: The pool can be imported despite missing or damaged devices.  The
fault tolerance of the pool may be compromised if imported.
   see: http://illumos.org/msg/ZFS-8000-EY
 config:

cetusPsysDEGRADED
  mirror-0   DEGRADED
8178308212021996317  UNAVAIL  cannot open
md3  ONLINE
  mirror-1   DEGRADED
md2p5ONLINE
4036286347185017167  UNAVAIL  cannot open

With 'zdb -lu /dev/md3' I picked a TXGid.

The following doesn't panic the kernel:
 zpool import -o readonly=on -f -R /net -F -X -n -T 3757541
13207378952432032998 cetusPalt

But:
As soon as I don't tell test-run (not specifying -n), it fails with:
cannot import 'cetusPsys': one or more devices is currently unavailable

I'm clueless again :-(
Is it impossible to import degraded pools in general, or only together
with "-X -T"?

Any tricks to convince zpool import to ignore the missing devices?
I dumped only 2 of 4 disks, the physical disks are in usage.
But people made me hope I have chances to recover data :-) Seems -T had
really offered a way to do that, but I wasn't aware of it some days ago,
so I only have these two tumps, but all data needed should be available
– I thought...

Thanks,

-harry




signature.asc
Description: OpenPGP digital signature

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

Bezüglich Andriy Gapon's Nachricht vom 02.10.2017 13:49 (localtime):
> On 01/10/2017 00:38, Harry Schmalzbauer wrote:
>> Now my striped mirror has all 4 devices healthy available, but all
>> datasets seem to be lost.
>> No problem for 450G (99,9_%), but there's a 80M dataset which I'm really
>> missing :-(
> 
> If it's not too late now, you may try to experiment with an "unwind" / 
> "extreme
> unwind" import using -F -n / -X -n.  Or manually specifying a txg number for
> import (in read-only mode).

Thanks for your reply!

I had dumped one of each mirror's drive and attaching it as memory disk
works as intended.
So "zfs import" offers me the corrupt backup (on the host with a already
recreated pool).

Unfortunately my knowledge about ZFS internals (transaction group number
relations to (ü)uberblocks) doesn't allow me to follow your hint.

How can I determine the last txg#, resp. the ones before the last?
I guess 'zpool import -t' is the tool/parameter to use.
ZFS has wonderful documentation, but although this was a perfect reason
to start learning the details about my beloved ZFS, I don't have the
time to.

Is there a zdb(8) aequivalent of 'zpool import -t', so I can issue the
zdb check, wich doesn't crash the kernel but only zdb(8)?

For regular 'zpool import', 'zdb -ce' seems to be such a synonym. At
least the crash report is identical, see my reply to Scott Bennett's post..

Thanks,

-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

Bezüglich Scott Bennett's Nachricht vom 01.10.2017 15:20 (localtime):
>  On Sat, 30 Sep 2017 23:38:45 +0200 Harry Schmalzbauer 
> 
> wrote:

…
>>
>> OpenIndiana also panics at regular import.
>> Unfortunately I don't know the aequivalent of vfs.zfs.recover in OI.
>>
>> panic[cpu1]/thread=ff06dafe8be0: blkptr at ff06dbe63000 has
>> invalid CHECKSUM 1
>>
>> Warning - stack not written to the dump buffer
>> ff001f67f070 genunix:vcmn_err+42 ()
>> ff001f67f0e0 zfs:zfs_panic_recover+51 ()
>> ff001f67f140 zfs:zfs_blkptr_verify+8d ()
>> ff001f67f220 zfs:zio_read+55 ()
>> ff001f67f310 zfs:arc_read+662 ()
>> ff001f67f370 zfs:traverse_prefetch_metadata+b5 ()
>> ff001f67f450 zfs:traverse_visitbp+1c3 ()
>> ff001f67f4e0 zfs:traverse_dnode+af ()
>> ff001f67f5c0 zfs:traverse_visitbp+6dd ()
>> ff001f67f720 zfs:traverse_impl+1a6 ()
>> ff001f67f830 zfs:traverse_pool+9f ()
>> ff001f67f8a0 zfs:spa_load_verify+1e6 ()
>> ff001f67f990 zfs:spa_load_impl+e1c ()
>> ff001f67fa30 zfs:spa_load+14e ()
>> ff001f67fad0 zfs:spa_load_best+7a ()
>> ff001f67fb90 zfs:spa_import+1b0 ()
>> ff001f67fbe0 zfs:zfs_ioc_pool_import+10f ()
>> ff001f67fc80 zfs:zfsdev_ioctl+4b7 ()
>> ff001f67fcc0 genunix:cdev_ioctl+39 ()
>> ff001f67fd10 specfs:spec_ioctl+60 ()
>> ff001f67fda0 genunix:fop_ioctl+55 ()
>> ff001f67fec0 genunix:ioctl+9b ()
>> ff001f67ff10 unix:brand_sys_sysenter+1c9 ()
>>
>> This is a important lesson.
>> My impression was that it's not possible to corrupt a complete pool, but
>> there's always a way to recover healthy/redundant data.
>> Now my striped mirror has all 4 devices healthy available, but all
>> datasets seem to be lost.
>> No problem for 450G (99,9_%), but there's a 80M dataset which I'm really
>> missing :-(
>>
>> Unfortunately I don't know the DVA and blkptr internals, so I won't
>> write a zfs_fsck(8) soon ;-)
>>
>> Does it make sense to dump the disks for further analysis?
>> I need to recreate the pool because I need the machine's resources... :-(
>> Any help highly appreciated!
>>
>  First, if it's not too late already, make a copy of the pool's cache 
> file,
> and save it somewhere in case you need it unchanged again.
>  Can zdb(8) see it without causing a panic, i.e., without importing the
> pool?  You might be able to track down more information if zdb can get you in.

Thank you very much for your help.

zdb(8) is able to get all config data, along with all dataset information.

For the records, I'll provide zdb(8) output beyond.

In the mean time I recreated the pool and the host is back to live.
Since other pools weren't affected and had plenty of space, I dumped two
of the 4 drives along with the zdb(8) -x dump, which I don't know what
it exactly dumps (all blocks accessed!?!; result is big sparse file, but
the time it took to write them down't allow them to have anything but
metadata, at best).

Attaching the two  native dumps as memory-disk works for "zpool import" :-)
To be continued as answer to Andriy Gaoon's reply from today...

>  Another thing you could try with an admittedly very low probability of
> working would be to try importing the pool with one drive of one mirror
> missing, then try it with a different drive of one mirror, and so on the minor
> chance that the critical error is limited to one drive.  If you find a case
> where that works, then you could try to rebuild the missing drive and then run
> a scrub.  Or vice versa.  This one is time-consuming, I would imagine, given

I did try, although I had no hope that this could change the picture,
since the cause of the incosistency wasn't drive related.
And as expected, I had no luck.

Dataset mos [META], ID 0, cr_txg 4, 19.2M, 6503550977762669098 objects

Object  lvl   iblk   dblk  dsize  lsize   %full  type
 21   128K512  05120.00  DSL directory

Dataset mos [META], ID 0, cr_txg 4, 19.2M, 6503550977762669098 objects

Object  lvl   iblk   dblk  dsize  lsize   %full  type
 21   128K512  05120.00  DSL directory

loading space map for vdev 1 of 2, metaslab 108 of 109 ...
error: blkptr at 0x80d726040 has invalid CHECKSUM 1

Traversing all blocks to verify checksums and verify nothing leaked ...

Assertion failed: (!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) ==
BP_EMBEDDED_TYPE_DATA), file
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c,
line 5220.
loading space map for vdev 1 of 2, metaslab 108 of 109 ...
error: blkptr at 0x80b482e80 has invalid CHECKSUM 1

Traversing all blocks to verify checksums and verify nothing leaked ...

Assertion failed: (!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) ==
BP_EMBEDDED_TYPE_DATA), file
/usr/local/share/deploy-tools/RELENG_11/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c,
line 5220.
loading space map for vdev 1 of 2, metaslab 108 of 109 ...
WARNING: Assertion failed: 

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Andriy Gapon]

On 01/10/2017 00:38, Harry Schmalzbauer wrote:
> Now my striped mirror has all 4 devices healthy available, but all
> datasets seem to be lost.
> No problem for 450G (99,9_%), but there's a 80M dataset which I'm really
> missing :-(

If it's not too late now, you may try to experiment with an "unwind" / "extreme
unwind" import using -F -n / -X -n.  Or manually specifying a txg number for
import (in read-only mode).

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Scott Bennett]

 On Sat, 30 Sep 2017 23:38:45 +0200 Harry Schmalzbauer 
wrote:
> Bez?glich Harry Schmalzbauer's Nachricht vom 30.09.2017 19:25 (localtime):
>>  Bez?glich Harry Schmalzbauer's Nachricht vom 30.09.2017 18:30 (localtime):
>>>  Bad surprise.
>>> Most likely I forgot to stop a PCIe-Passthrough NIC before shutting down
>>> that (byhve(8)) guest ? jhb@ helped my identifying this as the root
>>> cause for sever memory corruptions I regularly had (on stable-11).
>>>
>>> Now this time, corruption affected ZFS's RAM area, obviously.
>>>
>>> What I haven't expected is the panic.
>>> The machine has memory disk as root, so luckily I still can boot (from
>>> ZFS, ?> mdpreload rootfs) into single user mode, but early rc stage
>>> (most likely mounting ZFS datasets) leads to the following panic:
>>>
>>> Trying to mount root from ufs:/dev/ufs/cetusROOT []...
>>> panic: Solaris(panic): blkptr at 0xfe0005b6b000 has invalid CHECKSUM 1
>>> cpuid = 1
>>> KDB: stack backtrace:
>>>   [backtrace deleted  --SB]
>>> Haven't done any scrub attempts yet ? expectation is to get all datasets
>>> of the striped mirror pool back...
>>>
>>> Any hints highly appreciated.
>> Now it seems I'm in really big trouble.
>> Regular import doesn't work (also not if booted from cd9660).
>> I get all pools listed, but trying to import (unmounted) leads to the
>> same panic as initialy reported ? because rc is just doning the same.
>>
>> I booted into single user mode (which works since the bootpool isn't
>> affected and root is a memory disk from the bootpool)
>> and set vfs.zfs.recover=1.
>> But this time I don't even get the list of pools to import 'zpool'
>> import instantaniously leads to that panic:
>>
>> Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid CHECKSUM 1
>> Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid COMPRESS 0
>> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 0 has invalid VDEV
>> 2337865727
>> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 1 has invalid VDEV
>> 289407040
>> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 2 has invalid VDEV
>> 3959586324
>>
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 0; apic id = 00
>> fault virtual address   = 0x50
>> fault code  = supervisor read data, page not present
>> instruction pointer = 0x20:0x812de904
>> stack pointer   = 0x28:0xfe043f6bcbc0
>> frame pointer   = 0x28:0xfe043f6bcbc0
>> code segment= base 0x0, limit 0xf, type 0x1b
>> = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags= interrupt enabled, resume, IOPL = 0
>> current process = 44 (zpool)
>> trap number = 12
>> panic: page fault
>> cpuid = 0
>
>?
>
>OpenIndiana also panics at regular import.
>Unfortunately I don't know the aequivalent of vfs.zfs.recover in OI.
>
>panic[cpu1]/thread=ff06dafe8be0: blkptr at ff06dbe63000 has
>invalid CHECKSUM 1
>
>Warning - stack not written to the dump buffer
>ff001f67f070 genunix:vcmn_err+42 ()
>ff001f67f0e0 zfs:zfs_panic_recover+51 ()
>ff001f67f140 zfs:zfs_blkptr_verify+8d ()
>ff001f67f220 zfs:zio_read+55 ()
>ff001f67f310 zfs:arc_read+662 ()
>ff001f67f370 zfs:traverse_prefetch_metadata+b5 ()
>ff001f67f450 zfs:traverse_visitbp+1c3 ()
>ff001f67f4e0 zfs:traverse_dnode+af ()
>ff001f67f5c0 zfs:traverse_visitbp+6dd ()
>ff001f67f720 zfs:traverse_impl+1a6 ()
>ff001f67f830 zfs:traverse_pool+9f ()
>ff001f67f8a0 zfs:spa_load_verify+1e6 ()
>ff001f67f990 zfs:spa_load_impl+e1c ()
>ff001f67fa30 zfs:spa_load+14e ()
>ff001f67fad0 zfs:spa_load_best+7a ()
>ff001f67fb90 zfs:spa_import+1b0 ()
>ff001f67fbe0 zfs:zfs_ioc_pool_import+10f ()
>ff001f67fc80 zfs:zfsdev_ioctl+4b7 ()
>ff001f67fcc0 genunix:cdev_ioctl+39 ()
>ff001f67fd10 specfs:spec_ioctl+60 ()
>ff001f67fda0 genunix:fop_ioctl+55 ()
>ff001f67fec0 genunix:ioctl+9b ()
>ff001f67ff10 unix:brand_sys_sysenter+1c9 ()
>
>This is a important lesson.
>My impression was that it's not possible to corrupt a complete pool, but
>there's always a way to recover healthy/redundant data.
>Now my striped mirror has all 4 devices healthy available, but all
>datasets seem to be lost.
>No problem for 450G (99,9_%), but there's a 80M dataset which I'm really
>missing :-(
>
>Unfortunately I don't know the DVA and blkptr internals, so I won't
>write a zfs_fsck(8) soon ;-)
>
>Does it make sense to dump the disks for further analysis?
>I need to recreate the pool because I need the machine's resources... :-(
>Any help highly appreciated!
>
 First, if it's not too late already, make a copy of the pool's cache file,
and save it somewhere in case you need it unchanged again.
 Can zdb(8) see it without causing a panic, i.e., without importing the
pool?  You might be able to track down more information if zdb can get you in.
 Another thing you could try with an admittedly very low 

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 30.09.2017 19:25 (localtime):
>  Bezüglich Harry Schmalzbauer's Nachricht vom 30.09.2017 18:30 (localtime):
>>  Bad surprise.
>> Most likely I forgot to stop a PCIe-Passthrough NIC before shutting down
>> that (byhve(8)) guest – jhb@ helped my identifying this as the root
>> cause for sever memory corruptions I regularly had (on stable-11).
>>
>> Now this time, corruption affected ZFS's RAM area, obviously.
>>
>> What I haven't expected is the panic.
>> The machine has memory disk as root, so luckily I still can boot (from
>> ZFS, –> mdpreload rootfs) into single user mode, but early rc stage
>> (most likely mounting ZFS datasets) leads to the following panic:
>>
>> Trying to mount root from ufs:/dev/ufs/cetusROOT []...
>> panic: Solaris(panic): blkptr at 0xfe0005b6b000 has invalid CHECKSUM 1
>> cpuid = 1
>> KDB: stack backtrace:
>> #0 0x805e3837 at kdb_backtrace+0x67
>> #1 0x805a2286 at vpanic+0x186
>> #2 0x805a20f3 at panic+0x43
>> #3 0x81570192 at vcmn_err+0xc2
>> #4 0x812d7dda at zfs_panic_recover+0x5a
>> #5 0x812ff49b at zfs_blkptr_verify+0x8b
>> #6 0x812ff72c at zio_read+0x2c
>> #7 0x812761de at arc_read+0x6de
>> #8 0x81298b4d at traverse_prefetch_metadata+0xbd
>> #9 0x812980ed at traverse_visitbp+0x39d
>> #10 0x81298c27 at traverse_dnode+0xc7
>> #11 0x812984a3 at traverse_visitbp+0x753
>> #12 0x8129788b at traverse_impl+0x22b
>> #13 0x81297afc at traverse_pool+0x5c
>> #14 0x812cce06 at spa_load+0x1c06
>> #15 0x812cc302 at spa_load+0x1102
>> #16 0x812cac6e at spa_load_best+0x6e
>> #17 0x812c73a1 at spa_open_common+0x101
>> Uptime: 37s
>> Dumping 1082 out of 15733 MB:..2%..…
>> Dump complete
>> mps0: Sending StopUnit: path (xpt0:mps0:0:2:): handle 12
>> mps0: Incrementing SSU count
>> …
>>
>> Haven't done any scrub attempts yet – expectation is to get all datasets
>> of the striped mirror pool back...
>>
>> Any hints highly appreciated.
> Now it seems I'm in really big trouble.
> Regular import doesn't work (also not if booted from cd9660).
> I get all pools listed, but trying to import (unmounted) leads to the
> same panic as initialy reported – because rc is just doning the same.
>
> I booted into single user mode (which works since the bootpool isn't
> affected and root is a memory disk from the bootpool)
> and set vfs.zfs.recover=1.
> But this time I don't even get the list of pools to import 'zpool'
> import instantaniously leads to that panic:
>
> Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid CHECKSUM 1
> Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid COMPRESS 0
> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 0 has invalid VDEV
> 2337865727
> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 1 has invalid VDEV
> 289407040
> Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 2 has invalid VDEV
> 3959586324
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x50
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x812de904
> stack pointer   = 0x28:0xfe043f6bcbc0
> frame pointer   = 0x28:0xfe043f6bcbc0
> code segment= base 0x0, limit 0xf, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags= interrupt enabled, resume, IOPL = 0
> current process = 44 (zpool)
> trap number = 12
> panic: page fault
> cpuid = 0

…

OpenIndiana also panics at regular import.
Unfortunately I don't know the aequivalent of vfs.zfs.recover in OI.

panic[cpu1]/thread=ff06dafe8be0: blkptr at ff06dbe63000 has
invalid CHECKSUM 1

Warning - stack not written to the dump buffer
ff001f67f070 genunix:vcmn_err+42 ()
ff001f67f0e0 zfs:zfs_panic_recover+51 ()
ff001f67f140 zfs:zfs_blkptr_verify+8d ()
ff001f67f220 zfs:zio_read+55 ()
ff001f67f310 zfs:arc_read+662 ()
ff001f67f370 zfs:traverse_prefetch_metadata+b5 ()
ff001f67f450 zfs:traverse_visitbp+1c3 ()
ff001f67f4e0 zfs:traverse_dnode+af ()
ff001f67f5c0 zfs:traverse_visitbp+6dd ()
ff001f67f720 zfs:traverse_impl+1a6 ()
ff001f67f830 zfs:traverse_pool+9f ()
ff001f67f8a0 zfs:spa_load_verify+1e6 ()
ff001f67f990 zfs:spa_load_impl+e1c ()
ff001f67fa30 zfs:spa_load+14e ()
ff001f67fad0 zfs:spa_load_best+7a ()
ff001f67fb90 zfs:spa_import+1b0 ()
ff001f67fbe0 zfs:zfs_ioc_pool_import+10f ()
ff001f67fc80 zfs:zfsdev_ioctl+4b7 ()
ff001f67fcc0 genunix:cdev_ioctl+39 ()
ff001f67fd10 specfs:spec_ioctl+60 ()
ff001f67fda0 genunix:fop_ioctl+55 ()
ff001f67fec0 genunix:ioctl+9b ()
ff001f67ff10 unix:brand_sys_sysenter+1c9 ()

This is a important lesson.
My impression was that it's not possible to corrupt a complete pool, but
there's always a way to recover 

Re: panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 30.09.2017 18:30 (localtime):
>  Bad surprise.
> Most likely I forgot to stop a PCIe-Passthrough NIC before shutting down
> that (byhve(8)) guest – jhb@ helped my identifying this as the root
> cause for sever memory corruptions I regularly had (on stable-11).
>
> Now this time, corruption affected ZFS's RAM area, obviously.
>
> What I haven't expected is the panic.
> The machine has memory disk as root, so luckily I still can boot (from
> ZFS, –> mdpreload rootfs) into single user mode, but early rc stage
> (most likely mounting ZFS datasets) leads to the following panic:
>
> Trying to mount root from ufs:/dev/ufs/cetusROOT []...
> panic: Solaris(panic): blkptr at 0xfe0005b6b000 has invalid CHECKSUM 1
> cpuid = 1
> KDB: stack backtrace:
> #0 0x805e3837 at kdb_backtrace+0x67
> #1 0x805a2286 at vpanic+0x186
> #2 0x805a20f3 at panic+0x43
> #3 0x81570192 at vcmn_err+0xc2
> #4 0x812d7dda at zfs_panic_recover+0x5a
> #5 0x812ff49b at zfs_blkptr_verify+0x8b
> #6 0x812ff72c at zio_read+0x2c
> #7 0x812761de at arc_read+0x6de
> #8 0x81298b4d at traverse_prefetch_metadata+0xbd
> #9 0x812980ed at traverse_visitbp+0x39d
> #10 0x81298c27 at traverse_dnode+0xc7
> #11 0x812984a3 at traverse_visitbp+0x753
> #12 0x8129788b at traverse_impl+0x22b
> #13 0x81297afc at traverse_pool+0x5c
> #14 0x812cce06 at spa_load+0x1c06
> #15 0x812cc302 at spa_load+0x1102
> #16 0x812cac6e at spa_load_best+0x6e
> #17 0x812c73a1 at spa_open_common+0x101
> Uptime: 37s
> Dumping 1082 out of 15733 MB:..2%..…
> Dump complete
> mps0: Sending StopUnit: path (xpt0:mps0:0:2:): handle 12
> mps0: Incrementing SSU count
> …
>
> Haven't done any scrub attempts yet – expectation is to get all datasets
> of the striped mirror pool back...
>
> Any hints highly appreciated.

Now it seems I'm in really big trouble.
Regular import doesn't work (also not if booted from cd9660).
I get all pools listed, but trying to import (unmounted) leads to the
same panic as initialy reported – because rc is just doning the same.

I booted into single user mode (which works since the bootpool isn't
affected and root is a memory disk from the bootpool)
and set vfs.zfs.recover=1.
But this time I don't even get the list of pools to import 'zpool'
import instantaniously leads to that panic:

Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid CHECKSUM 1
Solaris: WARNING: blkptr at 0xfe0005a8e000 has invalid COMPRESS 0
Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 0 has invalid VDEV
2337865727
Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 1 has invalid VDEV
289407040
Solaris: WARNING: blkptr at 0xfe0005a8e000 DVA 2 has invalid VDEV
3959586324


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x50
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x812de904
stack pointer   = 0x28:0xfe043f6bcbc0
frame pointer   = 0x28:0xfe043f6bcbc0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 44 (zpool)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0x805e3837 at kdb_backtrace+0x67
#1 0x805a2286 at vpanic+0x186
#2 0x805a20f3 at panic+0x43
#3 0x808a4922 at trap_fatal+0x322
#4 0x808a4979 at trap_pfault+0x49
#5 0x808a41f8 at trap+0x298
#6 0x80889fb1 at calltrap+0x8
#7 0x812e58a3 at vdev_mirror_child_select+0x53
#8 0x812e535e at vdev_mirror_io_start+0x2ee
#9 0x81303aa1 at zio_vdev_io_start+0x161
#10 0x8130054c at zio_execute+0xac
#11 0x812ffe7b at zio_nowait+0xcb
#12 0x812761f3 at arc_read+0x6f3
#13 0x81298b4d at traverse_prefetch_metadata+0xbd
#14 0x812980ed at traverse_visitbp+0x39d
#15 0x81298c27 at traverse_dnode+0xc7
#16 0x812984a3 at traverse_visitbp+0x753
#17 0x8129788b at traverse_impl+0x22b

Now I hope any ZFS guru can help me out. Needless to mention that the
bits on this mirrored pool are important for me – no productive data,
but lots of intermediate...

Thanks,

-harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

panic: Solaris(panic): blkptr invalid CHECKSUM1

[Harry Schmalzbauer]

 Bad surprise.
Most likely I forgot to stop a PCIe-Passthrough NIC before shutting down
that (byhve(8)) guest – jhb@ helped my identifying this as the root
cause for sever memory corruptions I regularly had (on stable-11).

Now this time, corruption affected ZFS's RAM area, obviously.

What I haven't expected is the panic.
The machine has memory disk as root, so luckily I still can boot (from
ZFS, –> mdpreload rootfs) into single user mode, but early rc stage
(most likely mounting ZFS datasets) leads to the following panic:

Trying to mount root from ufs:/dev/ufs/cetusROOT []...
panic: Solaris(panic): blkptr at 0xfe0005b6b000 has invalid CHECKSUM 1
cpuid = 1
KDB: stack backtrace:
#0 0x805e3837 at kdb_backtrace+0x67
#1 0x805a2286 at vpanic+0x186
#2 0x805a20f3 at panic+0x43
#3 0x81570192 at vcmn_err+0xc2
#4 0x812d7dda at zfs_panic_recover+0x5a
#5 0x812ff49b at zfs_blkptr_verify+0x8b
#6 0x812ff72c at zio_read+0x2c
#7 0x812761de at arc_read+0x6de
#8 0x81298b4d at traverse_prefetch_metadata+0xbd
#9 0x812980ed at traverse_visitbp+0x39d
#10 0x81298c27 at traverse_dnode+0xc7
#11 0x812984a3 at traverse_visitbp+0x753
#12 0x8129788b at traverse_impl+0x22b
#13 0x81297afc at traverse_pool+0x5c
#14 0x812cce06 at spa_load+0x1c06
#15 0x812cc302 at spa_load+0x1102
#16 0x812cac6e at spa_load_best+0x6e
#17 0x812c73a1 at spa_open_common+0x101
Uptime: 37s
Dumping 1082 out of 15733 MB:..2%..…
Dump complete
mps0: Sending StopUnit: path (xpt0:mps0:0:2:): handle 12
mps0: Incrementing SSU count
…

Haven't done any scrub attempts yet – expectation is to get all datasets
of the striped mirror pool back...

Any hints highly appreciated.

-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"