Hello!
I have (remote) physical server with 2 SATA disks. These disks were partitioned with GPT into
"freebsd-boot" (ada{0|1}p1, legacy one, not EFI), "freebsd-swap" (ada{0|1}p2) and
"freebsd-zfs" (ada{0|1}p3).
Both disks were 512/512 (it looks important).
I have only one ZFS pool "zroot", mirror of "ada0p3" and "ada1p3".
I have very fresh "gptzfsboot" on both "ada0p1" and "ada1p1".
Now, ada0 failed. It was replaced by DC support with new disk, which is
512/4096.
After that my server fails to boot, gtpzfsboot from second disk (ada1) reports several
"zio_read error: 5" and
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool zroot
after that.
I've booted to rescue Linux (unfortunately, there is NO rescue FreeBSD at
Hetzner anymore), and Linux could import (degraded) pool no problem. But Linux
has problems with detecting pool on partition, so I don't do nothing under
Linux.
I've checked "live" disk under Linux, though: it reads, SMART is clear,
everything is Ok.
I've booted FreeBSD 13.2 from installation ISO under qemu with physical
devices as disks. Then I partitioned fresh HDD and started disk replacement in
mirror. It worked, but resilver was unbearable slow. I stopped VM with FreeBSD
to continue process after normal boot.
NO LUCK. "zio_read error: 5", boot failed.
Then I've overwrite ada0 (new disk) with FreeBSD memstick IMG and boot it -
it can import pool from ada1p3 but, of course, resilver is stopped.
I've removed all faulted components, effectivly converting mirror to "simple" device.
But "zpool status" shows that there is resilver!
And "gptzfsboot" still CAN NOT read this ZFS pool and find loader!
Ok, I've converted swap to UFS boot form UFS. It works. It can use pool as root. But
pool still is "reslivering".
Now I have very strange situation:
(1) I have ZFS pool with 1 device which says:
% zpool status -v zroot
pool: zroot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jan 5 19:24:07 2024
750G scanned at 472B/s, 40.5G issued at 25B/s, 974G total
0B resilvered, 4.16% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
errors: No known data errors
%
(2) gtpzfsboot from very this system version can not read this pool and bot
from it
(3) kernel can use this pool as source of root (and all other) filesystems.
--
// Lev Serebryakov
