I have a number of "ZFS backup" servers (about 1PB split between four
machines).
Some of them have 16x 18TB drives, but a couple have a mix of 12TB and 14TB
drives (because that's what we had).
All are running Ubuntu 20.04 LTS (with a snapshot build) or 22.04 LTS
(bundled).
We're just doing our first replace (actually swapping in a 16TB drive for a
12TB because that's all we have).
root@hqs1:~# zpool version
zfs-2.1.99-784_gae07fc139
zfs-kmod-2.1.99-784_gae07fc139
root@hqs1:~# zpool status
pool: hqs1p1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Dec 5 15:19:31 2022
177T scanned at 2.49G/s, 175T issued at 2.46G/s, 177T total
7.95T resilvered, 98.97% done, 00:12:39 to go
config:
NAME STATE READ WRITE CKSUM
hqs1p1 DEGRADED 0 0 0
draid2:3d:16c:1s-0 DEGRADED 0 0 0
780530e1-d2e4-0040-aa8b-8c7bed75a14a ONLINE 0 0 0
(resilvering)
9c4428e8-d16f-3849-97d9-22fc441750dc ONLINE 0 0 0
(resilvering)
0e148b1d-69a3-3345-9478-343ecf6b855d ONLINE 0 0 0
(resilvering)
98208ffe-4b31-564f-832d-5744c809f163 ONLINE 0 0 0
(resilvering)
3ac46b0a-9c46-e14f-8137-69227f3a890a ONLINE 0 0 0
(resilvering)
44e8f62f-5d49-c345-9c89-ac82926d42b7 ONLINE 0 0 0
(resilvering)
968dbacd-1d85-0b40-a1fc-977a09ac5aaa ONLINE 0 0 0
(resilvering)
e7ca2666-1067-f54c-b723-b464fb0a5fa3 ONLINE 0 0 0
(resilvering)
318ff075-8860-e84e-8063-f77775f57a2d ONLINE 0 0 0
(resilvering)
replacing-9 DEGRADED 0 0 0
2888151727045752617 UNAVAIL 0 0 0 was
/dev/disk/by-partuuid/85fa9347-8359-4942-a20d-da1f6016ea48
sdd ONLINE 0 0 0
(resilvering)
fd69f284-d05d-f145-9bdb-0da8a72bf311 ONLINE 0 0 0
(resilvering)
f40f997a-33a1-2a4e-bb8d-64223c441f0f ONLINE 0 0 0
(resilvering)
dbc35ea9-95d1-bd40-b79e-90d8a37079a6 ONLINE 0 0 0
(resilvering)
ac62bf3e-517e-a444-ae4f-a784b81cd14c ONLINE 0 0 0
(resilvering)
d211031c-54d4-2443-853c-7e5c075b28ab ONLINE 0 0 0
(resilvering)
06ba16e5-05cf-9b45-a267-510bfe98ceb1 ONLINE 0 0 0
(resilvering)
draid2:3d:6c:1s-1 ONLINE 0 0 0
be297802-095c-7d43-9132-360627ba8ceb ONLINE 0 0 0
e849981c-7316-cb47-b926-61d444790518 ONLINE 0 0 0
bbc6d66d-38e1-c448-9d00-10ba7adcd371 ONLINE 0 0 0
9fb44c95-5ea6-2347-ae97-38de283f45bf ONLINE 0 0 0
b212cae5-5068-8740-b120-0618ad459c1f ONLINE 0 0 0
8c771f6b-7d48-e744-9e25-847230fd2fdd ONLINE 0 0 0
spares
draid2-0-0 AVAIL
draid2-1-0 AVAIL
errors: No known data errors
root@hqs1:~#
Here's a snippet of zpool iostat -v 1 ...
capacity operations
bandwidth
pool alloc free read write read
write
---------------------------------------- ----- ----- ----- ----- -----
-----
hqs1p1 178T 67.1T 5.68K 259 585M
144M
draid2:3d:16c:1s-0 128T 63.1T 5.66K 259 585M
144M
780530e1-d2e4-0040-aa8b-8c7bed75a14a - - 208 0 25.4M
0
9c4428e8-d16f-3849-97d9-22fc441750dc - - 1010 2 102M
23.7K
0e148b1d-69a3-3345-9478-343ecf6b855d - - 145 0 34.1M
15.8K
98208ffe-4b31-564f-832d-5744c809f163 - - 101 1 28.3M
7.90K
3ac46b0a-9c46-e14f-8137-69227f3a890a - - 511 0 53.5M
0
44e8f62f-5d49-c345-9c89-ac82926d42b7 - - 12 0 4.82M
0
968dbacd-1d85-0b40-a1fc-977a09ac5aaa - - 22 0 5.43M
15.8K
e7ca2666-1067-f54c-b723-b464fb0a5fa3 - - 227 2 36.7M
23.7K
318ff075-8860-e84e-8063-f77775f57a2d - - 999 1 83.1M
7.90K
replacing-9 - - 0 243 0
144M
2888151727045752617 - - 0 0 0
0
sdd - - 0 243 0
144M
fd69f284-d05d-f145-9bdb-0da8a72bf311 - - 306 0 54.5M
15.8K
f40f997a-33a1-2a4e-bb8d-64223c441f0f - - 47 0 16.9M
0
dbc35ea9-95d1-bd40-b79e-90d8a37079a6 - - 234 0 16.7M
15.8K
ac62bf3e-517e-a444-ae4f-a784b81cd14c - - 417 0 15.9M
0
d211031c-54d4-2443-853c-7e5c075b28ab - - 911 0 48.4M
15.8K
06ba16e5-05cf-9b45-a267-510bfe98ceb1 - - 643 0 60.1M
15.8K
draid2:3d:6c:1s-1 50.6T 3.96T 19 0 198K
0
be297802-095c-7d43-9132-360627ba8ceb - - 3 0 39.5K
0
e849981c-7316-cb47-b926-61d444790518 - - 2 0 23.7K
0
bbc6d66d-38e1-c448-9d00-10ba7adcd371 - - 2 0 23.7K
0
9fb44c95-5ea6-2347-ae97-38de283f45bf - - 3 0 39.5K
0
b212cae5-5068-8740-b120-0618ad459c1f - - 3 0 39.5K
0
8c771f6b-7d48-e744-9e25-847230fd2fdd - - 1 0 31.6K
0
---------------------------------------- ----- ----- ----- ----- -----
-----
Lots of DRAID goodness there. Seems to be resilvering a q good whack.
My main question is: why have the DRAID vdevs been so disproportionately
allocated?
------------------------------------------
openzfs: openzfs-developer
Permalink:
https://openzfs.topicbox.com/groups/developer/T386dac0e170785f1-Ma99e75ef1c6e8ee3074003a0
Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
