Hi Phil, long time, no see. comments embedded below
On Tue, Dec 6, 2022 at 3:35 AM Phil Harman <phil.har...@gmail.com> wrote: > I have a number of "ZFS backup" servers (about 1PB split between four > machines). > > Some of them have 16x 18TB drives, but a couple have a mix of 12TB and > 14TB drives (because that's what we had). > > All are running Ubuntu 20.04 LTS (with a snapshot build) or 22.04 LTS > (bundled). > > We're just doing our first replace (actually swapping in a 16TB drive for > a 12TB because that's all we have). > > root@hqs1:~# zpool version > zfs-2.1.99-784_gae07fc139 > zfs-kmod-2.1.99-784_gae07fc139 > root@hqs1:~# zpool status > pool: hqs1p1 > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Mon Dec 5 15:19:31 2022 > 177T scanned at 2.49G/s, 175T issued at 2.46G/s, 177T total > 7.95T resilvered, 98.97% done, 00:12:39 to go > config: > > NAME STATE READ WRITE CKSUM > hqs1p1 DEGRADED 0 0 0 > draid2:3d:16c:1s-0 DEGRADED 0 0 0 > 780530e1-d2e4-0040-aa8b-8c7bed75a14a ONLINE 0 0 0 > (resilvering) > 9c4428e8-d16f-3849-97d9-22fc441750dc ONLINE 0 0 0 > (resilvering) > 0e148b1d-69a3-3345-9478-343ecf6b855d ONLINE 0 0 0 > (resilvering) > 98208ffe-4b31-564f-832d-5744c809f163 ONLINE 0 0 0 > (resilvering) > 3ac46b0a-9c46-e14f-8137-69227f3a890a ONLINE 0 0 0 > (resilvering) > 44e8f62f-5d49-c345-9c89-ac82926d42b7 ONLINE 0 0 0 > (resilvering) > 968dbacd-1d85-0b40-a1fc-977a09ac5aaa ONLINE 0 0 0 > (resilvering) > e7ca2666-1067-f54c-b723-b464fb0a5fa3 ONLINE 0 0 0 > (resilvering) > 318ff075-8860-e84e-8063-f77775f57a2d ONLINE 0 0 0 > (resilvering) > replacing-9 DEGRADED 0 0 0 > 2888151727045752617 UNAVAIL 0 0 0 was > /dev/disk/by-partuuid/85fa9347-8359-4942-a20d-da1f6016ea48 > sdd ONLINE 0 0 0 > (resilvering) > fd69f284-d05d-f145-9bdb-0da8a72bf311 ONLINE 0 0 0 > (resilvering) > f40f997a-33a1-2a4e-bb8d-64223c441f0f ONLINE 0 0 0 > (resilvering) > dbc35ea9-95d1-bd40-b79e-90d8a37079a6 ONLINE 0 0 0 > (resilvering) > ac62bf3e-517e-a444-ae4f-a784b81cd14c ONLINE 0 0 0 > (resilvering) > d211031c-54d4-2443-853c-7e5c075b28ab ONLINE 0 0 0 > (resilvering) > 06ba16e5-05cf-9b45-a267-510bfe98ceb1 ONLINE 0 0 0 > (resilvering) > draid2:3d:6c:1s-1 ONLINE 0 0 0 > be297802-095c-7d43-9132-360627ba8ceb ONLINE 0 0 0 > e849981c-7316-cb47-b926-61d444790518 ONLINE 0 0 0 > bbc6d66d-38e1-c448-9d00-10ba7adcd371 ONLINE 0 0 0 > 9fb44c95-5ea6-2347-ae97-38de283f45bf ONLINE 0 0 0 > b212cae5-5068-8740-b120-0618ad459c1f ONLINE 0 0 0 > 8c771f6b-7d48-e744-9e25-847230fd2fdd ONLINE 0 0 0 > spares > draid2-0-0 AVAIL > draid2-1-0 AVAIL > > errors: No known data errors > root@hqs1:~# > > Here's a snippet of zpool iostat -v 1 ... > > capacity operations > bandwidth > pool alloc free read write > read write > ---------------------------------------- ----- ----- ----- ----- > ----- ----- > hqs1p1 178T 67.1T 5.68K 259 > 585M 144M > draid2:3d:16c:1s-0 128T 63.1T 5.66K 259 > 585M 144M > 780530e1-d2e4-0040-aa8b-8c7bed75a14a - - 208 0 > 25.4M 0 > 9c4428e8-d16f-3849-97d9-22fc441750dc - - 1010 2 > 102M 23.7K > 0e148b1d-69a3-3345-9478-343ecf6b855d - - 145 0 > 34.1M 15.8K > 98208ffe-4b31-564f-832d-5744c809f163 - - 101 1 > 28.3M 7.90K > 3ac46b0a-9c46-e14f-8137-69227f3a890a - - 511 0 > 53.5M 0 > 44e8f62f-5d49-c345-9c89-ac82926d42b7 - - 12 0 > 4.82M 0 > 968dbacd-1d85-0b40-a1fc-977a09ac5aaa - - 22 0 > 5.43M 15.8K > e7ca2666-1067-f54c-b723-b464fb0a5fa3 - - 227 2 > 36.7M 23.7K > 318ff075-8860-e84e-8063-f77775f57a2d - - 999 1 > 83.1M 7.90K > replacing-9 - - 0 243 > 0 144M > 2888151727045752617 - - 0 0 > 0 0 > sdd - - 0 243 > 0 144M > fd69f284-d05d-f145-9bdb-0da8a72bf311 - - 306 0 > 54.5M 15.8K > f40f997a-33a1-2a4e-bb8d-64223c441f0f - - 47 0 > 16.9M 0 > dbc35ea9-95d1-bd40-b79e-90d8a37079a6 - - 234 0 > 16.7M 15.8K > ac62bf3e-517e-a444-ae4f-a784b81cd14c - - 417 0 > 15.9M 0 > d211031c-54d4-2443-853c-7e5c075b28ab - - 911 0 > 48.4M 15.8K > 06ba16e5-05cf-9b45-a267-510bfe98ceb1 - - 643 0 > 60.1M 15.8K > draid2:3d:6c:1s-1 50.6T 3.96T 19 0 > 198K 0 > be297802-095c-7d43-9132-360627ba8ceb - - 3 0 > 39.5K 0 > e849981c-7316-cb47-b926-61d444790518 - - 2 0 > 23.7K 0 > bbc6d66d-38e1-c448-9d00-10ba7adcd371 - - 2 0 > 23.7K 0 > 9fb44c95-5ea6-2347-ae97-38de283f45bf - - 3 0 > 39.5K 0 > b212cae5-5068-8740-b120-0618ad459c1f - - 3 0 > 39.5K 0 > 8c771f6b-7d48-e744-9e25-847230fd2fdd - - 1 0 > 31.6K 0 > ---------------------------------------- ----- ----- ----- ----- > ----- ----- > > Lots of DRAID goodness there. Seems to be resilvering a q good whack. > > My main question is: why have the DRAID vdevs been so disproportionately > allocated? > When doing a replace to disk, which is not the same as replace to logical spare, then the disk's write performance is the bottleneck. The draid rebuild time improvements only apply when rebuilding onto a logical spare. The data should be spread about nicely, but I don't think you'll see that with a short time interval. Over perhaps 100 seconds or so, it should look more randomly spread. -- richard > > *openzfs <https://openzfs.topicbox.com/latest>* / openzfs-developer / see > discussions <https://openzfs.topicbox.com/groups/developer> + participants > <https://openzfs.topicbox.com/groups/developer/members> + delivery options > <https://openzfs.topicbox.com/groups/developer/subscription> Permalink > <https://openzfs.topicbox.com/groups/developer/T386dac0e170785f1-Ma99e75ef1c6e8ee3074003a0> > ------------------------------------------ openzfs: openzfs-developer Permalink: https://openzfs.topicbox.com/groups/developer/T386dac0e170785f1-Md3fbc777aeb31f7281232275 Delivery options: https://openzfs.topicbox.com/groups/developer/subscription
