Re: [developer] DRAID impressions
Hi Richard, a very long time indeed! I was thinking more of the accumulated allocation (not instantaneous IOPS). The pool was created with both DRAID vdevs from the get-go. We had 32x 12TB and 12x 14TB drives to repurpose, and wanted to build two identical pools. Our other two servers have 16x new 18TB. Interestingly, (and more by luck than design) the usable space across all four servers is the same. In this pool, both vdevs use a 3d+2p RAID scheme. But whereas draid2:3d:16c:1s-0 has three 3d+2p, plus one spare (16x 12TB), draid2:3d:6c:1s-1 has just one 3d+2p, plus one spare (6x 14TB drives). The disappointing thing is the lop-sided allocation, such that draid2:3d:6c:1s-1 has 50.6T allocated and 3.96T free (i.e. 92% allocated), whereas raid2:3d:16c:1s-0 only has 128T allocated, and 63.1T free (i.e. about 50% allocated). Phil On Tue, 6 Dec 2022 at 13:10, Richard Elling < richard.ell...@richardelling.com> wrote: > Hi Phil, long time, no see. comments embedded below > > > On Tue, Dec 6, 2022 at 3:35 AM Phil Harman wrote: > >> I have a number of "ZFS backup" servers (about 1PB split between four >> machines). >> >> Some of them have 16x 18TB drives, but a couple have a mix of 12TB and >> 14TB drives (because that's what we had). >> >> All are running Ubuntu 20.04 LTS (with a snapshot build) or 22.04 LTS >> (bundled). >> >> We're just doing our first replace (actually swapping in a 16TB drive for >> a 12TB because that's all we have). >> >> root@hqs1:~# zpool version >> zfs-2.1.99-784_gae07fc139 >> zfs-kmod-2.1.99-784_gae07fc139 >> root@hqs1:~# zpool status >> pool: hqs1p1 >> state: DEGRADED >> status: One or more devices is currently being resilvered. The pool will >> continue to function, possibly in a degraded state. >> action: Wait for the resilver to complete. >> scan: resilver in progress since Mon Dec 5 15:19:31 2022 >> 177T scanned at 2.49G/s, 175T issued at 2.46G/s, 177T total >> 7.95T resilvered, 98.97% done, 00:12:39 to go >> config: >> >> NAME STATE READ WRITE CKSUM >> hqs1p1DEGRADED 0 0 0 >> draid2:3d:16c:1s-0 DEGRADED 0 0 0 >>780530e1-d2e4-0040-aa8b-8c7bed75a14a ONLINE 0 0 0 >> (resilvering) >>9c4428e8-d16f-3849-97d9-22fc441750dc ONLINE 0 0 0 >> (resilvering) >>0e148b1d-69a3-3345-9478-343ecf6b855d ONLINE 0 0 0 >> (resilvering) >>98208ffe-4b31-564f-832d-5744c809f163 ONLINE 0 0 0 >> (resilvering) >>3ac46b0a-9c46-e14f-8137-69227f3a890a ONLINE 0 0 0 >> (resilvering) >>44e8f62f-5d49-c345-9c89-ac82926d42b7 ONLINE 0 0 0 >> (resilvering) >>968dbacd-1d85-0b40-a1fc-977a09ac5aaa ONLINE 0 0 0 >> (resilvering) >>e7ca2666-1067-f54c-b723-b464fb0a5fa3 ONLINE 0 0 0 >> (resilvering) >>318ff075-8860-e84e-8063-f5f57a2d ONLINE 0 0 0 >> (resilvering) >>replacing-9 DEGRADED 0 0 0 >> 2888151727045752617 UNAVAIL 0 0 0 was >> /dev/disk/by-partuuid/85fa9347-8359-4942-a20d-da1f6016ea48 >> sdd ONLINE 0 0 0 >> (resilvering) >>fd69f284-d05d-f145-9bdb-0da8a72bf311 ONLINE 0 0 0 >> (resilvering) >>f40f997a-33a1-2a4e-bb8d-64223c441f0f ONLINE 0 0 0 >> (resilvering) >>dbc35ea9-95d1-bd40-b79e-90d8a37079a6 ONLINE 0 0 0 >> (resilvering) >>ac62bf3e-517e-a444-ae4f-a784b81cd14c ONLINE 0 0 0 >> (resilvering) >>d211031c-54d4-2443-853c-7e5c075b28ab ONLINE 0 0 0 >> (resilvering) >>06ba16e5-05cf-9b45-a267-510bfe98ceb1 ONLINE 0 0 0 >> (resilvering) >> draid2:3d:6c:1s-1 ONLINE 0 0 0 >>be297802-095c-7d43-9132-360627ba8ceb ONLINE 0 0 0 >>e849981c-7316-cb47-b926-61d444790518 ONLINE 0 0 0 >>bbc6d66d-38e1-c448-9d00-10ba7adcd371 ONLINE 0 0 0 >>9fb44c95-5ea6-2347-ae97-38de283f45bf ONLINE 0 0 0 >>b212cae5-5068-8740-b120-0618ad459c1f ONLINE 0 0 0 >>8c771f6b-7d48-e744-9e25-847230fd2fdd ONLINE 0 0 0 >> spares >> draid2-0-0 AVAIL >> draid2-1-0 AVAIL >> >> errors: No known data errors >> root@hqs1:~# >> >> Here's a snippet of zpool iostat -v 1 ... >> >> capacity operations >> bandwidth >> pool alloc free read write >> read write >> - - - - >> - - >> hqs1p1 178T 67.1T 5.68K259 >> 585M 144M >> draid2:3d:16c:1s-0 128T 63.1T 5.66K259 >> 585M 144M >>
Re: [developer] DRAID impressions
Hi Phil, long time, no see. comments embedded below On Tue, Dec 6, 2022 at 3:35 AM Phil Harman wrote: > I have a number of "ZFS backup" servers (about 1PB split between four > machines). > > Some of them have 16x 18TB drives, but a couple have a mix of 12TB and > 14TB drives (because that's what we had). > > All are running Ubuntu 20.04 LTS (with a snapshot build) or 22.04 LTS > (bundled). > > We're just doing our first replace (actually swapping in a 16TB drive for > a 12TB because that's all we have). > > root@hqs1:~# zpool version > zfs-2.1.99-784_gae07fc139 > zfs-kmod-2.1.99-784_gae07fc139 > root@hqs1:~# zpool status > pool: hqs1p1 > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scan: resilver in progress since Mon Dec 5 15:19:31 2022 > 177T scanned at 2.49G/s, 175T issued at 2.46G/s, 177T total > 7.95T resilvered, 98.97% done, 00:12:39 to go > config: > > NAME STATE READ WRITE CKSUM > hqs1p1DEGRADED 0 0 0 > draid2:3d:16c:1s-0 DEGRADED 0 0 0 >780530e1-d2e4-0040-aa8b-8c7bed75a14a ONLINE 0 0 0 > (resilvering) >9c4428e8-d16f-3849-97d9-22fc441750dc ONLINE 0 0 0 > (resilvering) >0e148b1d-69a3-3345-9478-343ecf6b855d ONLINE 0 0 0 > (resilvering) >98208ffe-4b31-564f-832d-5744c809f163 ONLINE 0 0 0 > (resilvering) >3ac46b0a-9c46-e14f-8137-69227f3a890a ONLINE 0 0 0 > (resilvering) >44e8f62f-5d49-c345-9c89-ac82926d42b7 ONLINE 0 0 0 > (resilvering) >968dbacd-1d85-0b40-a1fc-977a09ac5aaa ONLINE 0 0 0 > (resilvering) >e7ca2666-1067-f54c-b723-b464fb0a5fa3 ONLINE 0 0 0 > (resilvering) >318ff075-8860-e84e-8063-f5f57a2d ONLINE 0 0 0 > (resilvering) >replacing-9 DEGRADED 0 0 0 > 2888151727045752617 UNAVAIL 0 0 0 was > /dev/disk/by-partuuid/85fa9347-8359-4942-a20d-da1f6016ea48 > sdd ONLINE 0 0 0 > (resilvering) >fd69f284-d05d-f145-9bdb-0da8a72bf311 ONLINE 0 0 0 > (resilvering) >f40f997a-33a1-2a4e-bb8d-64223c441f0f ONLINE 0 0 0 > (resilvering) >dbc35ea9-95d1-bd40-b79e-90d8a37079a6 ONLINE 0 0 0 > (resilvering) >ac62bf3e-517e-a444-ae4f-a784b81cd14c ONLINE 0 0 0 > (resilvering) >d211031c-54d4-2443-853c-7e5c075b28ab ONLINE 0 0 0 > (resilvering) >06ba16e5-05cf-9b45-a267-510bfe98ceb1 ONLINE 0 0 0 > (resilvering) > draid2:3d:6c:1s-1 ONLINE 0 0 0 >be297802-095c-7d43-9132-360627ba8ceb ONLINE 0 0 0 >e849981c-7316-cb47-b926-61d444790518 ONLINE 0 0 0 >bbc6d66d-38e1-c448-9d00-10ba7adcd371 ONLINE 0 0 0 >9fb44c95-5ea6-2347-ae97-38de283f45bf ONLINE 0 0 0 >b212cae5-5068-8740-b120-0618ad459c1f ONLINE 0 0 0 >8c771f6b-7d48-e744-9e25-847230fd2fdd ONLINE 0 0 0 > spares > draid2-0-0 AVAIL > draid2-1-0 AVAIL > > errors: No known data errors > root@hqs1:~# > > Here's a snippet of zpool iostat -v 1 ... > > capacity operations > bandwidth > pool alloc free read write > read write > - - - - > - - > hqs1p1 178T 67.1T 5.68K259 > 585M 144M > draid2:3d:16c:1s-0 128T 63.1T 5.66K259 > 585M 144M > 780530e1-d2e4-0040-aa8b-8c7bed75a14a - -208 0 > 25.4M 0 > 9c4428e8-d16f-3849-97d9-22fc441750dc - - 1010 2 > 102M 23.7K > 0e148b1d-69a3-3345-9478-343ecf6b855d - -145 0 > 34.1M 15.8K > 98208ffe-4b31-564f-832d-5744c809f163 - -101 1 > 28.3M 7.90K > 3ac46b0a-9c46-e14f-8137-69227f3a890a - -511 0 > 53.5M 0 > 44e8f62f-5d49-c345-9c89-ac82926d42b7 - - 12 0 > 4.82M 0 > 968dbacd-1d85-0b40-a1fc-977a09ac5aaa - - 22 0 > 5.43M 15.8K > e7ca2666-1067-f54c-b723-b464fb0a5fa3 - -227 2 > 36.7M 23.7K > 318ff075-8860-e84e-8063-f5f57a2d - -999 1 > 83.1M 7.90K > replacing-9 - - 0243 > 0 144M > 2888151727045752617 - - 0 0 > 0 0 > sdd - - 0243 > 0 144M >
[developer] DRAID impressions
I have a number of "ZFS backup" servers (about 1PB split between four machines). Some of them have 16x 18TB drives, but a couple have a mix of 12TB and 14TB drives (because that's what we had). All are running Ubuntu 20.04 LTS (with a snapshot build) or 22.04 LTS (bundled). We're just doing our first replace (actually swapping in a 16TB drive for a 12TB because that's all we have). root@hqs1:~# zpool version zfs-2.1.99-784_gae07fc139 zfs-kmod-2.1.99-784_gae07fc139 root@hqs1:~# zpool status pool: hqs1p1 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Mon Dec 5 15:19:31 2022 177T scanned at 2.49G/s, 175T issued at 2.46G/s, 177T total 7.95T resilvered, 98.97% done, 00:12:39 to go config: NAME STATE READ WRITE CKSUM hqs1p1DEGRADED 0 0 0 draid2:3d:16c:1s-0 DEGRADED 0 0 0 780530e1-d2e4-0040-aa8b-8c7bed75a14a ONLINE 0 0 0 (resilvering) 9c4428e8-d16f-3849-97d9-22fc441750dc ONLINE 0 0 0 (resilvering) 0e148b1d-69a3-3345-9478-343ecf6b855d ONLINE 0 0 0 (resilvering) 98208ffe-4b31-564f-832d-5744c809f163 ONLINE 0 0 0 (resilvering) 3ac46b0a-9c46-e14f-8137-69227f3a890a ONLINE 0 0 0 (resilvering) 44e8f62f-5d49-c345-9c89-ac82926d42b7 ONLINE 0 0 0 (resilvering) 968dbacd-1d85-0b40-a1fc-977a09ac5aaa ONLINE 0 0 0 (resilvering) e7ca2666-1067-f54c-b723-b464fb0a5fa3 ONLINE 0 0 0 (resilvering) 318ff075-8860-e84e-8063-f5f57a2d ONLINE 0 0 0 (resilvering) replacing-9 DEGRADED 0 0 0 2888151727045752617 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/85fa9347-8359-4942-a20d-da1f6016ea48 sdd ONLINE 0 0 0 (resilvering) fd69f284-d05d-f145-9bdb-0da8a72bf311 ONLINE 0 0 0 (resilvering) f40f997a-33a1-2a4e-bb8d-64223c441f0f ONLINE 0 0 0 (resilvering) dbc35ea9-95d1-bd40-b79e-90d8a37079a6 ONLINE 0 0 0 (resilvering) ac62bf3e-517e-a444-ae4f-a784b81cd14c ONLINE 0 0 0 (resilvering) d211031c-54d4-2443-853c-7e5c075b28ab ONLINE 0 0 0 (resilvering) 06ba16e5-05cf-9b45-a267-510bfe98ceb1 ONLINE 0 0 0 (resilvering) draid2:3d:6c:1s-1 ONLINE 0 0 0 be297802-095c-7d43-9132-360627ba8ceb ONLINE 0 0 0 e849981c-7316-cb47-b926-61d444790518 ONLINE 0 0 0 bbc6d66d-38e1-c448-9d00-10ba7adcd371 ONLINE 0 0 0 9fb44c95-5ea6-2347-ae97-38de283f45bf ONLINE 0 0 0 b212cae5-5068-8740-b120-0618ad459c1f ONLINE 0 0 0 8c771f6b-7d48-e744-9e25-847230fd2fdd ONLINE 0 0 0 spares draid2-0-0 AVAIL draid2-1-0 AVAIL errors: No known data errors root@hqs1:~# Here's a snippet of zpool iostat -v 1 ... capacity operations bandwidth pool alloc free read write read write - - - - - - hqs1p1 178T 67.1T 5.68K259 585M 144M draid2:3d:16c:1s-0 128T 63.1T 5.66K259 585M 144M 780530e1-d2e4-0040-aa8b-8c7bed75a14a - -208 0 25.4M 0 9c4428e8-d16f-3849-97d9-22fc441750dc - - 1010 2 102M 23.7K 0e148b1d-69a3-3345-9478-343ecf6b855d - -145 0 34.1M 15.8K 98208ffe-4b31-564f-832d-5744c809f163 - -101 1 28.3M 7.90K 3ac46b0a-9c46-e14f-8137-69227f3a890a - -511 0 53.5M 0 44e8f62f-5d49-c345-9c89-ac82926d42b7 - - 12 0 4.82M 0 968dbacd-1d85-0b40-a1fc-977a09ac5aaa - - 22 0 5.43M 15.8K e7ca2666-1067-f54c-b723-b464fb0a5fa3 - -227 2 36.7M 23.7K 318ff075-8860-e84e-8063-f5f57a2d - -999 1 83.1M 7.90K replacing-9 - - 0243 0 144M 2888151727045752617 - - 0 0 0 0 sdd - - 0243 0 144M fd69f284-d05d-f145-9bdb-0da8a72bf311 - -306 0 54.5M 15.8K f40f997a-33a1-2a4e-bb8d-64223c441f0f - - 47 0 16.9M 0 dbc35ea9-95d1-bd40-b79e-90d8a37079a6 - -234 0 16.7M 15.8K ac62bf3e-517e-a444-ae4f-a784b81cd14c - -417 0
