So great to see it in action! Thanks gea for the benchmark! Looks like your benchmark is done with OmniOS, so SSD+HDD mirror is really a bad idea if on Illumos ZFS up to your OmniOS' version.
I can understand the benchmark result except for sole SSD randomrw.f be at 72.8 MB/s (vs 118.2 MB/s), it's rather much slower than sole HD! Why? I don't understand but a wild guess - somehow the cache works in write-through mode on SSD only vdev, while on HD involved vdev, the cache works in write-back mode? -- Then I googled more and found that OpenZFS people have tackled almost identical situation with great progress: https://github.com/openzfs/zfs/issues/1461#issuecomment-18399909 <https://github.com/openzfs/zfs/issues/1461#issuecomment-18399909> > For example in my tests with 1 x disk and 1 x ram device: > > Normal round robin: 80MB/s+80MB/s = 160MB/sec > Balancing off pending: 80MB/s + 200MB/s = 280MB/s > With slow vdevs offlined: 0MB/s + 400MB/s = 400MB/s https://github.com/openzfs/zfs/issues/1461#issuecomment-18783825 <https://github.com/openzfs/zfs/issues/1461#issuecomment-18783825> > Just for reference here's the patch I've proposed running against a ssd+hdd > pool. Because it sounds like your most interested in IO/s I've run fio using > 16 threads and a 4k random read workloads. > > $ zpool status -v > NAME STATE READ WRITE CKSUM > hybrid ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > ssd-1 ONLINE 0 0 0 > hdd-1 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > ssd-2 ONLINE 0 0 0 > hdd-2 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > ssd-3 ONLINE 0 0 0 > hdd-3 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > ssd-4 ONLINE 0 0 0 > hdd-4 ONLINE 0 0 0 > > $ iostat -mxz 5 > Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz > avgqu-sz await svctm %util > hdd-1 0.00 0.00 156.80 0.00 19.60 0.00 256.00 > 2.52 16.09 6.37 99.88 > hdd-2 0.00 0.00 149.80 0.00 18.73 0.00 256.00 > 2.51 16.78 6.66 99.80 > hdd-3 0.00 0.00 150.40 0.00 18.80 0.00 256.00 > 2.52 16.74 6.63 99.72 > hdd-4 0.00 0.00 154.20 0.00 19.28 0.00 256.00 > 2.46 15.98 6.47 99.70 > ssd-1 0.00 0.00 1250.80 0.00 156.35 0.00 256.00 > 1.32 1.06 0.62 77.70 > ssd-2 0.00 0.00 1237.20 0.00 154.65 0.00 256.00 > 1.31 1.06 0.62 77.16 > ssd-3 0.00 0.00 1243.40 0.00 155.42 0.00 256.00 > 1.33 1.07 0.62 77.68 > ssd-4 0.00 0.00 1227.60 0.00 153.45 0.00 256.00 > 1.29 1.05 0.63 77.36 > > $ fio rnd-read-4k > read: (groupid=0, jobs=16): err= 0: pid=13087 > read : io=1324.3MB, bw=22594KB/s, iops=5648 , runt= 60018msec > clat (usec): min=6 , max=276720 , avg=2762.90, stdev=1493.94 > lat (usec): min=6 , max=276720 , avg=2763.11, stdev=1493.94 > bw (KB/s) : min= 109, max= 2743, per=6.40%, avg=1446.40, stdev=67.99 > cpu : usr=0.16%, sys=1.37%, ctx=336494, majf=0, minf=432 > IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% > submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,>=64=0.0% > complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,>=64=0.0% > issued r/w/d: total=339009/0/0, short=0/0/0 > lat (usec): 10=0.53%, 20=2.81%, 50=0.02%, 100=0.01%, 750=0.01% > lat (usec): 1000=33.99% > lat (msec): 2=50.03%, 4=1.93%, 10=2.75%, 20=5.09%, 50=2.69% > lat (msec): 100=0.15%, 250=0.01%, 500=0.01% > > Run status group 0 (all jobs): > READ: io=1324.3MB, aggrb=22593KB/s, minb=23136KB/s, maxb=23136KB/s, > mint=60018msec, maxt=60018msec > From the above results you can quickly see a few things. > > 1. As expected the hard drives are nearly 100% utilized and limited at > roughly 150 IO/s. The ssds are not quite maxed out but they are handling the > vast majority of the IO/s at roughly 1200+ operations per second. Some > additional tuning is probably possible to increase that utilization but > that's not too shabby. > > 2. Sustained IO/s for the pool is a respectable 5648, but as expected you > have a bimodal distribution for the latency. Some IO/s are very fast when > serviced by the SSD others are slow when the HDD is used, for this case the > worst case latency was a quite large 276ms with an average of 2.7ms. We could > probably do a bit more to bound the worst case latencies but I think this is > a good start. https://github.com/openzfs/zfs/issues/1461#issuecomment-20507619 <https://github.com/openzfs/zfs/issues/1461#issuecomment-20507619> > This is better than a cure for cancer. Every time I watch you guys improve > ZFS, it's like magic. --- That leads to the patch [Improve N-way mirror performance](https://github.com/openzfs/zfs/commit/556011dbec2d10579819078559a77630fc559112) merged into OpenZFS on Jul 12 2013, so seems a similar solution has not found its way into Illumos ZFS? Can we initiate some effort to backport that into Illumos ZFS? Best regards, Compl > On 2022-01-23, at 22:48, Günther Ernst Alka <g...@napp-it.org> wrote: > > > I have made some benchmarks of the following mixed mirror > (fast Samsung SAS SSD 960, slow HGST 2TB HD, 12G HBA) > > <p6HxZ77PXXgwzzSb.png> > > a quick dd Benchmark of the hd alone (basic pool) gives > > 12.8 GB in 94.3s = 143.82 MB/s Write > 12.8 GB in 104.3s = 130.48 MB/s Read > > a dd Benchmark of the ssd alone (basic pool) gives > > 12.8 GB in 23.3s = 549.36 MB/s Write > 12.8 GB in 16.8s = 870.75 MB/s Read > > a dd Benchmark of the mirror gives > > 12.8 GB in 94.4s = 135.59 MB/s Write > 12.8 GB in 68.6s = 186.59 MB/s Read > > a multistream filebench benchmark of the mirror gives (filebench > 5streamread/write to check effects of parallel io) > > <cMT9iCkTNiNUk3LW.png> > > > Result from dd (singlstream): > Write mirror vs disks: Writing to the mixed pool is as slow as to the hd > Read mirror vs disks: Reading from the mixed pool is only slightly faster as > from the hd > > > singlestream filebench results > > randomread.f randomrw.f singlestreamr > pri/sec cache=all (hd) 103.4 MB/s 118.2 MB/s 1.0 GB/s > > randomread.f randomrw.f singlestreamr > pri/sec cache=all (ssd) 120.6 MB/s 72.8 MB/s 1.1 GB/s > > randomread.f randomrw.f singlestreamr > pri/sec cache=all (mirror) 92.8 MB/s 117.0 MB/s 990.4 MB/s > > ________________________________________________________________________________________ > > > Result from filebench (multistream): > Write mirror vs disks: Writing to the mixed pool is as slow as to the hd > Read mirror vs disks: Single/Random Reading from the mixed pool is as slow > as from the hd > > Filebench result > Single/Random Reading from the mixed pool is as slow as from the hd. > random r/w of the mixed pool is better than hd or ssd alone > Multistream Reading from the mixed pool is twice as fast as singlestream > read from the ssd or hd > and nearly twice as good as singlstreamread from the mixed pool > (read values are too good, readcache seems to improve values massively but > shows indeed the effect of parallel multistream reading) > > In the end especially the bad write and weak read results indicate that a > mixed mirror is a bad idea > > gea > > > > SmartOS <https://smartos.topicbox.com/latest> / smartos-discuss / see > discussions <https://smartos.topicbox.com/groups/smartos-discuss> + > participants <https://smartos.topicbox.com/groups/smartos-discuss/members> + > delivery options > <https://smartos.topicbox.com/groups/smartos-discuss/subscription>Permalink > <https://smartos.topicbox.com/groups/smartos-discuss/T07ae623311b6e168-M23f1851fb3d90d30cce7482c> ------------------------------------------ illumos: illumos-discuss Permalink: https://illumos.topicbox.com/groups/discuss/Tc34977e2e913194b-M52baae03e27a1259fcd51dcf Delivery options: https://illumos.topicbox.com/groups/discuss/subscription
