Dne 24. 06. 23 v 1:22 Anton Kulshenko napsal(a):
Hello.
Please help me figure out what my problem is. No matter how I configure the
system, I can't get high performance, especially on writes.
OS: Oracle Linux 8.6, 5.4.17-2136.311.6.el8uek.x86_64
Platform: Gigabyte R282-Z94 with 2x 7702 64cores AMD EPYC and 2 TB of RAM
Disks: NVMe Samsung PM1733 7.68 TB
What I do:
vgcreate vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
lvcreate -n thin_pool_1 -L 20T vg1 /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1
/dev/nvme3n1 -i 4 -I 4
-i4 for striping between all disks, -I4 strip size. Also I tried 8, 16, 32...
In my setup I can't find a big difference.
stripe size needs to be aligned with some 'hw' properties.
In the case of 'NVMe' where the write unit for optimal performance is usually
0.5M or more - using 4K block is basically massively destroying your
performance since you generate for each large write huge amount of splits.
I only get 40k iops, while one drive at the same load easily gives 130k iops.
I have tried different block sizes, strip sizes, etc. with no result. When I
look in iostat I see the load on the disk where the metadata is:
80 WMB/s, 12500 wrqm/s, 68 %wrqm
I don't understand what I'm missing when configuring the system.
As mentioned by Mathew - you likely should start with some 'initial' thin-pool
size - maybe sitting fully on single NVMe - possibly deploy metadata on 2nd.
NVMe for better bus utilization.
For striping - you would need to go with 512K units at least - then it's the
question how it fits your workload...
Anyway now you have way more things to experiment and benchmark and figure out
what is the best on your particular hw.
One more thing - increasing chunksize to 256K or 512K also may significantly
raise the performance - but at the price of reduced sharing in case of taking
a snapshot of a thin volume...
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
