Hello, I am running Lustre 2.10.0 on Centos 7.3 I have one MDS and two OSSes, each with one OST each OST is a ZFS raidz1 with 6 nvme disks each. The configuration of ZFS is done in a way to allow maximum write performances:
zfs set sync=disabled drpffb-ost02 zfs set atime=off drpffb-ost02 zfs set redundant_metadata=most drpffb-ost02 zfs set xattr=sa drpffb-ost02 zfs set recordsize=1M drpffb-ost02 every NVMe disk has 4K byte sector, zfs -o ashift=12 In a LOCAL raidz1 configuration I get 3.6GB/sec writings and 5GB/sec readings. The same configuration thru Lustre has very poor performances, 1.3GB/sec writes and 2GB/sec reads There must be something else to look for having better performances but a local ZFS raidz1 is working pretty good. this is the Lustre partition client side: 172.21.42.159@tcp5:/drpffb 10T 279G 9.8T 3% /drpffb UUID bytes Used Available Use% Mounted on drpffb-MDT0000_UUID 19.1G 2.1M 19.1G 0% /drpffb[MDT:0] drpffb-OST0001_UUID 5.0T 142.2G 4.9T 3% /drpffb[OST:1] drpffb-OST0002_UUID 5.0T 136.4G 4.9T 3% /drpffb[OST:2] filesystem_summary: 10.0T 278.6G 9.7T 3% /drpffb Tests both on Lustre/ZFS and local ZFS are based on 50 threads writing 4GB of data each and 50 threads reading using iozone: iozone -i 0 -t 50 -i 1 -t 50 -s4g I do not know what else I can do to improve performances here some details on the OSSes OSS01: NAME USED AVAIL REFER MOUNTPOINT drpffb-ost01 39.4G 4.99T 153K none drpffb-ost01/ost01 39.4G 4.99T 39.4G none pool: drpffb-ost01 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM drpffb-ost01 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 nvme0n1 ONLINE 0 0 0 nvme1n1 ONLINE 0 0 0 nvme2n1 ONLINE 0 0 0 nvme3n1 ONLINE 0 0 0 nvme4n1 ONLINE 0 0 0 nvme5n1 ONLINE 0 0 0 OSS02: NAME USED AVAIL REFER MOUNTPOINT drpffb-ost02 62.2G 4.97T 153K none drpffb-ost02/ost02 62.2G 4.97T 62.2G none pool: drpffb-ost02 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM drpffb-ost02 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 nvme0n1 ONLINE 0 0 0 nvme1n1 ONLINE 0 0 0 nvme2n1 ONLINE 0 0 0 nvme3n1 ONLINE 0 0 0 nvme4n1 ONLINE 0 0 0 nvme5n1 ONLINE 0 0 0 thanks to anyone who may help giving hints. Rick _______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org