We were building the ZFS based OST and we also found out I/O wait even when 4 OST per shelf. please lower the amount of OST available as the HBA may not be sustainable for that high I/O . we are now using enterprise Linux (like Cray) as they are designed to have high I/O. we would like to see if anyone has experience solving that part.
Regards, James ________________________________ From: lustre-discuss <[email protected]> on behalf of Kurt Strosahl via lustre-discuss <[email protected]> Sent: Monday, February 27, 2023 3:02 PM To: [email protected] <[email protected]> Subject: [lustre-discuss] OSTs per OSS question Good Morning, I'm in the early phases of a new lustre file system design. One thing we are observing on our lustre file system from 2019 (running 2.12.9-1) is that the oss systems report a lot of time spent in i/o wait. We are configured with two oss systems connected via sas cables to a pair of JBODS, and are wondering if going from two JBODs per OSS (so 6 osts per oss under normal operating conditions, 12 under failover) to 4 shelves (12 osts per oss under normal operating conditions, 24 under failover). The OSTs are 10 raidz2 vdevs, and we are planning on using 20TB drives in this new file system. Has anyone tried the 4 shelf / 2 oss configuration? Reading through the Lustre manual I see the following under table 1.2 "OSS: 1-128 TiB per OST, 1-8 OSTs per OSS" Is that an indication that more than 8 OSTs per OSS causes problems for the OSS systems? Our current OSS systems have run at 12 OSTs during failover situations, once for at least a few days due to a hardware failure on one of the OSS systems. respectfully, Kurt J. Strosahl (he/him) System Administrator: Lustre, HPC Scientific Computing Group, Thomas Jefferson National Accelerator Facility
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
