Thanks for this info. But actually I was really looking at the number of OSS, not OSTs :) This is really more how Lustre client nodes and MDT will cope with very large number of OSSes.
De : Andreas Dilger <adil...@whamcloud.com> Date : vendredi 4 octobre 2019 à 04:54 À : "Degremont, Aurelien" <degre...@amazon.com> Cc : "lustre-discuss@lists.lustre.org" <lustre-discuss@lists.lustre.org> Objet : Re: [lustre-discuss] Limit to number of OSS? On Oct 3, 2019, at 07:55, Degremont, Aurelien <degre...@amazon.com<mailto:degre...@amazon.com>> wrote: Hello all! This doc from the wiki says "Lustre can support up to 2000 OSS per file system" (http://wiki.lustre.org/Lustre_Server_Requirements_Guidelines). I'm a bit surprised by this statement. Does somebody has information about the upper limit to the number of OSSes? Or what could be the scaling limitator for this number of OSS? Network limit? Memory consumption? Other? That's likely a combination of a bit of confusion and a bit of safety on the part of Intel writing that document. The Lustre Operations Manual writes: Although a single file can only be striped over 2000 objects, Lustre file systems can have thousands of OSTs. The I/O bandwidth to access a single file is the aggregated I/O bandwidth to the objects in a file, which can be as much as a bandwidth of up to 2000 servers. On systems with more than 2000 OSTs, clients can do I/O using multiple files to utilize the full file system bandwidth. I think PNNL once tested up to 4000 OSTs, and I think the compile-time limit is/was 8000 OSTs (maybe it was made dynamic, I don't recall offhand), but the current code could _probably_ handle up to 65000 OSTs without significant problems. Beyond that, there is the 16-bit OST index limit in the filesystem device labels and the __u16 lov_user_md_v1->lmm_stripe_offset to specify the starting OST index for "lfs setstripe", but that could be overcome with some changes. Given OSTs are starting to approach 1PB with large drives and declustered-parity RAID, this would get us in the range 8-65EB, which is over 2^64 bytes (16EB), so I don't think it is an immediate concern. Let me know if you have any trouble with a 9000-OST filesystem... :-) Cheers, Andreas -- Andreas Dilger Principal Lustre Architect Whamcloud
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
