The 160 limit has been raised. I don't know what the new one is, but it is *quite* large. I'm pretty sure it's beyond practical interest today.
There are a few issues with having extremely large numbers of OSTs, especially if you are explicitly trading off 1 vs many OSTs. There are no particular scaling issues with number of OSTs of an OSS, so if you took the same storage and subdivided it to create more OSTs, there's no particular concern there. But that assumes you're taking the same storage and deciding how to subdivide it - Obviously, a given amount of CPU/RAM/network on the OSS can only "feed" so much storage, so if you're just *adding* storage, you will quickly exhaust your OSS resources. (Generally speaking one tries to match the two, so one does not have too much CPU/RAM/network bandwidth OR too much storage.) The two main problems I see with "many" OSTs are: 1. They can get rather small, and so they can fill up relatively easily. If your OSTs are really small and a few of the files assigned to that OST become large (so, they're assigned there when the OST is mostly empty, and then grow large), you'll run out of space on that OST and will no longer be able to write to files striped there. 2. As file stripe counts go up, the file layout - basically, the mapping from the byte range as seen in userspace to the actual objects on the OSTs - can become large enough that sending it around is a performance bottleneck. Opening a single file with hundreds of stripes from thousands of clients - like a large supercomputer center might do - can take a significant period of time. That second is the only scaling issue with OST *count* that I'm aware of, other than that there is a bit of memory overhead for tracking each OST - so 10 OSTs instead of 1 OST will use marginally more memory on servers and clients. This is pretty small, though. So in general, I would say you'd be happier with fewer & faster, rather than more & slower, especially when talking about very large OST counts. There are some performance issues with multiple writers to single files with low stripe counts, so it doesn't hold in extremis. This is all to say you'd be much better served with 10 OSTs than with *1*, but 100 is probably not a better idea than 10. - Patrick On 10/11/18, 1:07 PM, "lustre-discuss on behalf of Michael Di Domenico" <[email protected] on behalf of [email protected]> wrote: Is there a limit on the number of oss servers you can have in a single filesystem? is there one for ost's? I'm curious of the performance implications between two different configurations (this is just theory mind you)... 1000 oss with 1 ost each vs 100 oss with 10 ost each one could scale this up further 2000, 5000, 10000 oss's with 1 single ost each i did note two references one from 2012 by Oleg Drokin that tested 1300 OST's at ornl which "mostly worked" and a note from Andreas last year that quoted 2000 OST's before scaling issues. I'm curious if there's a fundamental issue with scaling lustre, which might be based on the presumption that oss's are typically fatter nodes (getting fatter everyday) rather than large quantities of skinny nodes I'm also curious if there is still a 160 OST limit for file striping as well. _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
