All,
I asked this question a month or so back, but thought I'd ask it again to
see if anyone else had insight. I'm in the process of designing a new lustre
file system to replace an existing 2.3PB lustre file system. We have over a
thousand clients connecting to lustre over mostly QDR IB, as well as two
clusters on omnipath that connect through four lnet routers. Presently we are
running 2.5.42, we have a metadata system that runs on ldsik that presently has
52GB of space used (164.0M inodes) and has 64GB of memory (the system is from
2014). It is a system with two identical heads attached to a disk shelf,
configured with one head as the active head and one as the standby. I've been
told to plan for a system that is 4x the size of the current one. We are
looking at Lustre 2.10 or 2.12 (favoring 2.12 to allow data on the MDT).
With that background, here is what I've found from the last time I asked this
(and from my reading online).
1) For the CPU on the MDT system. Faster cores + cache on CPU is more
important then more cores, but the system should have at least 16 cores.
2) ZFS is ready for use on the MDT (assuming zfs 0.7.x.x)
3) I'm going to recommend at least 128GB of memory, but that 256GB would be
better.
4) for the MDT itself striping across mirrors provides superior speed over
raidz2. SSDs will be used.
For fault tolerance the new system will have two heads, with the MDT/MGS
failing over between the two of them... which opens up the possibility of
splitting the MDT into two MDTs, then having each MDT mounted on a head. This
was mentioned the last time I asked about this, and spreads the MDT load
between two systems. If we do that I assume that I'll have to set the striping
on the first directories created, and then further directories will inherit
that striping.
w/r,
Kurt J. Strosahl
System Administrator: Lustre, HPC
Scientific Computing Group, Thomas Jefferson National Accelerator Facility
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
