On Oct 23, 2008 08:33 -0400, Craig Prescott wrote: > I am considering hardware requirements for an > MDS to be paired with a 500TB Lustre filesystem. > I have a question regarding the sizing guidelines > described in the manual. > > For an anticipated average file size of 1MB, the > MDT size guideline from section 21.3.2 works out > to 4TB. For comparison, on our production 28TB > Lustre filesystem, we have:
The 4kB/inode (and doubled) value is just a safe rule of thumb. > # lfs df > UUID 1K-blocks Used Available Use% Mounted on > ufhpc-MDT0000_UUID 213655168 17329964 196325204 8% > /ufhpc/scratch[MDT:0] > ... > filesystem summary: 29966190744 20728270148 9237920596 69% /ufhpc/scratch > > # lfs df -i > UUID Inodes IUsed IFree IUse% Mounted on > ufhpc-MDT0000_UUID 61049728 21416476 39633252 35% > /ufhpc/scratch[MDT:0] > ... > filesystem summary: 61049728 21416476 39633252 35% /ufhpc/scratch It does indeed appear that you have 1MB average file size, and are about 4x over-provisioned on the MDS (you use about 1kB/inode instead of 4kB/inode). > My concern is that if I follow the guidelines, I would > over-provision the MDS with space we would never use. > > I understand the inodes are pre-allocated and won't show up in > the "Used" column above. Under what conditions would the actual > MDT space get used more significantly? If you suddenly get more small files for some reason your space can disappear quickly - there are 60M inodes remaining, about 50% more than you need when the fs is full. The other thing that consumes MDS space is extended attributes. If you have lots of widely-striped files, or SELinux labels or ACLs these may require an extra block on the MDS. In high-performance MDTs you end up adding disks just to improve the performance, and get enough capacity for free. The above system only has a 200GB MDS, probably only 2 disks mirrored. I'd say for the large system you could get by with a 2TB MDT (you should specify "-i 1500" or so, to increase the inode count over the default). This is again only 2 or 4 disks (mirrored) in the end... Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
