On 2011-01-13, at 11:38, Nathan Dauchy wrote: > On a somewhat related note... we have recently discovered that the > object caching added in 1.8 consumes all the memory on the OSS nodes, > leaving insufficient block device cache for the inodes. This was making > 'ls -l' and 'du' run 10-20x longer than when we were running > lustre-1.6.7. If you are running 1.8 and want to try turning off the > object caching, these are the settings you should look at: > > lctl conf_param fsname-OST00XX.ost.read_cache_enable=0 > lctl conf_param fsname-OST00XX.ost.writethrough_cache_enable=0
It would probably be better to set: lctl conf_param fsname-OST00XX.ost.readcache_max_filesize=32M or similar, to limit the read cache to files 32MB in size or less (or whatever you consider "small" files at your site. That allows the read cache for config files and such, while not thrashing the cache while accessing large files. We should probably change this to be the default, but at the time the read cache was introduced, we didn't know what should be considered a small vs. large file, and the amount of RAM and number of OSTs on an OSS, and the uses varies so much that it is difficult to pick a single correct value for this. Cheers, Andreas -- Andreas Dilger Principal Engineer Whamcloud, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
