On Thu, Jan 13, 2011 at 05:28:23PM -0500, Kit Westneat wrote: >> It would probably be better to set: >> >> lctl conf_param fsname-OST00XX.ost.readcache_max_filesize=32M >> >> or similar, to limit the read cache to files 32MB in size or less (or >> whatever you consider "small" files at your site. That allows the read >> cache for config files and such, while not thrashing the cache while >> accessing large files. >> >> We should probably change this to be the default, but at the time the read >> cache was introduced, we didn't know what should be considered a small vs. >> large file, and the amount of RAM and number of OSTs on an OSS, and the uses >> varies so much that it is difficult to pick a single correct value for this.
limiting the total amount of OSS cache used in order to leave room for inodes/dentries might be more useful. the data cache will always fill up and push out inodes otherwise. Nathan's approach of turning off the caches entirely is extreme, but if it gives us back some metadata performance then it might be worth it. or is there a Lustre or VM setting to limit overall OSS cache size? I presume that Lustre's OSS caches are subject to normal Linux VM pagecache tweakables, but I don't think such a knob exists in Linux at the moment... >I was looking through the Linux vm settings and saw vfs_cache_pressure - >has anyone tested performance with this parameter? Do you know if this >would this have any effect on file caching vs. ext4 metadata caching? > >For us, Linux/Lustre would ideally push out data before the metadata, as >the performance penalty for doing 4k reads on the s2a far outweighs any >benefits of data caching. good idea. if all inodes are always cached on OSS's then the fs should be far more responsive to stat loads... 4k/inode shouldn't use up too much of the OSS's ram (probably more like 1 or 2k/inode really). anyway, following your idea, we tried vfs_cache_pressure=50 on our OSS's a week or so ago, but hit this within a couple of hours https://bugzilla.lustre.org/show_bug.cgi?id=24401 could have been a coincidence I guess. did anyone else give it a try? BTW, we recently had the opposite problem on a client that scans the filesystem - too many inodes were cached leading to low memory problems on the client. we've had vfs_cache_pressure=150 set on that machine for the last month or so and it seems to help. although a more effective setting in this case was limiting ldlm locks. eg. from the Lustre manual lctl set_param ldlm.namespaces.*osc*.lru_size=10000 cheers, robin -- Dr Robin Humble, HPC Systems Analyst, NCI National Facility _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
