On 13-Oct-09, at 04:58, Brian J. Murrell wrote: > On Mon, 2009-10-12 at 17:06 -0700, David Simas wrote: >> During this operation, we notice the value of "buffers" >> reported in '/proc/meminfo' on the OSSs involved increasing >> monotonically >> until it apparently take up all the system's memory - 32 GB. > > This would likely be OSS read cache, if you have it enabled. If you > do, > you should disable it due to a potential corruption issue. Details > were > given previously on this list on how to do that. Check the archives. > > But that's not directly related to what you are seeing. > > Having "buffers" consume all of available memory is SOP (standard > Operating Procedure) for Linux. The philosophy is that > "free" (unused) > memory is wasted memory and as such, any memory not needed by > applications or other kernel processing is used to buffer disk I/O. > The > performance spiff of such is obvious I think. > >> Then 'kswapd' >> starts consuming a large amount of CPU, the load increases (100+), >> and the >> system, including Lustre, slows to crawl and becomes quite useless. > > This doesn't sound normal or good.
There is a recent bug for memory pressure on the OSS. I believe it was fixed for the next release. >> If we >> stop Lustre I/O at this point, 'kswapd' and the system load calm >> down, but >> the "buffers" value does not decrease. > > Right. The buffers will not get "emptied" until something else needs > the memory. Again, unused memory is wasted memory. > >> We have observed the monotonically increasing "buffers" condition >> with non-Lustre I/O on systems running the Lustre 1.8.1 kernel >> (2.6.18-128.1.14.el5_lustre.1.8.1), > > Indeed. The using up of memory by the buffer cache is a standard > (i.e. > non-Lustre-specific) feature, and you will find the same thing on > non-Lustre kernels as well. > > b. > > _______________________________________________ > Lustre-discuss mailing list > [email protected] > http://lists.lustre.org/mailman/listinfo/lustre-discuss Cheers, Andreas -- Andreas Dilger Sr. Staff Engineer, Lustre Group Sun Microsystems of Canada, Inc. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
