The approach that we have been taking more recently is incremental CMM
commands. Large CMM "flush" operations were very expensive in that any
pages out on paging device had to be paged in before being "flushed".
Things "froze" during the flush. Doing a few incremental CMM commands
each minute keeps the servers fresh, with their storage size meeting the
workload requirements. zVRM (Velocity Resource Manager) evaluates all
servers every minute and will pick the top candidates for CMM, and
increase the CMM "balloon" incrementally. This minimizes impact and
returns unused pages back to the VM page list. Automation is the key
buzz word today. Automating Linux storage management with a resource
manager helps a lot. Oh, and overhead is negligible.
Contact me direct if you would like to understand more.
On 8/28/2024 5:19 PM, Rodriguez-Bell, Ted wrote:
For a long time we've tried to reduce VM page thrashing by running cmmflush on
all our non-production servers at night. Cmmflush is a script that Rob van der
Heij wrote about a decade ago that goes through the Linux cache looking for
pages that it wants to return to VM. We run it in the wee hours on most dev
systems trying to kick things out of the Linux buffer cache because double
buffering leads to bad performance. Our memory is also heavily overcommitted
in dev.
Recently, we found a serious problem that cmmflush might be causing and IBM via
Suse recommended that we just stop. They say that newer versions of the kernel
have background cmma flushing enabled by default, and if it's working we
shouldn't need cmmflush. A Redhat bug report I found says that one shouldn't
use old-style CMM (like cmmflush) and hot-pluggable memory together; we do.
So: should we just retire cmmflush? How do we make sure that the newer cmma
is enabled? It's supposedly the default but I don't see it in /proc/cmdline on
any guests where I've looked. Are there kernel parameters in /sys we should
look for?
In case you're not familiar with it, cmmflush pretty much freezes the server as
it runs through the buffers. We already don't do it if there's software that
doesn't like it. The new problem was that a few servers with an application
that had filled more than half of swap lost their primary network interfaces.
We couldn't recover the network so we had to just halt and reboot the servers.
The problem seemed to start right after cmmflush ran. We're not certain that
cmmflush caused the problem but the correlation is striking.
I don't want to sound like I'm criticizing Mr. van der Heij for this at all.
cmmflush was a big help when we started using it. If we missed updates that
made it unnecessary that's on us.
Rob's presentation at SHARE 2013:
https://share.confex.com/share/121/webprogram/Handout/Session13486/2013-13486.pdf
IBM kernel docs: cmma - Reduce hypervisor paging I/O overhead - IBM
Documentation<https://www.ibm.com/docs/en/linux-on-systems?topic=kp-cmma-2>
Redhat bug report (membership needed):
https://access.redhat.com/solutions/4621911
Ted Rodriguez-Bell
z/VM and z/Linux, Mainframe and Midrange Services
Wells Fargo | 333 Market St, Floor 10, San Francisco, CA 94105-2102
MAC A0109-103 | 415-516-7913 mobile,
4155167...@vtext.com<mailto:4155167...@vtext.com> or
http://www.vtext.com<http://www.vtext.com/> text paging
te...@wellsfargo.com<mailto:te...@wellsfargo.com>
Company policy requires: This message may contain confidential and/or
privileged information. If you are not the addressee or authorized to receive
this for the addressee, you must not use, copy, disclose, or take any action
based on this message or any information herein. If you have received this
message in error, please advise the sender immediately by reply e-mail and
delete this message. Thank you for your cooperation.
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www2.marist.edu/htbin/wlvindex?LINUX-390
