Everything changes with a JVM on the picture. We have triple memory management, or quadruple, if we consider the application as managing memory too. We have zVM and real storage, LInux and its memory management, JVM and its allocated memory, and the application.
You cannot tell Java to release memory. You can call System.gc() to suggest JVM to release memory, but JVM will decide if and when to release any. You can limit the memory JVM will get, so using -Xmx and -XX:MaxPermSize switches will reduce the memory usage for JVM. A smaller memory means the Garbage Collector will run more times, but will take less time to release memory (the same case of $100k at one time versus 1000 times $100). It can kill your application too, so do some stressing and research before defining this limit. A small JVM memory means java dumps and out of memory errors too. To get a good number for memory usage, you can use some memory profiler for JVM. Take a look at http://java-source.net/open-source/profilers to get some suggestions. Plug the chosen profiler on the JVM, stress the application and take a look on the memory usage, garbage collection runs, and so. Ccut the memory high enough to not have out of memory errors and low enough to not have JVM waste GBs of memory. After that, ask the developers to let you reduce the memory for their JVM. Even harder than configuring the memory for JVM is getting the used memory from the developers. They usually inflate the needed memory, because having lots of memory is a good thing. And you will make no friends saying "I will reduce your JVM memory"... Mauro http://mauro.limeiratem.com - registered Linux User: 294521 Scripture is both history, and a love letter from God. 2013/8/16 Veencamp, Jonathon D. <[email protected]> > Thanks again! I've got some research ahead of me. > > Jon > > -----Original Message----- > From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Rob > van der Heij > Sent: Friday, August 16, 2013 7:51 AM > To: [email protected] > Subject: Re: linux cache > > When the pages are owned by the JVM, there's nothing Linux can do about it. > Since the application is not touching the stuff in the heap, VM will > eventually page it out. When the application ramps up again, paging will > slow you down because VM block paging can't help you much. > > What you could do is set your minimum heap size low enough that you > encourage the JVM to clean up earlier and give extents back to Linux when > it can. That way Linux sees the memory is free and could return it to z/VM > again (with my cmmflush). Problem is knowing when to do that. As you point > out, when the JVM shrinks the heap, that gives Linux room to grow the page > cache. In some cases there are tasks that grow the page cache very fast > (backup, file scanning). I have some ideas to handle that: > http://zvmperf.wordpress.com/2013/01/27/taming-linux-page-cache/ > > Another thing I would want to try in your case is to use large pages for > the JVM. Because that is pre-allocated, z/VM block paging could be an order > of magnitude less painful when the application comes back after a long idle > period. > > As long as memory is used by the JVM, cpuplugd can't do anything either. > When you have the JVM return the extents when possible, cpuplugd could > return the memory to z/VM. But my experience is that it keeps things too > tight and it would take very long before it deflates the balloon enough to > let the heap grow again. > > My script has a modprobe of the cmm module, that's the only thing I need to > be prepared. It also tolerates a fixed setting of the balloon if you had > that before. > > Rob > > > On 16 August 2013 14:21, Veencamp, Jonathon D. <[email protected] > >wrote: > > > I have some development zLinux guests with 25 JVM's. Most of these > aren't > > used on a typical day and can idle along with a 200MB heap. But if a > > development team is doing load testing, then the heap and memory > footprint > > for a JVM can grow to 1.5GB+. If 2-3 teams are doing testing, my Linux > > memory utilization on that guest just grow by 4-6 GB. I cannot "right > > size" some of these guests without fixed heaps which seems wasteful, and > > relying on multiple 500MB VDISK linux swaps for Linux can have downsides. > > I've had guest VDISKS paged out by VM with terrible performance > penalties. > > I'd hypothesize VDISK can be far more likely to get paged out than > memory > > by the file cache! Swappiness=0 doesn't prevent the cache from being > > populated. If it's populated, VM still might not see that memory as LRU. > > Unless I can help out the hypervisor by preventing some memory from being > > used, then it limits our ability to overcommit memory on VM. > > > > So a month ago I asked the list about cpuplugd, CMM, & CMMA. Everyone > > said to stay away from these. But this is exactly the problem I was > trying > > to solve, and why I went with the drop caches approach. I don't need > > "always on" help, just the ability to suppress the linux file cache. > Drop > > caches seems to help. On a guest described like above, I'm going into > the > > day with a 1.5GB cache (still too big), but 3.5GB of free memory that is > a > > candidate for LRU. That seems to work for me. > > > > Rob - For your example script to work, does CMM need to be fully > > configured? > > > > And with "fairness in opinions" in mind, how would you recommend I > > approach my situation Mauro? > > > > Jon > > > > > > -----Original Message----- > > From: Linux on 390 Port [mailto:[email protected]] On Behalf Of > Rob > > van der Heij > > Sent: Friday, August 16, 2013 5:22 AM > > To: [email protected] > > Subject: Re: linux cache > > > > The thing that happened to your servers is that drop_caches frees up page > > frames in Linux that have not been referenced for a long time. Linux > starts > > to use them for something else, and requires z/VM to page it in. To make > > room for that, VM will page out other parts of the guests. > > > > You're still missing the point. We don't drop the caches because Linux > > needs memory but because we want to determine what memory to give back to > > VM. Yes, you also lose the good things from cache, but that's a trade-off > > to make with performance instrumentation. As I started my first response > in > > this thread, if you don't plan to give the pages back with CMM or CMMA, > it > > is not a good idea to use drop_caches and swappiness might be a better > way > > to help workloads that get in trouble. > > > > Rob > > > > > > On 15 August 2013 19:16, Mauro Souza <[email protected]> wrote: > > > > > I have read it now. And as I think there's no CMM involved, I still > > > disagree. > > > I said I would never ever drop caches because I saw it almost killing a > > > LPAR. I had a customer that put drop_caches on crontab, and at that > time > > > *every* guest started dropping caches and zVM was crazy doing a lot of > > > page-ins and page-outs. Nobody understood why the system was > > unresponsible. > > > I think is less costly to zVM to handle dozens of page-ins and outs > for a > > > Linux needing 10MB of memory here and there, than to drop 2GB of caches > > at > > > the same time. If you drop all the cache at the same time, Linux will > > have > > > to make an I/O operation for every file it will read, and zVM will > have > > to > > > read a lot of data from PAGE that will be just ignored. > > > No matter if you drop all the cache at the same time, or drop a page > when > > > you need it, zVM will have to bring it back. Dropping all caches is > like > > > paying a $100k loan in one time, instead of splitting it in 1000 times > of > > > $100. > > > > > > > > > Mauro > > > http://mauro.limeiratem.com - registered Linux User: 294521 > > > Scripture is both history, and a love letter from God. > > > > > > > > > 2013/8/15 Rob van der Heij <[email protected]> > > > > > > > On 15 August 2013 17:34, Mauro Souza <[email protected]> wrote: > > > > > > > > > > > > > You can safely ignore the cache usage on Linux, zVM will realise > the > > > page > > > > > was not in use and drop it itself. When Linux needs memory, it will > > > > reclaim > > > > > cache pages automatically. > > > > > > > > > > So we disagree. Maybe we wouldn't if you had read the page that I > > > linked > > > > to. > > > > > > > > The point that you miss is when Linux needs memory and reclaims a > page > > > > frame from page cache, z/VM will first bring back the old contents > that > > > > Linux did not care about anymore. This is where getting a free page > > will > > > > slow you down. Especially since the LRU patterns interfere. The idea > is > > > > that you want to drop the backing page from z/VM either before it is > > > paged > > > > out or before z/VM will page it back in. If you have CMMA enabled, > then > > > > having Linux free the page (by drop_caches) will prevent z/VM from > > paging > > > > it out. > > > > > > > > Rob > > > > > > > > > ---------------------------------------------------------------------- > > > > For LINUX-390 subscribe / signoff / archive access instructions, > > > > send email to [email protected] with the message: INFO > LINUX-390 > > or > > > > visit > > > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > > > > ---------------------------------------------------------------------- > > > > For more information on Linux on System z, visit > > > > http://wiki.linuxvm.org/ > > > > > > > > > > ---------------------------------------------------------------------- > > > For LINUX-390 subscribe / signoff / archive access instructions, > > > send email to [email protected] with the message: INFO LINUX-390 > or > > > visit > > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > > ---------------------------------------------------------------------- > > > For more information on Linux on System z, visit > > > http://wiki.linuxvm.org/ > > > > > > > ---------------------------------------------------------------------- > > For LINUX-390 subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO LINUX-390 or > > visit > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > ---------------------------------------------------------------------- > > For more information on Linux on System z, visit > > http://wiki.linuxvm.org/ > > > > ________________________________ > > > > The information contained in this e-mail message is intended only for the > > personal and confidential use of the designated recipient(s) named above. > > This message may be an attorney-client or work product communication > which > > is privileged and confidential. It may also contain protected health > > information that is protected by federal law. If you have received this > > communication in error, please notify us immediately by telephone and > > destroy (shred) the original message and all attachments. Any review, > > dissemination, distribution or copying of this message by any person > other > > than the intended recipient(s) or their authorized agents is strictly > > prohibited. Thank you. > > > > ---------------------------------------------------------------------- > > For LINUX-390 subscribe / signoff / archive access instructions, > > send email to [email protected] with the message: INFO LINUX-390 or > > visit > > http://www.marist.edu/htbin/wlvindex?LINUX-390 > > ---------------------------------------------------------------------- > > For more information on Linux on System z, visit > > http://wiki.linuxvm.org/ > > > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > > ________________________________ > > The information contained in this e-mail message is intended only for the > personal and confidential use of the designated recipient(s) named above. > This message may be an attorney-client or work product communication which > is privileged and confidential. It may also contain protected health > information that is protected by federal law. If you have received this > communication in error, please notify us immediately by telephone and > destroy (shred) the original message and all attachments. Any review, > dissemination, distribution or copying of this message by any person other > than the intended recipient(s) or their authorized agents is strictly > prohibited. Thank you. > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or > visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
