The memory sizes here look different than you posted before. You have 8G of memory here (well 8220492k) but only 780M of swap. Your WAS heap sizes totaled around 7G if I recall (28 JVMs x 256M ). You can see what the running heap size is with "ps -ef | grep Xmx" and look for the value on the Xmx (i.e. -Xmx512m ) So, yeah, this time the OOM condition is expected.
Marcy "This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation." -----Original Message----- From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Daniel Tate Sent: Tuesday, July 27, 2010 12:30 PM To: [email protected] Subject: Re: [LINUX-390] OOM Condition on SLES11 running WAS - Tuning problems? Unfortunately it did not. It continued to grow and grow until oom killed in a loop. We're supposed to have a max size of 512MB per VM. Last time i did any WAS stuff was on Linux but on version 3 when it was new.. a long time ago. top - 14:04:46 up 2:09, 5 users, load average: 60.09, 18.03, 6.96 Tasks: 144 total, 2 running, 142 sleeping, 0 stopped, 0 zombie Cpu(s): 30.9%us, 67.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.2%hi, 0.1%si, 1.8%st Mem: 8220492k total, 8183456k used, 37036k free, 680k buffers Swap: 779880k total, 779880k used, 0k free, 54288k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4520 wasadmin 20 0 1178m 576m 4460 S 107 7.2 11:59.54 java 8943 wasadmin 20 0 566m 254m 4372 S 15 3.2 2:11.86 java 10326 wasadmin 20 0 581m 220m 4224 S 15 2.7 2:01.12 java 9678 wasadmin 20 0 563m 297m 20m S 15 3.7 2:11.54 java 4368 wasadmin 20 0 1148m 603m 2848 S 13 7.5 11:55.56 java 9321 wasadmin 20 0 597m 262m 4720 S 10 3.3 2:15.96 java 11107 wasadmin 20 0 555m 272m 4048 S 10 3.4 2:02.47 java 10706 wasadmin 20 0 509m 241m 2720 S 9 3.0 2:05.35 java 4920 wasadmin 20 0 540m 228m 3296 S 8 2.8 2:20.82 java 5298 wasadmin 20 0 586m 253m 2792 S 8 3.2 2:06.52 java 7794 wasadmin 20 0 594m 307m 4644 S 7 3.8 2:06.89 java 6021 wasadmin 20 0 542m 208m 2908 S 5 2.6 1:53.18 java 6205 wasadmin 20 0 562m 226m 2980 S 5 2.8 1:57.61 java 7265 wasadmin 20 0 517m 218m 2888 S 4 2.7 1:51.77 java 9992 wasadmin 20 0 624m 223m 2896 S 4 2.8 2:07.83 java 5481 wasadmin 20 0 707m 254m 4516 S 3 3.2 2:28.43 java 6390 wasadmin 20 0 509m 167m 3672 S 3 2.1 2:13.46 java 7516 wasadmin 20 0 531m 272m 2272 S 3 3.4 2:05.57 java 11502 wasadmin 20 0 483m 207m 2772 S 3 2.6 1:17.93 java 12248 wasadmin 20 0 520m 203m 3544 S 3 2.5 1:22.76 java 6805 wasadmin 20 0 577m 254m 2980 S 3 3.2 2:08.65 java 5657 wasadmin 20 0 572m 262m 22m S 1 3.3 2:12.52 java 8615 wasadmin 20 0 574m 255m 2792 S 1 3.2 2:12.48 java 5068 wasadmin 20 0 580m 243m 2560 S 0 3.0 2:11.30 java 6586 wasadmin 20 0 580m 220m 3112 S 0 2.8 2:08.78 java 7036 wasadmin 20 0 664m 244m 2504 S 0 3.0 2:02.78 java 8066 wasadmin 20 0 578m 233m 3128 S 0 2.9 1:58.66 java 11924 wasadmin 20 0 542m 203m 4780 S 0 2.5 1:21.65 java 4052 wasadmin 20 0 9968 2624 1188 S 0 0.0 0:00.15 bash 5008 wasadmin 20 0 6900 1748 1028 S 0 0.0 0:23.92 top 5181 wasadmin 20 0 9972 2724 1192 S 0 0.0 0:00.22 bash 5839 wasadmin 20 0 696m 217m 2716 S 0 2.7 1:54.87 java 8347 wasadmin 20 0 528m 242m 22m S 0 3.0 1:55.78 java On Tue, Jul 27, 2010 at 10:26 AM, Marcy Cortes <[email protected]> wrote: > WAS calls non-heap memory "native". > That's what you seem to be using up. > Look at this http://www-01.ibm.com/support/docview.wss?uid=swg21373312 > And if these things don't help, open a PMR with WAS support. > Did it stabilize at 18G? > The panic_on_oom is a good idea - you can then get a dump (set up a linux > dump volume if you are in a hurry - the VMDUMP command takes all day :) > > > Marcy > > "This message may contain confidential and/or privileged information. If you > are not the addressee or authorized to receive this for the addressee, you > must not use, copy, disclose, or take any action based on this message or any > information herein. If you have received this message in error, please advise > the sender immediately by reply e-mail and delete this message. Thank you for > your cooperation." > > > -----Original Message----- > From: Linux on 390 Port [mailto:[email protected]] On Behalf Of Daniel > Tate > Sent: Tuesday, July 27, 2010 7:53 AM > To: [email protected] > Subject: Re: [LINUX-390] OOM Condition on SLES11 running WAS - Tuning > problems? > > Yes, but no memory ever gets freed. The system stops responding and > is too busy dumping information to the console screen to respond to > anything at the console or network. > > And a system-wide panic would provide me with a core file to analyze > and send off; though the progress we've made thanks to this group has > helped a bit (swappiness = 0) has helped it remain stable longer.. > right now we're using 18GB of swap instead of the 35 we were before. > i haven't tried panic_on_oom, but thats a good idea and i was > braindead not to think of it. > > On Mon, Jul 26, 2010 at 8:48 PM, Shane G <[email protected]> wrote: >> Some more info please. >> ... you get a OOM condition ?. >> ... the/a large consumer gets killed ? >> ... the system "halts" (explain) ?. >> >> You *want* a system-wide panic ?. If so, setting /proc/sys/vm/panic_on_oom >> to "1" will >> have the desired effect on non zLinux. >> >> Shane ... >> >> On Tue, Jul 27th, 2010 at 1:24 AM, Daniel Tate <[email protected]> wrote: >> >>> When websphere starts, it consumes all the memory eventually and >>> halts, but not panics, the system. >> <snip> ... >>> At this point i see two problems: >>> >>> 1) Why is OOM Kill not functioning properly >> >> ---------------------------------------------------------------------- >> For LINUX-390 subscribe / signoff / archive access instructions, >> send email to [email protected] with the message: INFO LINUX-390 or >> visit >> http://www.marist.edu/htbin/wlvindex?LINUX-390 >> ---------------------------------------------------------------------- >> For more information on Linux on System z, visit >> http://wiki.linuxvm.org/ >> > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > > ---------------------------------------------------------------------- > For LINUX-390 subscribe / signoff / archive access instructions, > send email to [email protected] with the message: INFO LINUX-390 or visit > http://www.marist.edu/htbin/wlvindex?LINUX-390 > ---------------------------------------------------------------------- > For more information on Linux on System z, visit > http://wiki.linuxvm.org/ > ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/ ---------------------------------------------------------------------- For LINUX-390 subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 ---------------------------------------------------------------------- For more information on Linux on System z, visit http://wiki.linuxvm.org/
