Right. All your memory appears to be anon segments - 13.4GB worth. About 9GB of that is the shared memory segments. That leaves 4.4GB.
I see 13 Java processes listed. Assuming they have a similar memory footprint as the one pmap example, which shows about 40MB of RSS, that's (40MB x 13) about 500MB. It's of course possible that the other JVMs are using more memory than the 1 pmap example. The ps output or prstat would be helpful here. You still need to account for about 3.9GB of anon usage... Thanks, /jim Simon wrote: > Hi Michael, > > Now the system been reset seems that process disappear,but I found > some similar processes which launched by the same user "kplus",as: > > kplus 20905 0.1 1.1464984180288 ? S 08:18:38 2:17 > /usr/java/bin/java -Dprogram.name=run.sh -server -Xms128m -Xmx512m > -Dsun.rmi.dgc.client.gcInterval=3600000 > -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true > -Djava.endorsed.dirs=/export/home1/jboss/jboss-4.0.5.GA/lib/endorsed > <http://jboss-4.0.5.GA/lib/endorsed> -classpath > /export/home1/jboss/jboss-4.0.5.GA/bin/run.jar:/usr/java/lib/tools.jar > <http://jboss-4.0.5.GA/bin/run.jar:/usr/java/lib/tools.jar> org.jboss.Main > > Thanks. > Best Regards, > Simon > > > On Sat, Nov 21, 2009 at 4:22 PM, Michael Schulte > <mschulte at sunspezialist.de <mailto:mschulte at sunspezialist.de>> wrote: > > Hey Simon, > > > ># pmap -x 28447 > >28447: /export/home1/bea/jdk160_05//bin/java > -Djava.awt.headless=true -Xbootc > > Can you give the full argument list of this java? > > Memory de-allocation is Java is fully asynchronous in Garbage > Collection and can be > tuned by command line options when starting the application. Just > Google for > the exakt syntax. > > Michael > > > Simon wrote: > > Hi Jim, > > Thanks for your reply,here's my update: > > Did you check for additional virtual space usage in /tmp? > > > "df -k" shows only 1% used in "/tmp" filesystem: > swap 10943720 968 10942752 1% /tmp > swap 10942832 80 10942752 1% /var/run > > Are you using ZFS (ARC space needed for that)? > > No any zfs used,all filesystems are UFS. > > You can also try using the "::memstat" mdb dcmd to break > out kernel > memory further. > > > > ::memstat > > Page Summary Pages MB %Tot > ------------ ---------------- ---------------- ---- > Kernel 111925 874 5% > Anon 1715077 13399 83% > Exec and libs 64697 505 3% > Page cache 71828 561 3% > Free (cachelist) 51148 399 2% > Free (freelist) 43872 342 2% > > Total 2058547 16082 > Physical 2037012 15914 > > As above,the Anonymous memory is very high,I think some user > thread using the memory in an abnormal way,I checked one of > process with "pmap -x" and found many of stack/heap,as: > > # ps -ef |grep bea |grep -v grep > kplustp 28447 1 0 07:01:37 ? 0:26 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28447 1 0 07:01:37 ? 0:26 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28443 1 0 07:01:37 ? 2:29 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28445 1 0 07:01:37 ? 1:24 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28457 1 0 07:01:38 ? 0:50 > /export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m > -Djava.awt.headless=tr > kplustp 28453 1 0 07:01:37 ? 1:55 > /export/home1/bea/jdk160_05//bin/java -Xms512m -Xmx1024m > -Xbootclasspath/p:./.. > kplustp 28449 1 0 07:01:37 ? 0:25 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28508 1 0 07:01:44 ? 1:15 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -classpath ./../ > kplustp 28451 1 0 07:01:37 ? 1:25 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28455 1 0 07:01:37 ? 1:27 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28439 1 0 07:01:36 ? 0:28 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28441 1 0 07:01:36 ? 0:26 > /export/home1/bea/jdk160_05//bin/java -Djava.awt.headless=true > -Xbootclasspath/ > kplustp 28459 1 0 07:01:38 ? 0:26 > /export/home1/bea/jdk160_05//bin/java > -Djdbc.drivers=com.sybase.jdbc3.jdbc.SybD > > # pmap -x 28447 > 28447: /export/home1/bea/jdk160_05//bin/java > -Djava.awt.headless=true -Xbootc > Address Kbytes RSS Anon Locked Mode Mapped File > 00010000 48 48 - - r-x-- java > 0002A000 8 8 - - rwx-- java > 0002C000 3920 272 264 - rwx-- [ heap ] > 00400000 4096 - - - rwx-- [ heap ] > B62F8000 32 32 32 - rwx-R [ stack tid=24 ] > B647A000 8 8 8 - rwx-R [ stack tid=23 ] > B6678000 16 16 16 - rwx-R [ stack tid=21 ] > B677A000 8 8 8 - rwx-R [ stack tid=20 ] > B6878000 16 16 16 - rwx-R [ stack tid=19 ] > B68FE000 8 8 8 - rwx-R [ stack tid=18 ] > B6FFE000 8 8 8 - rwx-R [ stack tid=11 ] > B7070000 1584 1504 - - r--s- dev:85,50 > ino:129269 > B77C0000 32 32 - - r-x-- libaio.so.1 > B77D8000 8 8 - - rwx-- libaio.so.1 > B77E0000 24 24 - - r-x-- librt.so.1 > B77F6000 8 8 - - rwx-- librt.so.1 > B7800000 16384 12288 12288 - rwx-- [ anon ] > BB800000 176128 - - - rwx-- [ anon ] > E6400000 90112 8192 8192 - rwx-- [ anon ] > FBC10000 336 336 - - r-x-- libtibrv.so > FBC72000 24 24 24 - rwx-- libtibrv.so > FBD10000 24 24 - - r-x-- libnio.so > FBD24000 16 16 8 - rwx-- libnio.so > FBD30000 8 8 - - r-x-- libkstat.so.1 > FBD42000 8 8 - - rwx-- libkstat.so.1 > FBD50000 88 88 - - r-x-- libtibrvcm.so > FBD74000 16 16 8 - rwx-- libtibrvcm.so > FBE10000 24 24 - - r-x-- libtibrvft.so > FBE24000 8 8 - - rwx-- libtibrvft.so > FBE30000 48 48 - - r-x-- libtibrvcmq.so > FBE4A000 8 8 - - rwx-- libtibrvcmq.so > FBE50000 72 56 - - r-x-- libnet.so > FBE70000 8 8 - - rwx-- libnet.so > FBE80000 344 - - - rwx-- [ anon ] > FBFE0000 32 32 - - r-x-- libtibrvj.so > FBFF0000 16 16 - - r--s- dev:85,50 > ino:128526 > FBFF6000 16 8 - - rwx-- libtibrvj.so > FC000000 4096 4096 4096 - rwx-- [ anon ] > FE010000 32 32 - - r--s- dev:85,50 > ino:129270 > FE020000 16 16 - - r-x-- libpthread.so.1 > FE030000 16 16 - - r-x-- libKtpcrypt.so > FE042000 16 8 - - rwx-- libKtpcrypt.so > FE04C000 160 160 - - r--s- dev:85,60 > ino:274206 > FE080000 32 - - - rwx-- [ anon ] > FE0A0000 344 - - - rwx-- [ anon ] > FE1F6000 176 8 8 - rwx-- [ anon ] > FE2A2000 8 - - - rwx-- [ anon ] > FE2B0000 32 32 - - r--s- dev:85,50 > ino:128648 > FE2C0000 32 24 - - r--s- dev:85,60 > ino:314837 > FE2D2000 152 144 - - r--s- dev:85,60 > ino:314730 > FE300000 24 - - - rwx-- [ anon ] > FE390000 32 32 - - r--s- dev:85,60 > ino:314841 > FE3A0000 32 - - - rwx-- [ anon ] > FE3D0000 64 64 - - r-x-- libzip.so > FE3E0000 8 - - - rwx-- libzip.so > FE3E8000 16 - - - r--s- dev:85,60 > ino:274380 > FE3F0000 152 136 - - r-x-- libjava.so > FE418000 24 - - - r--s- dev:85,60 > ino:274288 > FE420000 8 - - - r--s- dev:85,60 > ino:134780 > FE426000 8 - - - rwx-- libjava.so > FE430000 56 56 - - r-x-- libverify.so > FE440000 40 40 - - r--s- dev:85,5 ino:29959 > FE44E000 8 - - - rwx-- libverify.so > FE460000 64 - - - rwx-- [ anon ] > FE510000 32 32 - - r-x-- libhpi.so > FE520000 8 8 8 - rwx-- [ anon ] > FE528000 8 - - - rwx-- libhpi.so > FE52A000 8 - - - rwx-- libhpi.so > FE530000 64 56 56 - rwx-- [ anon ] > FE550000 64 - - - rw--- [ anon ] > FE570000 64 32 32 - rw--- [ anon ] > FE590000 16 16 - - r-x-- libmp.so.2 > FE5A0000 8 8 8 - rwx-- [ anon ] > FE5A4000 8 - - - rwx-- libmp.so.2 > FE5B0000 80 80 - - r-x-- libmd.so.1 > FE5CC000 16 16 - - r--s- dev:85,60 > ino:314729 > FE5D4000 8 8 - - rwx-- libmd.so.1 > FE5E0000 24 24 - - r-x-- libgen.so.1 > FE5E8000 32 32 - - r--s- dev:85,60 > ino:314782 > FE5F6000 8 8 - - rwx-- libgen.so.1 > FE600000 680 680 - - r-x-- libm.so.2 > FE6B0000 8 - - - r--s- dev:85,60 > ino:134781 > FE6B8000 32 32 - - rwx-- libm.so.2 > FE6C2000 96 96 - - r--s- dev:85,60 > ino:314727 > FE6E0000 32 32 - - r-x-- libuutil.so.1 > FE6F0000 16 16 - - r--s- dev:85,60 > ino:314707 > FE6F8000 8 8 - - rwx-- libuutil.so.1 > FE700000 584 584 - - r-x-- libnsl.so.1 > FE7A2000 40 40 - - rwx-- libnsl.so.1 > FE7AC000 24 - - - rwx-- libnsl.so.1 > FE7C0000 16 16 - - r--s- dev:85,60 > ino:314725 > FE7D0000 96 96 - - r-x-- libscf.so.1 > FE7F0000 8 8 8 - rwx-- [ anon ] > FE7F8000 8 8 - - rwx-- libscf.so.1 > FE800000 9064 8616 - - r-x-- libjvm.so > FF0E0000 32 16 - - rw-s- dev:324,2 > ino:106631901 > FF0EA000 280 80 80 - rwx-- libjvm.so > FF130000 88 56 56 - rwx-- libjvm.so > FF150000 8 8 8 - rwx-- [ anon ] > FF160000 8 8 - - r-x-- libdoor.so.1 > FF172000 8 8 - - rwx-- libdoor.so.1 > FF180000 56 56 - - r-x-- libCrun.so.1 > FF190000 8 8 8 - rwx-- [ anon ] > FF19C000 8 8 - - rwx-- libCrun.so.1 > FF19E000 24 - - - rwx-- libCrun.so.1 > FF1B0000 16 8 - - r-x-- libm.so.1 > FF1C2000 8 - - - rwx-- libm.so.1 > FF1D0000 48 48 - - r-x-- libsocket.so.1 > FF1E0000 8 8 - - r---- [ anon ] > FF1EC000 8 8 8 - rwx-- libsocket.so.1 > FF1F0000 8 8 - - r-x-- libsched.so.1 > FF200000 1208 1208 - - r-x-- libc.so.1 > FF330000 24 8 8 - rwx-- [ anon ] > FF33E000 40 40 32 - rwx-- libc.so.1 > FF348000 8 8 8 - rwx-- libc.so.1 > FF350000 8 8 - - r-x-- libdl.so.1 > FF35C000 16 16 - - r--s- dev:85,60 > ino:314813 > FF362000 8 8 - - rwx-- libdl.so.1 > FF370000 32 24 - - r-x-- libjli.so > FF380000 8 8 8 - rwx-- [ anon ] > FF386000 16 8 - - rwx-- libjli.so > FF390000 8 8 - - r-x-- libc_psr.so.1 > FF3A0000 16 16 - - r-x-- libthread.so.1 > FF3B0000 208 208 - - r-x-- ld.so.1 > FF3E8000 8 - - - r--s- dev:85,60 > ino:274115 > FF3F0000 8 8 8 - rwx-- [ anon ] > FF3F4000 8 8 8 - rwx-- ld.so.1 > FF3F6000 8 8 8 - rwx-- ld.so.1 > FF3FA000 8 8 - - rwxs- [ anon ] > FFBFA000 24 8 8 - rwx-- [ stack ] > -------- ------- ------- ------- ------- > total Kb 312632 40560 25344 - > > Other processes initialized by user "kplustp" has similar > memory usage as above. > > Thanks. > Best Regards, > Simon > > On Fri, Nov 20, 2009 at 10:02 PM, Jim Fiori <Jim.Fiori at sun.com > <mailto:Jim.Fiori at sun.com> <mailto:Jim.Fiori at sun.com > <mailto:Jim.Fiori at sun.com>>> wrote: > > Simon, > > For a 16GB box, the page scanner kicks in when freemem > drops below > 1/64th of memory, or about 256MB. Doesn't matter if the > system is > idle or not. > > The 'w' column numbers mean that threads were swapped out > at some > point in the past because of a severe memory shortage and never > swapped backed in (because they've not been awoken yet). So > it's > normal for that column to stay high even if much of the > memory was > released. > > It looks to me like you're just oversubscribing memory. If > you look > at the prstat output I see easily 13-14GB of physical > memory in use, > plus you have the kernel memory. As for virtual memory, > about 23GB > shows up at least. > > Did you check for additional virtual space usage in /tmp? > > Are you using ZFS (ARC space needed for that)? > > You can also try using the "::memstat" mdb dcmd to break > out kernel > memory further. > > Jim > > Simon wrote: > > Hi Experts, > > Here's the performance related question,please help to > review > what can I > do to get the issue fixed ? > > IHAC who has one M5000 with Solaris 10 10/08(KJP: > 138888-01) > installed > and 16GB RAM configured,running sybase ASE 12.5 and JBOSS > application,recently,they felt the OS got very slow > after OS > running for > some sime,collected vmstat data points out memory > shortage,as: > > # vmstat 5 > kthr memory page disk faults cpu > r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy > cs us sy id > 0 0 153 6953672 254552 228 228 1843 1218 1687 0 685 3 2 > 0 0 2334 > 32431 3143 1 1 97 > 0 0 153 6953672 259888 115 115 928 917 917 0 264 0 35 0 > 2 2208 > 62355 3332 7 3 90 > 0 0 153 6953672 255688 145 145 1168 1625 1625 0 1482 0 > 6 1 0 > 2088 40113 3070 2 1 96 > 0 0 153 6953640 256144 111 111 894 1371 1624 0 1124 0 6 > 0 0 2080 > 55278 3106 3 3 94 > 0 0 153 6953640 256048 241 241 1935 2585 3035 0 1009 0 > 18 0 0 > 2392 40643 3164 2 2 96 > 0 0 153 6953648 257112 236 235 1916 1710 1710 0 1223 0 > 7 0 0 > 2672 62582 3628 3 4 93 > > As above,the "w" column is very high all time,and "sr" > column > also kept > very high,which indicates the page scanner is activated and > busying for > page out,but the CPU is very idle,checked > "/etc/system",found one > improper entry: > set shmsys:shminfo_shmmax = 0xffffffffffff > > So I think it's the improper share memory setting to > cause too many > physical RAM was reserved by application and suggest to > adjustment the > share memory to 8GB(0x200000000),but as customer > feedback,seems > it got > worst result based on new vmstat output: > > kthr memory page disk faults cpu > r b w swap free re mf pi po fr de sr m0 m1 m4 m5 in sy > cs us sy id > 0 6 762 3941344 515848 18 29 4544 0 0 0 0 4 562 0 1 > 2448 25687 > 3623 1 2 97 > 0 6 762 4235016 749616 66 21 4251 2 2 0 0 0 528 0 0 > 2508 50540 > 3733 2 5 93 > 0 6 762 4428080 889864 106 299 4694 0 0 0 0 1 573 0 7 2741 > 182274 3907 10 4 86 > 0 5 762 4136400 664888 19 174 4126 0 0 0 0 6 511 0 0 > 2968 241186 > 4417 18 9 73 > 0 7 762 3454280 193776 103 651 2526 3949 4860 0 121549 > 11 543 0 > 5 2808 149820 4164 10 12 78 > 0 9 762 3160424 186016 61 440 1803 7362 15047 0 189720 > 12 567 0 > 5 3101 119895 4125 6 13 81 > 0 6 762 3647456 403056 44 279 4260 331 331 0 243 10 540 > 0 3 2552 > 38374 3847 5 3 92 > > the "w" & "sr" value increased instead,why ? > > And I also attached the "prstat" outout,it's a prstat > snapshot after > share memory adjustment,please help to have a look ? > what can I > do next > to get the issue solved ? what's the possible factors > to cause > memory > shortage again and again,even they have 16GB RAM + 16GB > Swap the > physical RAM really shortage? > Or is there any useful dtrace script to trace the problem ? > Thanks very much ! > > Best Regards, > Simon > > > > > ------------------------------------------------------------------------ > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > perf-discuss mailing list > perf-discuss at opensolaris.org > <mailto:perf-discuss at opensolaris.org> > <mailto:perf-discuss at opensolaris.org > <mailto:perf-discuss at opensolaris.org>> > > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > perf-discuss mailing list > perf-discuss at opensolaris.org <mailto:perf-discuss at > opensolaris.org> > > > > -- > Michael Schulte > mschulte at sunspezialist.de <mailto:mschulte at sunspezialist.de> > OpenSolaris Kernel Development > http://opensolaris.org/ > > _______________________________________________ > perf-discuss mailing list > perf-discuss at opensolaris.org <mailto:perf-discuss at opensolaris.org> > > > ------------------------------------------------------------------------ > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
