After starting the hbase in jroсkit found the same memory leakage. After the launch
Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head Tue Jan 11 16:49:31 2011 11 16:49:31 MSK 2011 PID RSS VSZ% CPU 7863 2547760 5576744 78.7 JR dumps: Total mapped 5576740KB (reserved = 2676404KB) - Java heap 2048000KB (reserved = 1472176KB) - GC tables 68512KB - Thread stacks 37236KB (# threads = 111) - Compiled code 1048576KB (used = 2599KB) - Internal 1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB (malloced = 1110KB # 3285) - Java class data 20224KB (malloced = 20002KB # 15134 in 3285 classes) - Native memory tracking 1024KB (malloced = 325KB +10 KB # 20) After running the mr which make high write load (~1hour) Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head Tue Jan 11 17:08:56 2011 11 17:08:56 MSK 2011 PID RSS VSZ% CPU 7863 4072396 5459572 100 JR said not important below specify why) http://paste.ubuntu.com/552820/ <http://paste.ubuntu.com/552820/> 7863: Total mapped 5742628KB +165888KB (reserved=1144000KB -1532404KB) - Java heap 2048000KB (reserved=0KB -1472176KB) - GC tables 68512KB - Thread stacks 38028KB +792KB (#threads=114 +3) - Compiled code 1048576KB (used=3376KB +776KB) - Internal 1480KB +256KB - OS 517944KB -31744KB - Other 1996792KB +195816KB - Classblocks 1280KB (malloced=1156KB +45KB #3421 +136) - Java class data 20992KB +768KB (malloced=20843KB +840KB #15774 +640 in 3421 classes) - Native memory tracking 1024KB (malloced=325KB +10KB #20) +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ OS *java r x 0x0000000000400000.( 76KB) OS *java rw 0x0000000000612000 ( 4KB) OS *[heap] rw 0x0000000000613000.( 478712KB) INT Poll r 0x000000007fffe000 ( 4KB) INT Membar rw 0x000000007ffff000.( 4KB) MSP Classblocks (1/2) rw 0x0000000082ec0000 ( 768KB) MSP Classblocks (2/2) rw 0x0000000082f80000 ( 512KB) HEAP Java heap rw 0x0000000083000000.(2048000KB) rw 0x00007f2574000000 ( 65500KB) 0x00007f2577ff7000.( 36KB) rw 0x00007f2584000000 ( 65492KB) 0x00007f2587ff5000.( 44KB) rw 0x00007f258c000000 ( 65500KB) 0x00007f258fff7000 ( 36KB) rw 0x00007f2590000000 ( 65500KB) 0x00007f2593ff7000 ( 36KB) rw 0x00007f2594000000 ( 65500KB) 0x00007f2597ff7000 ( 36KB) rw 0x00007f2598000000 ( 131036KB) 0x00007f259fff7000 ( 36KB) rw 0x00007f25a0000000 ( 65528KB) 0x00007f25a3ffe000 ( 8KB) rw 0x00007f25a4000000 ( 65496KB) 0x00007f25a7ff6000 ( 40KB) rw 0x00007f25a8000000 ( 65496KB) 0x00007f25abff6000 ( 40KB) rw 0x00007f25ac000000 ( 65504KB) So, the difference was in the pieces of memory like this: rw 0x00007f2590000000 (65500KB) 0x00007f2593ff7000 (36KB) Looks like HLog allocates memory (looks like HLog, becase it is very similar size) If we count this blocks we get amount of lost memory: 65M * 32 + 132M = 2212M So, it looks like HLog allcates to many memory, and question is: how to restrict it? 2010/12/30 Andrey Stepachev <[email protected]> > Hi All. > > After heavy load into hbase (single node, nondistributed test system) I got > 4Gb process size of my HBase java process. > On 6GB machine there was no room for anything else (disk cache and so on). > > Does anybody knows, what is going on, and how you solve this. What heap > memory is set on you hosts > and how much of RSS hbase process actually use. > > I don't see such things before, all tomcat and other java apps don't eats > significally more memory then -Xmx. > > Connection name: pid: 23476 org.apache.hadoop.hbase.master.HMaster > start Virtual Machine: Java HotSpot(TM) 64-Bit Server VM version > 17.1-b03 Vendor: Sun Microsystems Inc. Name: 23...@mars Uptime: > 12 > hours 4 minutes Process CPU time: 5 hours 45 minutes JIT compiler: > HotSpot > 64-Bit Server Compiler Total compile time: 19,223 seconds > ------------------------------ > Current heap size: 703 903 kbytes Maximum heap size: 2 030 > 976kbytes Committed memory: > 2 030 976 kbytes Pending finalization: 0 objects Garbage > collector: Name = 'ParNew', Collections = 9 990, Total time spent = 5 > minutes Garbage collector: Name = 'ConcurrentMarkSweep', Collections = > 20, Total time spent = 35,754 seconds > ------------------------------ > Operating System: Linux 2.6.34.7-0.5-xen Architecture: amd64 > Number of processors: > 8 Committed virtual memory: 4 403 512 kbytes Total physical > memory: 6 815 744 kbytes Free physical memory: 82 720 kbytes Total > swap space: > 8 393 924 kbytes Free swap space: 8 050 880 kbytes > > > >
