Hi, Have you tried the ASF version of hadoop-core? (The one distributed with HBase 0.90RC.)
It doesn't call reinit() so I'm hoping it will just work fine with the latest hadoop-lzo and other compressors. Thanks, -- Tatsuya Kawano (Mr.) Tokyo, Japan On Jan 12, 2011, at 7:51 PM, Friso van Vollenhoven <[email protected]> wrote: > Thanks. > > I went back to hbase 0.89 with 0.1 LZO, which works fine and does not show > this issue. > > I tried with a newer Hbase and LZO version, also with the MALLOC... setting > but without max direct memory set, so I was wondering whether you need a > combination of the two to fix things (apparently not). > > Now i am wondering whether I did something wrong setting the env var. It > should just be picked up when it's in hbase-env.sh, right? > > > Friso > > > > On 12 jan 2011, at 10:59, Andrey Stepachev wrote: > >> with MALLOC_ARENA_MAX=2 >> >> I check -XX:MaxDirectMemorySize=256m, before, but it doesn't affect anything >> (even no OOM >> exceptions or so on). >> >> But it looks like i have exactly the same issue (it looks like). I have many >> 64Mb anon memory blocks. >> (sometimes they 132MB). And on heavy load i have rapidly growing rss size of >> jvm process. >> >> 2011/1/12 Friso van Vollenhoven <[email protected]> >> >>> Just to clarify: you fixed it by setting the MALLOC_MAX_ARENA=? in >>> hbase-env.sh? >>> >>> Did you also use the -XX:MaxDirectMemorySize=256m ? >>> >>> It would be nice to check that this is a different than the leakage with >>> LZO... >>> >>> >>> Thanks, >>> Friso >>> >>> >>> On 12 jan 2011, at 07:46, Andrey Stepachev wrote: >>> >>>> My bad. All things work. Thanks for Todd Lipcon :) >>>> >>>> 2011/1/11 Andrey Stepachev <[email protected]> >>>> >>>>> I tried to set MALLOC_ARENA_MAX=2. But still the same issue like in LZO >>>>> problem thread. All those 65M blocks here. And JVM continues to eat >>> memory >>>>> on heavy write load. And yes, I use "improved" kernel >>>>> Linux 2.6.34.7-0.5. >>>>> >>>>> 2011/1/11 Xavier Stevens <[email protected]> >>>>> >>>>> Are you using a newer linux kernel with the new and "improved" memory >>>>>> allocator? >>>>>> >>>>>> If so try setting this in hadoop-env.sh: >>>>>> >>>>>> export MALLOC_ARENA_MAX=<number of cores you want to use> >>>>>> >>>>>> Maybe start by setting it to 4. You can thank Todd Lipcon if this >>> works >>>>>> for you. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> >>>>>> -Xavier >>>>>> >>>>>> On 1/11/11 7:24 AM, Andrey Stepachev wrote: >>>>>>> No. I don't use LZO. I tried even remove any native support (i.e. all >>>>>> .so >>>>>>> from class path) >>>>>>> and use java gzip. But nothing. >>>>>>> >>>>>>> >>>>>>> 2011/1/11 Friso van Vollenhoven <[email protected]> >>>>>>> >>>>>>>> Are you using LZO by any chance? If so, which version? >>>>>>>> >>>>>>>> Friso >>>>>>>> >>>>>>>> >>>>>>>> On 11 jan 2011, at 15:57, Andrey Stepachev wrote: >>>>>>>> >>>>>>>>> After starting the hbase in jroсkit found the same memory leakage. >>>>>>>>> >>>>>>>>> After the launch >>>>>>>>> >>>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head >>>>>>>>> Tue Jan 11 16:49:31 2011 >>>>>>>>> >>>>>>>>> 11 16:49:31 MSK 2011 >>>>>>>>> PID RSS VSZ% CPU >>>>>>>>> 7863 2547760 5576744 78.7 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> JR dumps: >>>>>>>>> >>>>>>>>> Total mapped 5576740KB (reserved = 2676404KB) - Java heap 2048000KB >>>>>>>>> (reserved = 1472176KB) - GC tables 68512KB - Thread stacks 37236KB >>> (# >>>>>>>>> threads = 111) - Compiled code 1048576KB (used = 2599KB) - Internal >>>>>>>>> 1224KB - OS 549688KB - Other 1800976KB - Classblocks 1280KB >>> (malloced >>>>>>>>> = 1110KB # 3285) - Java class data 20224KB (malloced = 20002KB # >>> 15134 >>>>>>>>> in 3285 classes) - Native memory tracking 1024KB (malloced = 325KB >>> +10 >>>>>>>>> KB # 20) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> After running the mr which make high write load (~1hour) >>>>>>>>> >>>>>>>>> Every 2,0 s: date & & ps - sort =- rss-eopid, rss, vsz, pcpu | head >>>>>>>>> Tue Jan 11 17:08:56 2011 >>>>>>>>> >>>>>>>>> 11 17:08:56 MSK 2011 >>>>>>>>> PID RSS VSZ% CPU >>>>>>>>> 7863 4072396 5459572 100 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> JR said not important below specify why) >>>>>>>>> >>>>>>>>> http://paste.ubuntu.com/552820/ >>>>>>>>> <http://paste.ubuntu.com/552820/> >>>>>>>>> >>>>>>>>> >>>>>>>>> 7863: >>>>>>>>> Total mapped 5742628KB +165888KB >>> (reserved=1144000KB >>>>>>>>> -1532404KB) >>>>>>>>> - Java heap 2048000KB (reserved=0KB >>>>>>>> -1472176KB) >>>>>>>>> - GC tables 68512KB >>>>>>>>> - Thread stacks 38028KB +792KB (#threads=114 +3) >>>>>>>>> - Compiled code 1048576KB (used=3376KB >>> +776KB) >>>>>>>>> - Internal 1480KB +256KB >>>>>>>>> - OS 517944KB -31744KB >>>>>>>>> - Other 1996792KB +195816KB >>>>>>>>> - Classblocks 1280KB (malloced=1156KB >>>>>>>>> +45KB #3421 +136) >>>>>>>>> - Java class data 20992KB +768KB (malloced=20843KB >>>>>>>>> +840KB #15774 +640 in 3421 classes) >>>>>>>>> - Native memory tracking 1024KB (malloced=325KB >>>>>> +10KB >>>>>>>> #20) >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>>>>>>>> OS *java r x 0x0000000000400000.( >>>>>>>> 76KB) >>>>>>>>> OS *java rw 0x0000000000612000 ( >>>>>>>> 4KB) >>>>>>>>> OS *[heap] rw 0x0000000000613000.( >>>>>>>> 478712KB) >>>>>>>>> INT Poll r 0x000000007fffe000 ( >>>>>>>> 4KB) >>>>>>>>> INT Membar rw 0x000000007ffff000.( >>>>>>>> 4KB) >>>>>>>>> MSP Classblocks (1/2) rw 0x0000000082ec0000 ( >>>>>>>> 768KB) >>>>>>>>> MSP Classblocks (2/2) rw 0x0000000082f80000 ( >>>>>>>> 512KB) >>>>>>>>> HEAP Java heap rw >>>>>>>> 0x0000000083000000.(2048000KB) >>>>>>>>> rw 0x00007f2574000000 ( >>>>>>>> 65500KB) >>>>>>>>> 0x00007f2577ff7000.( >>>>>>>> 36KB) >>>>>>>>> rw 0x00007f2584000000 ( >>>>>>>> 65492KB) >>>>>>>>> 0x00007f2587ff5000.( >>>>>>>> 44KB) >>>>>>>>> rw 0x00007f258c000000 ( >>>>>>>> 65500KB) >>>>>>>>> 0x00007f258fff7000 ( >>>>>>>> 36KB) >>>>>>>>> rw 0x00007f2590000000 ( >>>>>>>> 65500KB) >>>>>>>>> 0x00007f2593ff7000 ( >>>>>>>> 36KB) >>>>>>>>> rw 0x00007f2594000000 ( >>>>>>>> 65500KB) >>>>>>>>> 0x00007f2597ff7000 ( >>>>>>>> 36KB) >>>>>>>>> rw 0x00007f2598000000 ( >>>>>>>> 131036KB) >>>>>>>>> 0x00007f259fff7000 ( >>>>>>>> 36KB) >>>>>>>>> rw 0x00007f25a0000000 ( >>>>>>>> 65528KB) >>>>>>>>> 0x00007f25a3ffe000 ( >>>>>>>> 8KB) >>>>>>>>> rw 0x00007f25a4000000 ( >>>>>>>> 65496KB) >>>>>>>>> 0x00007f25a7ff6000 ( >>>>>>>> 40KB) >>>>>>>>> rw 0x00007f25a8000000 ( >>>>>>>> 65496KB) >>>>>>>>> 0x00007f25abff6000 ( >>>>>>>> 40KB) >>>>>>>>> rw 0x00007f25ac000000 ( >>>>>>>> 65504KB) >>>>>>>>> >>>>>>>>> >>>>>>>>> So, the difference was in the pieces of memory like this: >>>>>>>>> >>>>>>>>> rw 0x00007f2590000000 (65500KB) >>>>>>>>> 0x00007f2593ff7000 (36KB) >>>>>>>>> >>>>>>>>> >>>>>>>>> Looks like HLog allocates memory (looks like HLog, becase it is very >>>>>>>> similar >>>>>>>>> size) >>>>>>>>> >>>>>>>>> If we count this blocks we get amount of lost memory: >>>>>>>>> >>>>>>>>> 65M * 32 + 132M = 2212M >>>>>>>>> >>>>>>>>> So, it looks like HLog allcates to many memory, and question is: how >>>>>> to >>>>>>>>> restrict it? >>>>>>>>> >>>>>>>>> 2010/12/30 Andrey Stepachev <[email protected]> >>>>>>>>> >>>>>>>>>> Hi All. >>>>>>>>>> >>>>>>>>>> After heavy load into hbase (single node, nondistributed test >>> system) >>>>>> I >>>>>>>> got >>>>>>>>>> 4Gb process size of my HBase java process. >>>>>>>>>> On 6GB machine there was no room for anything else (disk cache and >>> so >>>>>>>> on). >>>>>>>>>> Does anybody knows, what is going on, and how you solve this. What >>>>>> heap >>>>>>>>>> memory is set on you hosts >>>>>>>>>> and how much of RSS hbase process actually use. >>>>>>>>>> >>>>>>>>>> I don't see such things before, all tomcat and other java apps >>> don't >>>>>>>> eats >>>>>>>>>> significally more memory then -Xmx. >>>>>>>>>> >>>>>>>>>> Connection name: pid: 23476 >>> org.apache.hadoop.hbase.master.HMaster >>>>>>>>>> start Virtual Machine: Java HotSpot(TM) 64-Bit Server VM >>> version >>>>>>>>>> 17.1-b03 Vendor: Sun Microsystems Inc. Name: 23...@mars >>>>>>>> Uptime: 12 >>>>>>>>>> hours 4 minutes Process CPU time: 5 hours 45 minutes JIT >>>>>> compiler: >>>>>>>> HotSpot >>>>>>>>>> 64-Bit Server Compiler Total compile time: 19,223 seconds >>>>>>>>>> ------------------------------ >>>>>>>>>> Current heap size: 703 903 kbytes Maximum heap size: 2 >>> 030 >>>>>>>> 976kbytes Committed memory: >>>>>>>>>> 2 030 976 kbytes Pending finalization: 0 objects Garbage >>>>>>>>>> collector: Name = 'ParNew', Collections = 9 990, Total time spent >>> = >>>>>> 5 >>>>>>>>>> minutes Garbage collector: Name = 'ConcurrentMarkSweep', >>>>>> Collections >>>>>>>> = >>>>>>>>>> 20, Total time spent = 35,754 seconds >>>>>>>>>> ------------------------------ >>>>>>>>>> Operating System: Linux 2.6.34.7-0.5-xen Architecture: >>> amd64 >>>>>>>> Number of processors: >>>>>>>>>> 8 Committed virtual memory: 4 403 512 kbytes Total physical >>>>>>>>>> memory: 6 815 744 kbytes Free physical memory: 82 720 >>> kbytes >>>>>>>> Total swap space: >>>>>>>>>> 8 393 924 kbytes Free swap space: 8 050 880 kbytes >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >>> >>> >
