Can you take heapdump and check ..? Here you can check which objects are using how much..
Command : jmap -histo:live namenodepid ________________________________ From: Personal [[email protected]] Sent: Tuesday, June 18, 2013 7:20 AM To: [email protected] Subject: Re: Namenode memory usage E Lego On Mon, Jun 17, 2013 at 1:04 PM, Patai Sangbutsarakum <[email protected]<mailto:[email protected]>> wrote: Hi Hadoopers, My dedicated Namenode box has 48G of memory, 40G is allocated for NN. HEAP This is from 50070/dfshealth.jsp 28540193 files and directories, 32324098 blocks = 60864291 total. Heap Size is 23.34 GB / 38.54 GB (60%) The Heap is fluctuating between less than 20G up to almost 100% However, from top command Residence size is constantly at 39G not matter how low of the memory usage in defshealth.jsp page PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4628 apps 19 0 40.4g 39g 22m S 132.9 83.9 44821:08 /usr/java/jdk/jre/bin/java -Dproc_namenode -Xmx40000m Is this the time to upgrade the ram to the namenode box? I remember the easy rule of thumb is 150 bye of every 1M for blocks+file+dir, so 60864291 * 150byte is around 9G. I just don't understand why 40G seems to be used up.? Please educate.. Hope this make sense P
