The two RS could have died due to GC or some connectivity issues with ZK. Checking the logs would help us get a clear idea.
REgards Ram On Thu, Feb 21, 2013 at 7:09 AM, Joarder KAMAL <[email protected]> wrote: > Hi Ted, > > After login into the HBase UI I found two of my region servers were dead. > Don't have any clue why. Master is still up and running for the past 4 > days. > Memory is still running pretty low without any workload and two region > servers!! > > -------------------------------------------------- > > root@hdb1-01:/home/ubuntu# jps > 1064 NameNode > 1578 JobTracker > 10770 Jps > 2481 HRegionServer > 2261 HMaster > 1280 DataNode > 1769 TaskTracker > 2197 HQuorumPeer > 1480 SecondaryNameNode > -------------------------------------------------- > root@hdb1-02:/home/ubuntu# jps > 1492 HRegionServer > 1315 TaskTracker > 1137 DataNode > 3290 Jps > -------------------------------------------------- > root@hdb1-03:/home/ubuntu# jps > 1527 HRegionServer > 1171 DataNode > 7475 Jps > 1349 TaskTracker > > -------------------------------------------------- > PID found 2197, 2261 and 2481 for 'hbase' > -------------------------------------------------- > root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2197 > 2197 1 52248 1938732 > -------------------------------------------------- > root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2261 > 2261 1 108724 2007912 > -------------------------------------------------- > root@hdb1-01:/home/ubuntu# ps axo pid,ppid,rss,vsz | grep 2481 > 2481 1 686252 2031376 > -------------------------------------------------- > root@hdb1-01:/home/ubuntu# free -m > total used free shared buffers cached > Mem: 1652 1616 36 0 51 305 > > -/+ buffers/cache: 1259 393 > Swap: 895 8 887 > > > Thanks a lot again ... > > > Regards, > Joarder Kamal > > > > On 21 February 2013 11:08, Ted Yu <[email protected]> wrote: > > > What do you see when you use 'ps aux | grep <pid>' command ? > > A sample is shown below: > > > > USER PID %CPU %MEM VSZ RSS TT STAT STARTED > > TIME COMMAND > > tyu 16468 232.2 1.4 4974136 240528 s002 R+ 4:05PM > > 2:11.15 > > > > > /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/javadoc > > -J-Xmx2048m @options @packages > > > > Please give us the VSZ and RSS readings. > > > > Thanks > > > > On Wed, Feb 20, 2013 at 3:56 PM, Joarder KAMAL <[email protected]> > wrote: > > > > > Sorry may be I didn't clarify the question properly. > > > > > > Q2. As per the default setting maxHeapMB size is 991, so in saturation > > > condition HBase can't use more RAM (even if it is available, for > example > > > the system memory size is 4GB) than this defined size, is it right? > > > > > > > > > Many thanks again... > > > > > > On 21 February 2013 10:48, Ted Yu <[email protected]> wrote: > > > > > > > I am not familiar with monitoring tools on EC2. > > > > > > > > You can use Ganglia to monitor hadoop / HBase. > > > > > > > > Cheers > > > > > > > > On Wed, Feb 20, 2013 at 3:42 PM, Joarder KAMAL <[email protected]> > > > wrote: > > > > > > > > > Hi Ted, > > > > > > > > > > Thanks for the reply. > > > > > > > > > > Q1. Cleared now. Sorry I didn't count the -ROOT- and .META. tables > > > > > Q2. Not clear. Could explain a bit more? I am using the default > > > setting. > > > > > > > > > > Thanks again. > > > > > > > > > > > > > > > On 21 February 2013 10:36, Ted Yu <[email protected]> wrote: > > > > > > > > > > > bq. but when I click to view the table details it shows there > are 5 > > > > > > regions. > > > > > > > > > > > > Did you include -ROOT- and .META. tables above ? > > > > > > > > > > > > bq. as you can see the maxHeapMB size is 991 > > > > > > > > > > > > Attachment is automatically stripped from your email. FYI > > > > > > > > > > > > On Wed, Feb 20, 2013 at 3:26 PM, Joarder KAMAL < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Dear All, > > > > > > > > > > > > > > I am currently running a 3 node HBase cluster with Amazon EC2 > > small > > > > > > > instance (1CPU, 1.7GB Mem). After running a simple workload > (type > > > A) > > > > > > using > > > > > > > YCSB benchmarking tool, I noticed in the Web UI that the number > > of > > > > > > regions > > > > > > > in the 3 region servers are 7 but when I click to view the > table > > > > > details > > > > > > it > > > > > > > shows there are 5 regions. I am a bit confuse. Does the Web UI > > > > showing > > > > > > > wrong results? How can I check region details from the shell/is > > > there > > > > > any > > > > > > > other command line tools for that? I am using Hbase 0.94.3 (as > it > > > is > > > > > the > > > > > > > latest version compatible with YCSB at this moment) with > > > > Hadoop-1.0.4. > > > > > > > > > > > > > > Another thing, as you can see the maxHeapMB size is 991 > (default > > > > > setting) > > > > > > > then whenever I am running workloads the server free memory > > status > > > is > > > > > > > terrible - almost 0MB for 1M/10M transactions > (YCSB/Workload-A). > > My > > > > > > > question is if the maxHeapMB is fixed at 1GB then why I am > > loosing > > > > > extra > > > > > > > 700MB memory? How can I monitor cpu/memory/io usage for > > individual > > > > > > cluster > > > > > > > components (HBase/Hadoop/MapRed/ZK, etc.)? Is there any tool > > > > available > > > > > > with > > > > > > > HBase? > > > > > > > > > > > > > > Thanks a lot. > > > > > > > > > > > > > > > > > > > > > [image: Inline images 1] > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > Joarder Kamal > > > > > > > > > > > > > > ------------------------------------------ > > > > > > > PhD Student > > > > > > > Gippsland School of Information Technology, > > > > > > > Faculty of Information Technology, > > > > > > > MONASH University > > > > > > > Churchill VIC, Australia. 3842 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
