Re: Fwd: HBase Pseudo mode - RegionServer disconnects after some time

Fri, 18 Oct 2013 09:01:56 -0700

Ooh no :(. Sorry Laxmi. I had same issue. I said wrong settings :) I should set up 

- hbase.client.scanner.caching (My value: 200)
- hbase.regionserver.handler.count (My Value: 20)
- hbase.client.write.buffer (My Value:20971520)

You should set your region load. It will be solved.

Talat


18-10-2013 17:52 tarihinde, A Laxmi yazdı:

Hi Talat,
I am sorry to say - I have not fixed it yet. I have spent literally sleepless nights to debug that issue. No matter what I do, RegionServer always used to get disconnected. :(
Since now I have a deadline in two days, I will go with your advise to one of my email to use HBase *standalone *mode since I am crawling about 10 urls to achieve about 300,000 urls. Once I get that done, I will retry debugging the regionserver issue. I remember you use Hadoop 1.x and not 0.20.205.0 like me, I am not sure if there is any bug with I am using? 

Thanks,
Laxmi
On Fri, Oct 18, 2013 at 10:47 AM, Talat UYARER <[email protected] <mailto:[email protected]>> wrote: 

    Hi Laxmi,
    You are welcome. I know very well that feel. I havent used
    Cloudera. I use pure install hadoop-hbase cluster base on Centos.
    I am happy because of you fixed your issue.

    Talat


    18-10-2013 17:36 tarihinde, A Laxmi yazdı:

    Thanks for the article, Talat! It was so annoying to see
    RegionSever getting disconnected under heavy load while
    everything works. Have you used Cloudera for Nutch?


    On Thu, Oct 17, 2013 at 6:50 PM, Talat UYARER
    <[email protected] <mailto:[email protected]>> wrote:

        Hi Laxmi,

        It didnt come me. I understand, your RegionServer is gone
        away. Because of this problem is your Hbase heap size or
        xceivers count is not enough. I had same issue. I set up my
        xceivers count. I am not sure what it will be count.But you
        should check your heaps size usage. If it is enough you can
        set up your property. This article[1] very good about this
        property.

        [1] http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/

        Talat

        17-10-2013 22:20 tarihinde, A Laxmi yazdı:

        Hi Talat,

        I hope all is well. Could you please advise me on this issue?

        Thanks for your help!

        ---------- Forwarded message ----------
        From: *A Laxmi* <[email protected]
        <mailto:[email protected]>>
        Date: Tue, Oct 15, 2013 at 12:07 PM
        Subject: HBase Pseudo mode - RegionServer disconnects after
        some time
        To: [email protected] <mailto:[email protected]>


        Hi -

        Please find the below log of HBase-master. I have tried all
        sorts of fixes mentioned in various threads yet I could not
        overcome this issue. I made sure I dont have 127.0.1.1 in
        /etc/hosts file. I pinged my localhost (hostname) which
        gives back the actual IP and not 127.0.0.1 using ping -c 1
        localhost. I have 'localhost' in my /etc/hostname and actual
        IP address mapped to localhost.localdomain and localhost as
        alias - something like

        /etc/hosts -

        192.***.*.*** localhost.localdomain localhost

        /etc/hostname -

        localhost

        I am using *Hadoop 0.20.205.0 and HBase 0.90.6 in Pseudo
        mode* for storing crawled data from a crawler - Apache Nutch
        2.2.1. I can start Hadoop and HBase and when I do jps it
        shows all good, then after that when I start Nutch crawl
        after about 40mins of crawling or so, I can see Nutch
        hanging up while in about 4th iteration of parsing and at
        the same time when I do jps in HBase, I can see everything
        except HRegionServer. Below is the log.

        I tried all possible ways but couldn't overcome this issue.
        I really need someone from HBase list to help me with this
        issue.


        2013-10-15 02:02:08,285 DEBUG
        org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
        Pushed=56 entries from
        hdfs://localhost:8020/hbase/.logs/127.0.0.1
        <http://127.0.0.1>,60020,1381814216471/127.0.0.1
        <http://127.0.0.1>%3A60020.1381816329235
        2013-10-15 02:02:08,285 DEBUG
        org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
        Splitting hlog 28 of 29:
        hdfs://localhost:8020/hbase/.logs/127.0.0.1
        <http://127.0.0.1>,60020,1381814216471/127.0.0.1
        <http://127.0.0.1>%3A60020.1381816367672, length=64818440
        2013-10-15 02:02:08,285 WARN
        org.apache.hadoop.hbase.util.FSUtils: Running on HDFS
        without append enabled may result in data loss
        2013-10-15 02:02:08,554 DEBUG
        org.apache.*hadoop.hbase.master.HMaster: Not running
        balancer because processing dead regionserver(s):
        [127.0.0.1,60020*,1381814216471]
        2013-10-15 02:02:08,556 INFO
        org.apache.hadoop.hbase.catalo*g.CatalogTracker: Failed
        verification of .META.,,1 at address=127.0.0.1:60020
        <http://127.0.0.1:60020>; java.net.ConnectException:
        Connection refused*
        2013-10-15 02:02:08,559 INFO
        org.apache.hadoop.hbase.catalog.*CatalogTracker: Current
        cached META location is not valid*, resetting
        2013-10-15 02:02:08,601 WARN
        org.apache.hadoop.*hbase.master.CatalogJanitor: Failed scan
        of catalog table
        org.apache.hadoop.hbase.NotAllMetaRegionsOnlineException:
        Timed out (2147483647ms)*
                at
        
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:390)
                at
        
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:422)
                at
        org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255)
                at
        org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:237)
                at
        
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:120)
                at
        
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:88)
                at org.apache.hadoop.hbase.Chore.run(Chore.java:66)
        2013-10-15 02:02:08,842 INFO
        org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter:
        syncFs -- HDFS-200 -- not available, dfs.support.append=false
        2013-10-15 02:02:08,842 DEBUG
        org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
        Creating writer
        
path=hdfs://localhost:8020/hbase/1_webpage/853ef78be7c0853208e865a9ff13d5fb/recovered.edits/0000000000000001556.tempregion=853ef78be7c0853208e865a9ff13d5fb
        2013-10-15 02:02:09,443 DEBUG
        org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
        Pushed=39 entries from
        hdfs://localhost:8020/hbase/.logs/127.0.0.1
        <http://127.0.0.1>,60020,1381814216471/127.0.0.1
        <http://127.0.0.1>%3A60020.1381816367672
        2013-10-15 02:02:09,444 DEBUG
        org.apache.hadoop.hbase.regionserver.wal.HLogSplitter:
        Splitting hlog 29 of 29:
        hdfs://localhost:8020/hbase/.logs/127.0.0.1
        <http://127.0.0.1>,60020,1381814216471/127.0.0.1
        <http://127.0.0.1>%3A60020.1381816657239, length=0

        Thanks for your help!

Reply via email to