I had this problem yesterday, which I think was caused by the -ROOT- region losing it's only row: the one that points to .META.
To fix it i * stopped the HMaster * changed it's config file to point to /hbase2 instead of /hbase * started it so that it would generate a ROOT region in the new directory * stopped it * changed back to /hbase * used 'hadoop dfs cp.... ' to copy ROOT's hdfs directory from /hbase2 to /hbase * started HMaster back up apparently it doesn't care what the content of the ROOT region is on startup On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov <[email protected]> wrote: > Hi, > > after restart of hbase cluster, which was running for a while (and perhaps > under some stress), my hbase UI fails with the message below. the logs also > have the same message. > > My other clients are spitting similar messages > > tried to do compactions, flush on .meta. table, which seem to succeed but > otherwise hbase is still defunct. > > hbase hdfs directory seems to be looking ok . > > how can i troubleshoot it further? > > it's 0.20.5. > > Thanks in advance. > -Dmitriy > > > HTTP ERROR: 500 > > Trying to contact region server null for region , row '', but failed > after 3 attempts. > Exceptions: > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > > RequestURI=/master.jsp > Caused by: > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact region server null for region , row '', but failed after 3 > attempts. > Exceptions: > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > org.apache.hadoop.hbase.TableNotFoundException: .META. > > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75) > at > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454) > at > org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) > at > org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152) > at > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:324) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) > > > > on the clients : > > org.apache.hadoop.hbase.TableNotFoundException: .META. > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112) > ... >
