Ok, thanks, it fixed itself. Patience is the key i guess : ) thanks a bunch.
but i see that a lot of people running into this problem with -ROOT- now and then. i wonder why that is. On Wed, Sep 22, 2010 at 3:10 PM, Dmitriy Lyubimov <[email protected]> wrote: > Thanks. > > already tried this. > > now i am getting another stack traces > > HTTP ERROR: 500 > > Trying to contact region server 192.168.2.34:60020 for region .META.,,1, row > '', but failed after 3 attempts. > > Exceptions: > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > RequestURI=/master.jsp > Caused by: > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server 192.168.2.34:60020 for region .META.,,1, row '', but failed > after 3 attempts. > > Exceptions: > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: .META.,,1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2269) > > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1848) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) > > > > > On Wed, Sep 22, 2010 at 3:08 PM, Matt Corgan <[email protected]> wrote: > >> I had this problem yesterday, which I think was caused by the -ROOT- >> region >> losing it's only row: the one that points to .META. >> >> To fix it i >> * stopped the HMaster >> * changed it's config file to point to /hbase2 instead of /hbase >> * started it so that it would generate a ROOT region in the new directory >> * stopped it >> * changed back to /hbase >> * used 'hadoop dfs cp.... ' to copy ROOT's hdfs directory from /hbase2 to >> /hbase >> * started HMaster back up >> >> apparently it doesn't care what the content of the ROOT region is on >> startup >> >> >> On Wed, Sep 22, 2010 at 5:45 PM, Dmitriy Lyubimov <[email protected]> >> wrote: >> >> > Hi, >> > >> > after restart of hbase cluster, which was running for a while (and >> perhaps >> > under some stress), my hbase UI fails with the message below. the logs >> also >> > have the same message. >> > >> > My other clients are spitting similar messages >> > >> > tried to do compactions, flush on .meta. table, which seem to succeed >> but >> > otherwise hbase is still defunct. >> > >> > hbase hdfs directory seems to be looking ok . >> > >> > how can i troubleshoot it further? >> > >> > it's 0.20.5. >> > >> > Thanks in advance. >> > -Dmitriy >> > >> > >> > HTTP ERROR: 500 >> > >> > Trying to contact region server null for region , row '', but failed >> > after 3 attempts. >> > Exceptions: >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > >> > RequestURI=/master.jsp >> > Caused by: >> > >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to >> > contact region server null for region , row '', but failed after 3 >> > attempts. >> > Exceptions: >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1037) >> > at >> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:75) >> > at >> > org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:48) >> > at >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.listTables(HConnectionManager.java:454) >> > at >> > >> org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:127) >> > at >> > >> org.apache.hadoop.hbase.generated.master.master_jsp._jspService(master_jsp.java:152) >> > at >> > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) >> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) >> > at >> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) >> > at >> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) >> > at >> > >> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) >> > at >> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) >> > at >> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) >> > at >> > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) >> > at >> > >> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) >> > at >> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) >> > at org.mortbay.jetty.Server.handle(Server.java:324) >> > at >> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) >> > at >> > >> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) >> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) >> > at >> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) >> > at >> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) >> > at >> > >> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) >> > at >> > >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> > >> > >> > >> > on the clients : >> > >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:698) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) >> > at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) >> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:134) >> > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:112) >> > ... >> > >> > >
