Have you run hbck on testnamespace:testtable ? bq. major compaction was running for a table.
Which table was being compacted ? Is it possible for you to share the region server log of the new server ? Cheers On Thu, Aug 20, 2015 at 4:58 AM, mukund murrali <[email protected]> wrote: > Hi > > Recently we tried adding a new regionserver to the cluster. The region > server had the following logs for all the regions it tried to open. > > 2015-08-14 01:23:52,175 ERROR [RS_OPEN_REGION-localhost:16020-0] > handler.OpenRegionHandler: Failed open of > > region=testnamespace:testtable,test_14390754745,1439165529250.7aaa895f2ec3ca602c50fcc69120c01e., > starting to roll back the global memstore size. > java.lang.IllegalStateException: Could not instantiate a region instance. > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4566) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4873) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4845) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4801) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4752) > at > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356) > at > > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4563) > ... 10 more > Caused by: java.lang.IllegalArgumentException: Need table descriptor > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:601) > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:578) > ... 15 more > > > In addition to this, all other region server began to stop serving request. > > One thing to note is during this time a major compaction was running for a > table. > > Also the hbase:meta table was in the new rs. > > The clients had the exception > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > location > > > and jstack of clients were waiting at locateRegionInMeta. > > On decommissioning the new node, the existing regionserver started serving > request. What would have caused this problem? We are using hbase-1.0.0. > > Thanks > > Regards > Mukund Murrali >
