Hi Folks,
We met a very weird scenario.
We are running PE tool, during testing we found all regions are in transition
in state FAILED_OPEN.
Region server are failed to open the regions with below exception,
2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed open
of
region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb.,
starting to roll back the global memstore size. |
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386)
java.lang.IllegalStateException: Could not instantiate a region instance.
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at
org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226)
... 10 more
Caused by: java.lang.IllegalArgumentException: Need table descriptor
at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695)
at
org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672)
... 14 more
In HDFS, Table region was written as below,
drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05
/hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb
-rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05
/hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo
Here table name "TestTable" has come twice, how is that possible?
Have anyone met this scenario?
We are using HBase 1.0.2 and HDFS 2.7.2 version.
Any suggestion/help would be much appreciated.
Regards,
Pankaj
