Thanks Ted, All regions are in the same condition. Even the table descriptor also,
drwxr-xr-x - hbase hadoop 0 2015-11-18 00:04 /hbase/data/default/TestTable/TestTable/.tabledesc -rw-r--r-- 3 hbase hadoop 289 2015-11-18 00:04 /hbase/data/default/TestTable/TestTable/.tabledesc/.tableinfo.0000000001 So restarting HM/RS doesn't help. Regards, Pankaj -----Original Message----- From: Ted Yu [mailto:[email protected]] Sent: 21 November 2015 22:24 To: [email protected] Subject: Re: Table regions are written in wrong path in HDFS Can you trace this region through master / region server log to see if there is some clue ? Cheers > On Nov 21, 2015, at 2:56 AM, Pankaj kr <[email protected]> wrote: > > Hi Folks, > > We met a very weird scenario. > We are running PE tool, during testing we found all regions are in transition > in state FAILED_OPEN. > > Region server are failed to open the regions with below exception, > 2015-11-18 02:20:38,503 | ERROR | RS_OPEN_REGION-HOSTNAME:PORT-2 | Failed > open of > region=TestTable,00000000000000000025000000,1447776261671.d5b64ad96deb3b47467db97669c009fb., > starting to roll back the global memstore size. | > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:386) > java.lang.IllegalStateException: Could not instantiate a region instance. > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6229) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6536) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6508) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6464) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6415) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.reflect.InvocationTargetException > at > sun.reflect.GeneratedConstructorAccessor17.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at > java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:6226) > ... 10 more > Caused by: java.lang.IllegalArgumentException: Need table descriptor > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:695) > at > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:672) > ... 14 more > > In HDFS, Table region was written as below, > drwxr-xr-x - hbase hadoop 0 2015-11-18 00:05 > /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb > -rw-r--r-- 3 hbase hadoop 96 2015-11-18 00:05 > /hbase/data/default/TestTable/TestTable/d5b64ad96deb3b47467db97669c009fb/.regioninfo > > Here table name "TestTable" has come twice, how is that possible? > > Have anyone met this scenario? > We are using HBase 1.0.2 and HDFS 2.7.2 version. > > Any suggestion/help would be much appreciated. > > Regards, > Pankaj
