Which HBase / hadoop releases are you using ? See http://stackoverflow.com/questions/19205057/how-to-fix-corrupt-hadoop-hdfs
On Fri, Feb 28, 2014 at 7:14 AM, jeba earnest <[email protected]> wrote: > In a hbase cluster, all the slave nodes got restarted. When I started > Hbase services, > one of the table(test) become inconsistent. > > HDFS some blocks were missing(hbase blocks). so it was in safe mode. I > gave "safemode -leave" > Then hbase table (test)became inconsistent. > > Below actions I performed > > > 1. executed "hbase hbck" several times > 2 inconsistencies found for table "test" > > 2. hbase hbck -fixMeta -fixAssignments > HBaseFsckRepair: Region still in transition, waiting for it to become > assigned: {NAME = > 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.', > STARTKEY = '1m\x00\x03\x1B\x15', ENDKEY = '', ENCODED = > 4c213a47bba83c47075f21fec7c6d862,} > > 3. hbase hbck -repair > HBaseFsckRepair: Region still in transition, waiting for it to become > assigned: {NAME = > 'test,1m\x00\x03\x1B\x15,1393439284371.4c213a47bba83c47075f21fec7c6d862.', > STARTKEY = '1m\x00\x03\x1B\x15', ENDKEY = '', ENCODED = > 4c213a47bba83c47075f21fec7c6d862,} > > 4. I checked datanode logs parallely > Log: org.apache.hadoop.hdfs.server.datanode.DataNode: opReadBlock > BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846 > received exception java.io.EOFException > WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(192.168.1.12, > storageID=DS-831971799-192.168.1.12-50010-1391193910800, infoPort=50075, > ipcPort=50020, > storageInfo=lv=-40;cid=CID-7f99a9de-258c-493c-9db0-46b9e84b4c12;nsid=1286773982;c=0):Got > exception while serving > BP-1015188871-192.168.1.11-1391187113543:blk_7616957984716737802_27846 to / > 192.168.1.12:36127 > > > 5. Checked Namenode logs > ERROR org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:ubuntu (auth:SIMPLE) > cause:java.io.FileNotFoundException: File does not exist: > /hbase/test/4c213a47bba83c47075f21fec7c6d862/C > 2014-02-28 14:13:15,738 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 6 on 9000, call > org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from > 10.10.242.31:42149: error: java.io.FileNotFoundException: File does not > exist: /hbase/test/4c213a47bba83c47075f21fec7c6d862/C > java.io.FileNotFoundException: File does not exist: > /hbase/test/4c213a47bba83c47075f21fec7c6d862/C at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1301) > > But, I am able to browse and download the file from HDFS. How can recover > the data? > How can I make the "test" table consistent. > > > Regards, > > Jeba
