bq. Invalid HFile version: 0 The hfile, 77a0546b73d04dffa02597255aff9a08, is corrupt. Consider sidelining it and continue.
How was the loading done ? Did this problem happen only recently ? Cheers On Sat, Sep 23, 2017 at 8:25 AM, Subash Kunjupillai <[email protected]> wrote: > We have been loading data in our HBase and we observed that few regions > were > in FAILED_OPEN state. On going through few suggestions, I tried to run hbck > with no success of fixing the region. Then tried by deleting the WAL > directory from HDFS. Now I'm able to see the below error in RegionServer > log. > > 2017-09-23 16:31:40,660 INFO [RS_OPEN_REGION-dl360x2923:16020-2] > coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => > 54b0bce397eefb930bce5767513220d7, NAME => > 'event_2017-01-17,eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00,1505148266980. > 54b0bce397eefb930bce5767513220d7.', > STARTKEY => > 'eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', > ENDKEY => > 'f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00'} > failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version > 7978 > 2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0] > regionserver.HRegion: Could not initialize all stores for the > region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964. > 16ccc4ba3f32254e418e8a5fdc69178d. > 2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0] > handler.OpenRegionHandler: Failed open of > region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964. > 16ccc4ba3f32254e418e8a5fdc69178d., > starting to roll back the global memstore size. > java.io.IOException: java.io.IOException: > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading > HFile Trailer from file > maprfs:///hbase/data/default/event_2017-01-02/ > 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08 > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores( > HRegion.java:935) > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals( > HRegion.java:823) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:798) > at > org.apache.hadoop.hbase.regionserver.HRegion. > openHRegion(HRegion.java:6299) > at > org.apache.hadoop.hbase.regionserver.HRegion. > openHRegion(HRegion.java:6260) > at > org.apache.hadoop.hbase.regionserver.HRegion. > openHRegion(HRegion.java:6231) > at > org.apache.hadoop.hbase.regionserver.HRegion. > openHRegion(HRegion.java:6187) > at > org.apache.hadoop.hbase.regionserver.HRegion. > openHRegion(HRegion.java:6138) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion( > OpenRegionHandler.java:362) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process( > OpenRegionHandler.java:129) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: > org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading > HFile Trailer from file > maprfs:///hbase/data/default/event_2017-01-02/ > 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08 > at > org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java: > 556) > at > org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java: > 511) > at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:273) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion. > java:4980) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call( > HRegion.java:909) > at org.apache.hadoop.hbase.regionserver.HRegion$1.call( > HRegion.java:906) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ... 3 more > Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem > reading HFile Trailer from file > maprfs:///hbase/data/default/event_2017-01-02/ > 16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08 > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477) > at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505) > at > org.apache.hadoop.hbase.regionserver.StoreFile$Reader. > <init>(StoreFile.java:1066) > at > org.apache.hadoop.hbase.regionserver.StoreFileInfo. > open(StoreFileInfo.java:251) > at > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:394) > at > org.apache.hadoop.hbase.regionserver.StoreFile. > createReader(StoreFile.java:495) > at > org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader( > HStore.java:661) > at > org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:129) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531) > at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:528) > ... 6 more > Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 0 > (expected to be between 2 and 3) > at > org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:852) > at > org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream( > FixedFileTrailer.java:402) > at > org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462) > ... 15 more > 2017-09-23 16:31:41,134 INFO [RS_OPEN_REGION-dl360x2923:16020-0] > coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => > 16ccc4ba3f32254e418e8a5fdc69178d, NAME => > 'event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964. > 16ccc4ba3f32254e418e8a5fdc69178d.', > STARTKEY => > '\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00', > ENDKEY => > '\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\ > x00\x00\x00\x00\x00\x00'} > failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version > 7978 > > We are using HBase 1.1.8. Can someone help me on how to overcome this > issue. > > > > > -- > Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User- > f4020416.html >
