We have been loading data in our HBase and we observed that few regions were
in FAILED_OPEN state. On going through few suggestions, I tried to run hbck
with no success of fixing the region. Then tried by deleting the WAL
directory from HDFS. Now I'm able to see the below error in RegionServer
log.
2017-09-23 16:31:40,660 INFO [RS_OPEN_REGION-dl360x2923:16020-2]
coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
54b0bce397eefb930bce5767513220d7, NAME =>
'event_2017-01-17,eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148266980.54b0bce397eefb930bce5767513220d7.',
STARTKEY =>
'eP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
ENDKEY =>
'f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'}
failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
7978
2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
regionserver.HRegion: Could not initialize all stores for the
region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.
2017-09-23 16:31:41,133 ERROR [RS_OPEN_REGION-dl360x2923:16020-0]
handler.OpenRegionHandler: Failed open of
region=event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:935)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:823)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:798)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6299)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6260)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6231)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6187)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6138)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
at
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:556)
at
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:511)
at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:273)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:4980)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:909)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:906)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
maprfs:///hbase/data/default/event_2017-01-02/16ccc4ba3f32254e418e8a5fdc69178d/d/77a0546b73d04dffa02597255aff9a08
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1066)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:251)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:394)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:495)
at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:661)
at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:129)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:528)
... 6 more
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 0
(expected to be between 2 and 3)
at
org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:852)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:402)
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462)
... 15 more
2017-09-23 16:31:41,134 INFO [RS_OPEN_REGION-dl360x2923:16020-0]
coordination.ZkOpenRegionCoordination: Opening of region {ENCODED =>
16ccc4ba3f32254e418e8a5fdc69178d, NAME =>
'event_2017-01-02,\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1505148256964.16ccc4ba3f32254e418e8a5fdc69178d.',
STARTKEY =>
'\x14P\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
ENDKEY =>
'\x15\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'}
failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version
7978
We are using HBase 1.1.8. Can someone help me on how to overcome this issue.
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
