HBase master unable to recover with error "Cannot seek after EOF"

[Claude M]

Hello,

I have the following installed:

   - Hadoop 3.2.2
   - HBase 2.3.5


When all the datanodes in Hadoop are stopped but the HBase cluster is still
running, the HBase master crashes w/ the attached exception and is not
recoverable.

If I delete the contents under the following directories in hdfs, the
master will then recover:

   - /hbase/MasterData/WALs/
   - /hbase/MasterData/data/master/store/*/recovered.wals/

Is this an appropriate way to resolve the issue?  If not, what should be
done?


Thanks

java.io.EOFException: Cannot seek after EOF
    at org.apache.hadoop.hdfs.DFSInputStream.seek(DFSInputStream.java:1648)
    at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:66)
    at 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initInternal(ProtobufLogReader.java:211)
    at 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.initReader(ProtobufLogReader.java:173)
    at 
org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:64)
    at 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogReader.init(ProtobufLogReader.java:168)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:323)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:305)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:293)
    at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:429)
    at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:4863)
    at 
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:4769)
    at 
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:1013)
    at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:955)
    at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7497)
    at 
org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7455)
    at 
org.apache.hadoop.hbase.master.region.MasterRegion.open(MasterRegion.java:269)
    at 
org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:309)
    at 
org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104)
    at 
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:948)
    at 
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2240)
    at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:621)
    at java.lang.Thread.run(Thread.java:748)