[
https://issues.apache.org/jira/browse/HBASE-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-12830.
-----------------------------------------
Resolution: Duplicate
> Unreadable HLogs stuck in replication queues
> --------------------------------------------
>
> Key: HBASE-12830
> URL: https://issues.apache.org/jira/browse/HBASE-12830
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.9
> Reporter: Andrew Kyle Purtell
> Priority: Major
>
> We had an incident where underlying infrastructure issues caused HDFS
> namenodes to go down, not at the same time, leading to periods of HDFS
> service outage and recovery as namenodes failed over. These clusters had
> replication enabled. We had some Regionservers roll logs during partial HDFS
> availability. Namespace entries for these HLogs were created but the first
> block-being-written was lost or unable to complete, leading to dead 0-length
> HLogs in the queues of active sources. When this happens the queue becomes
> stuck on the dead 0-length HLog reporting EOFExceptions whenever the source
> wakes up and tries to (re)open the current file like so:
> {noformat}
> 2015-01-08 18:50:40,956 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> 1-<hostname removed>,60020,1418764167084 Got:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1845)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1810)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1759)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:70)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.reset(SequenceFileLogReader.java:175)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.initReader(SequenceFileLogReader.java:184)
> at
> org.apache.hadoop.hbase.regionserver.wal.ReaderBase.init(ReaderBase.java:70)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:128)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:91)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:79)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:68)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:506)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:309)
> {noformat}
> This exception originates from where SequenceFile tries to read in the 4-byte
> version header from position 0.
> In ReplicationSource#run we have an active loop:
> {code}
> // Loop until we close down
> while (isActive()) {
> ...
> }
> {code}
> Within this loop we iterate over paths in the replication queue. For each
> path, we attempt to open it:
> {code}
> // Open a reader on it
> if (!openReader(sleepMultiplier)) {
> // Reset the sleep multiplier, else it'd be reused for the next file
> sleepMultiplier = 1;
> continue;
> }
> {code}
> When we have a zero length file openReader returns TRUE but this.reader is
> set to NULL (look at the catch of the outer try block) and we fall through
> the conditional to:
> {code}
> // If we got a null reader but didn't continue, then sleep and continue
> if (this.reader == null) {
> if (sleepForRetries("Unable to open a reader", sleepMultiplier)) {
> sleepMultiplier++;
> }
> continue;
> }
> {code}
> We will keep trying to open the current file for a long time. The queue will
> be stuck until sleepMultiplier == maxRetriesMultiplier (conf
> "replication.source.maxretriesmultiplier", default 10), with a base sleep
> time of "replication.source.sleepforretries" (default 1000) ms, then we will
> call ReplicationSource#processEndOfFile().
> By default we will spin on opening the dead 0-length HLog for (1000*1) +
> (1000*2) ... + (1000*10) milliseconds before processing the file out of the
> queue. In HBASE-11964 we recommend increasing
> "replication.source.maxretriesmultiplier" to 300. Using the updated
> configuration we will wait for (1000*1) + (1000*2) ... + (1000*300)
> milliseconds before processing the file out of the queue.
> There should be some way to break out of this very long wait for a 0-length
> or corrupt file that is blocking the queue. Perhaps we can check if there are
> later files in the queue and not wait for maxRetriesMultipler in that case.
> Some different threshold could be set for that. Or, now we can't short
> circuit this manually by removing the 0-length log files, but we could enable
> this. Currently a FNFE triggers an attempt to find where the log has gone. If
> it can't be found, we fall through to
> {code}
> // TODO What happens the log is missing in both places?
> {code}
> and leave openReader() returning TRUE as above, the sleep multiplier is
> increased, and we continue to spin on the file with increasing wait times.
> (And having that TODO unimplemented is also not good.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
