Partial mail was sent accidentally. Sorry for that.
Resending with complete details, analysis and logs.
20-append version we are using.
To summarize there are two problems [One each from HDFS and HBase] we
noticed in this flow.
1) From HDFS
Even though client is getting the updated block info from Namenode on first
read failure, client is discarding the new info and using the old info only
to retrieve the data from datanode. So, all the read
retries are failing. [Method parameter reassignment - Not reflected in
caller]
HDFS Code snippet
org.apache.hadoop.hdfs.DFSClient.DFSInputStream.chooseDataNode
private DNAddrPair chooseDataNode(LocatedBlock block)
throws IOException {
...
...
block = getBlockAt(block.getStartOffset(), false);
...
...
}
Here method parameter "block" is assigned with the new block info which is
not reflected in the caller "blockSeekTo(long target)".
2) From HBase
Excerpt from my previous mail.
> As the recovery is an asynchronous operation recoverLease call will return
> immediately and may end up with read failure as the recovery is in
progress.
>
> This may lead to some regions to be in offline state only
> One approach is to introduce a delay in between recovery and read. But,
this
> may not be a fool proof way to address this.
I've noticed the delay is already present in HBase code. But as I mentioned
this may not be a fool proof mechanism to handle this scenario.
HBase Code snippet
In the class HLogSplitter the splitLog() calls recoverFileLease().
In recoverFileLease()
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
new InterruptedIOException().initCause(ex);
}
Once the recover call is made we sleep for one sec and proceed with
parseHLog().
Here is the log
2011-07-21 17:01:19,642 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_1311262402613_3094 from any node: java.io.IOException: No
live nodes contain current block. Will get new block locations from namenode
and retry...
2011-07-21 17:01:20,650 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_1311262402613_3094 from any node: java.io.IOException: No
live nodes contain current block. Will get new block locations from namenode
and retry...
2011-07-21 17:01:21,669 INFO org.apache.hadoop.hdfs.DFSClient: Could not
obtain block blk_1311262402613_3094 from any node: java.io.IOException: No
live nodes contain current block. Will get new block locations from namenode
and retry...
2011-07-21 17:01:22,677 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read:
java.io.IOException: Could not obtain block: blk_1311262402613_3318
file=/hbase/.logs/158-1-101-222,20020,1311260346420/158-1-101-222%3A20020.13
11265398432
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.jav
a:2491)
at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:2
256)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:2441)
at java.io.DataInputStream.read(DataInputStream.java:132)
at java.io.DataInputStream.readFully(DataInputStream.java:178)
at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:63)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:101)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1984)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1884)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1930)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(Sequence
FileLogReader.java:198)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(Sequence
FileLogReader.java:172)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.parseHLog(HLogSplitter
.java:429)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.
java:262)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.
java:188)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.ja
va:201)
-----Original Message-----
From: Stack [mailto:saint....@gmail.com]
Sent: Monday, August 01, 2011 9:03 PM
To: d...@hbase.apache.org
Cc: d...@hbase.apache.org
Subject: Re: Handling read failures during recovery
Which hdfs version and what is the error u see? Thanks.
Stack
On Aug 1, 2011, at 4:33, Laxman <lakshman...@huawei.com> wrote:
> Hi Everyone,
>
>
>
> In HBase we try to recover the HLog file and then immediately proceed with
> read operation.
>
> As the recovery is an asynchronous operation recoverLease call will return
> immediately and may end up with read failure as the recovery is in
progress.
>
> This may lead to some regions to be in offline state only.
>
>
>
> One approach is to introduce a delay in between recovery and read. But,
this
> may not be a fool proof way to address this.
>
>
>
> How do we handle this scenario?
>
>
>
> Please do correct me if my understanding went wrong.
>
> --Laxman
>
