[
https://issues.apache.org/jira/browse/HDFS-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694212#comment-17694212
]
ASF GitHub Bot commented on HDFS-16896:
---------------------------------------
mccormickt12 commented on code in PR #5322:
URL: https://github.com/apache/hadoop/pull/5322#discussion_r1119409826
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPread.java:
##########
@@ -603,7 +603,9 @@ public Void answer(InvocationOnMock invocation) throws
Throwable {
input.read(0, buffer, 0, 1024);
Assert.fail("Reading the block should have thrown
BlockMissingException");
} catch (BlockMissingException e) {
- assertEquals(3, input.getHedgedReadOpsLoopNumForTesting());
+ // The result of 9 is due to 2 blocks by 4 iterations plus one because
+ // hedgedReadOpsLoopNumForTesting is incremented at start of the loop.
+ assertEquals(9, input.getHedgedReadOpsLoopNumForTesting());
Review Comment:
we are actually 4x'ing, the comment was meant to help clarify.
If you recall the issue was we only previously tried each block once, the
change is to make hedged reads follow the same number of retires as non hedged
reads which has 3 retry loops.
This example has 2 blocks, and the last loop is when it exists.
Previously 2+1 and now 8+1
> HDFS Client hedged read has increased failure rate than without hedged read
> ---------------------------------------------------------------------------
>
> Key: HDFS-16896
> URL: https://issues.apache.org/jira/browse/HDFS-16896
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Reporter: Tom McCormick
> Assignee: Tom McCormick
> Priority: Major
> Labels: pull-request-available
>
> When hedged read is enabled by HDFS client, we see an increased failure rate
> on reads.
> *stacktrace*
>
> {code:java}
> Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain
> block: BP-1183972111-10.197.192.88-1590025572374:blk_17114848218_16043459722
> file=/data/tracking/streaming/AdImpressionEvent/daily/2022/07/18/compaction_1/part-r-1914862.1658217125623.1362294472.orc
> at
> org.apache.hadoop.hdfs.DFSInputStream.refetchLocations(DFSInputStream.java:1077)
> at
> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1060)
> at
> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:1039)
> at
> org.apache.hadoop.hdfs.DFSInputStream.hedgedFetchBlockByteRange(DFSInputStream.java:1365)
> at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1572)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1535)
> at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at
> org.apache.hadoop.fs.RetryingInputStream.lambda$readFully$3(RetryingInputStream.java:172)
> at org.apache.hadoop.fs.RetryPolicy.lambda$run$0(RetryPolicy.java:137)
> at org.apache.hadoop.fs.NoOpRetryPolicy.run(NoOpRetryPolicy.java:36)
> at org.apache.hadoop.fs.RetryPolicy.run(RetryPolicy.java:136)
> at
> org.apache.hadoop.fs.RetryingInputStream.readFully(RetryingInputStream.java:168)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at
> org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:112)
> at
> io.trino.plugin.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:76)
> ... 46 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
