[
https://issues.apache.org/jira/browse/HDFS-14384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796972#comment-16796972
]
Surendra Singh Lilhore commented on HDFS-14384:
-----------------------------------------------
{{DFSInputStream.getBlockAt(long offset)}} give always last block for below
condition.
{code:java}
else if (offset >= locatedBlocks.getFileLength()) {
// offset to the portion of the last block,
// which is not known to the name-node yet;
// getting the last block
blk = locatedBlocks.getLastLocatedBlock();
}{code}
When the token get expire for lastLocatedblock, it will not be updated in
{{DFSInputStream.fetchBlockAt(offset,length,useCache)}}. For last block always
token get expired and when all the DN marked as dead LocatedBlocks will be
refetched with some sleep interval. This sleep interval increase file read time
and this should be avoided
@DFSInputStream#refetchLocations()
{code:java}
// Introducing a random factor to the wait time before another retry.
// The wait time is dependent on # of failures and a random factor.
// At the first time of getting a BlockMissingException, the wait time
// is a random number between 0..3000 ms. If the first retry
// still fails, we will wait 3000 ms grace period before the 2nd retry.
// Also at the second retry, the waiting window is expanded to 6000 ms
// alleviating the request rate from the server. Similarly the 3rd retry
// will wait 6000ms grace period before retry and the waiting window is
// expanded to 9000ms.
final int timeWindow = dfsClient.getConf().getTimeWindow();
// grace period for the last round of attempt
double waitTime = timeWindow * failures +
// expanding time window for each failure
timeWindow * (failures + 1) *
ThreadLocalRandom.current().nextDouble();
DFSClient.LOG.warn("DFS chooseDataNode: got # " + (failures + 1) +
" IOException, will wait for " + waitTime + " msec.");
Thread.sleep((long)waitTime);
{code}
> When lastLocatedBlock token expire, it will take 1~3s second to refetch it.
> ---------------------------------------------------------------------------
>
> Key: HDFS-14384
> URL: https://issues.apache.org/jira/browse/HDFS-14384
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.7.2
> Reporter: Surendra Singh Lilhore
> Assignee: Surendra Singh Lilhore
> Priority: Major
>
> Scenario :
> 1. Write file with one block which is in-progress.
> 2. Open input stream and close the output stream.
> 3. Wait for block token expiration and read the data.
> 4. Last block read take 1~3 sec to read it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
