[
https://issues.apache.org/jira/browse/HADOOP-18425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17585392#comment-17585392
]
Steve Loughran commented on HADOOP-18425:
-----------------------------------------
we've hit this in job commit at scale; the manifest committer is designed to
mitigate it by reducing load, using etags in a private rename call and self
throttling.
We also store the etag when we do a head request within the rename itself.
What we don't do yet is that head request on a normal rename. If you are seeing
this in situations other than job commit then it is something we could
consider… when the commitSingleFileByRename() operation passes in that etag
then the call could be skipped. It will of course increase load and so the
probability of throttling and this problem surfacing. But it would be the only
way to do transparent recovery of the failure in unmodified applications. And
because the manifest commiter job commit won't do those HEAD requests (it saves
the etags from the listings it does in task commit) we won't be generating any
extra load during job commit.
> [ABFS]: RenameFilePath Source File Not Found (404) error in retry loop
> ----------------------------------------------------------------------
>
> Key: HADOOP-18425
> URL: https://issues.apache.org/jira/browse/HADOOP-18425
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Sree Bhattacharyya
> Assignee: Sree Bhattacharyya
> Priority: Minor
>
> RenameFilePath on its first try receives a Request timed out error with code
> 500. On retrying the same operation, a Source file not found (404) error is
> received.
> Possible mitigation: Check whether etags remain the same before and after the
> retry and accordingly send an Operation Successful result, instead of source
> file not found.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
