[
https://issues.apache.org/jira/browse/HADOOP-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440564#comment-17440564
]
Steve Loughran commented on HADOOP-17691:
-----------------------------------------
happens with oauth as it does permissions checks.
rename doesn't have this problem, so moving under trash is a possible recovery
mechanism.
the MAPREDUCE-7341 manifest committer will workaround this problem by
* deleting individual task attempt dirs before the whole job attempt
* supporting rename to trash as sole/fallback strategy
> Abfs directory delete times out on large directory tree w/ Oauth:
> OperationTimedOut
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-17691
> URL: https://issues.apache.org/jira/browse/HADOOP-17691
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Major
>
> Timeouts surfacing on abfs when a delete of a large directory tree is invoked.
> {code}
> StatusDescription=Operation could not be completed within the specified time.
> ErrorCode=OperationTimedOut
> ErrorMessage=Operation could not be completed within the specified time.
> {code}
> This has surfaced in v1 FileOutputCommitter cleanups, implying the
> directories created there (many many dirs, no files remaining after the job
> commit) is sufficient to create the problem.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
