Steve Loughran created MAPREDUCE-7474:
-----------------------------------------
Summary: [ABFS] Improve commit resilience and performance in
Manifest Committer
Key: MAPREDUCE-7474
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7474
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.3.6, 3.4.0
Reporter: Steve Loughran
* Manifest committer is not resilient to rename failures on task commit without
HADOOP-18012 rename recovery enabled.
* large burst of delete calls noted: are they needed
relates to HADOOP-19093 but takes a more minimal approach with goal of changes
in manifest committer only.
Initial proposed changes
* retry recovery on task commit rename, always (repeat save, delete, rename)
* audit delete use and see if it can be pruned
* maybe: rate limit some IO internally, but not delegate to abfs
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
