Steve Loughran created HADOOP-17981:
---------------------------------------
Summary: Support etag-assisted renames in FileOutputCommitter
Key: HADOOP-17981
URL: https://issues.apache.org/jira/browse/HADOOP-17981
Project: Hadoop Common
Issue Type: New Feature
Components: fs, fs/azure
Affects Versions: 3.4.0
Reporter: Steve Loughran
Assignee: Steve Loughran
To deal with some throttling/retry issues in object stores,
pass the FileStatus entries retrieved during listing
into a private interface ResilientCommitByRename which filesystems
may implement to use extra attributes in the listing (etag, version)
to constrain and validate the operation.
Although targeting azure, GCS and others could use. no point in S3A as they
shouldn't use this committer.
# And we are not going to do any changes to FileSystem as there are explicit
guarantees of public use and stability.
I am not going to make that change as the hive thing that will suddenly start
expecting it to work forever.
# I'm not planning to merge this in, as the manifest committer is going to
include this and more (MAPREDUCE-7341)
However, I do need to get this in on a branch, so am doing this work on trunk
for dev & test and for others to review
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
