Steve Loughran created MAPREDUCE-6823:
-----------------------------------------
Summary: FileOutputFormat to support configurable
FileOutputCommitter factory
Key: MAPREDUCE-6823
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6823
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0-alpha2
Environment: Targeting S3 as the output of work
Reporter: Steve Loughran
Assignee: Steve Loughran
In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which
can talk direct to the S3A Filesystem for more efficient operations, better
failure modes, and, most critically, as part of HADOOP-13345, atomic commit of
output. The normal committer relies on directory rename() being atomic for
this; for S3 we don't have that luxury.
To support a custom committer, we need to be able to tell FileOutputFormat (and
implicitly, all subclasses which don't have their own custom committer), to use
our new {{S3AOutputCommitter}}.
I propose:
# {{FileOutputFormat}} takes a factory to create committers.
# The factory to take a URI and {{TaskAttemptContext}} and return a committer
# the default implementation always returns a {{FileOutputCommitter}}
# A configuration option allows a new factory to be named
# An {{S3AOutputCommitterFactory}} to return a {{FileOutputCommitter}} or new
{{S3AOutputCommitter}} depending upon the URI of the destination.
Note that MRv1 already supports configurable committers; this is only the V2 API
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
