[
https://issues.apache.org/jira/browse/HADOOP-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16428.
-------------------------------------
Resolution: Won't Fix
Release Note:
closing as WONTFIX on the basis that this would be a fundamental rewrite of
DistCp, which, as it is a critical path of many workflows, is not something I'd
rush to do. You'd do it in its own module/project.
HADOOP-15281 will actually be faster than postponing upload commits until any
job commit
> Distcp don't make use of S3a Committers, be it magic or staging
> ---------------------------------------------------------------
>
> Key: HADOOP-16428
> URL: https://issues.apache.org/jira/browse/HADOOP-16428
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, tools/distcp
> Affects Versions: 3.1.1
> Reporter: Sahil Kaw
> Priority: Minor
>
> Currently, I don't see Distcp make use of S3a Committers, be it Magic or
> Staging and I have noticed most of the jobs which use MapReduce frameworks
> use S3 committers except distcp. Distcp makes use of the FileOutputCommitter
> even if S3a committer parameters are specified in the core-site.xml. Is this
> by design? If yes, can someone please explain the reason for that. Are there
> any limitations or potential risks of using S3a committers with Distcp?
> I know there is a "-direct" option that can be used with the
> FileOutputCommitter in order to avoid renaming while committing fr object
> Stores. But if anyone can put some light on the current limitation of S3a
> committers with distcp and reason for choosing FileOutputCommitters for
> Distcp over S3a committers, it would be helpful. Thanks
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
