[
https://issues.apache.org/jira/browse/HADOOP-13975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881831#comment-15881831
]
Hadoop QA commented on HADOOP-13975:
------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13975 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12854364/HADOOP-distcp-multithreaded-mapper-branch26.7.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11704/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Allow DistCp to use MultiThreadedMapper
> ---------------------------------------
>
> Key: HADOOP-13975
> URL: https://issues.apache.org/jira/browse/HADOOP-13975
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools/distcp
> Affects Versions: 3.0.0-alpha1
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Priority: Minor
> Attachments: HADOOP-distcp-multithreaded-mapper-branch26.1.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.2.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.3.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.4.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.5.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.6.patch,
> HADOOP-distcp-multithreaded-mapper-branch26.7.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.1.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.2.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.3.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.4.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.5.patch,
> HADOOP-distcp-multithreaded-mapper-trunk.6.patch
>
>
> Although distcp allow users to control the parallelism via number of mappers,
> sometimes it's desirable to run fewer mappers but more threads per mapper.
> Since distcp is network bound (either by throughput or more frequently by
> latency of creating connections, opening files, reading/writing files, and
> closing files), this can make each mapper much more efficient. When WebHDFS
> protocol is used either as source or target, this MultiThreaded approach can
> make the HTTP connection reuse (to the NameNode) more efficient as well.
> In that way, a lot of resources can be shared so we can save memory and
> connections to NameNode.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
