[
https://issues.apache.org/jira/browse/MAPREDUCE-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725543#comment-16725543
]
Hadoop QA commented on MAPREDUCE-7173:
--------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 26s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 1 new + 153 unchanged - 0 fixed = 154 total (was 153) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 42 new + 1044 unchanged - 4 fixed = 1086 total (was 1048)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
9s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | MAPREDUCE-7173 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12952439/MAPREDUCE-7173.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux a815afd4f152 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 499c70e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7566/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7566/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7566/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7566/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1232 (vs. ulimit of 10000) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7566/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add ability to shuffle intermediate map task output to a distributed
> filesystem
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7173
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 2.9.2
> Reporter: Mikayla Konst
> Priority: Major
> Attachments: MAPREDUCE-7173.patch
>
>
> If nodes are lost during the course of a mapreduce job, the map tasks that
> ran on those nodes need to be re-run. Writing intermediate map task output to
> a distributed file system eliminates this problem in environments in which
> nodes are frequently lost, for example, in clusters that make heavy use of
> Google's Preemptible VMs or AWS's Spot Instances.
> *Example Usage:*
> *Job-scoped properties:*
> 1. Don't re-run an already-finished map task when we realize the node it ran
> on is now unusable:
> mapreduce.map.rerun-if-node-unusable=false (see MAPREDUCE-7168)
> 2. On the map side, use a new implementation of MapOutputFile that provides
> paths relative to the staging dir for the job (which is cleaned up when the
> job is done):
> mapreduce.task.general.output.class=org.apache.hadoop.mapred.HCFSOutputFiles
> 3. On the reduce side, use a new implementation of ShuffleConsumerPlugin that
> fetches map task output directly from a distributed filesystem:
> mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.HCFSShuffle
> 4. (Optional) Edit the buffer size for the output stream used when writing
> map task output
> mapreduce.map.shuffle.output.buffer.size=8192
> *Cluster-scoped properties* (see YARN-9106):
> 1. When gracefully decommissioning a node, only wait for the containers on
> that node to finish, not the applications associated with those containers
> (we don't need to wait on the applications to finish since this node is not
> serving shuffle data)
> yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-applications=false
> 2. When gracefully decommissioning a node, do not wait for app masters
> running on the node to finish so that this node can be decommissioned as soon
> as possible (failover to an app master on another node that isn't being
> decommissioned is pretty quick)
> yarn.resourcemanager.decommissioning-nodes-watcher.wait-for-app-masters=false
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
