[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[
https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273952#comment-16273952
]
Hadoop QA commented on MAPREDUCE-5124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
39s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
53s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
branch-2.7 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client
generated 0 new + 689 unchanged - 2 fixed = 689 total (was 691) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 14 new +
616 unchanged - 8 fixed = 630 total (was 624) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 34s{color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}309m 31s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
48s{color} | {color:red} The patch generated 16 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}357m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-mapreduce-client-jobclient:13 |
| Failed junit tests | hadoop.mapreduce.v2.app.job.impl.TestJobImpl |
| | hadoop.mapred.TestMRIntermediateDataEncryption |
| | hadoop.mapred.TestJobSysDirWithDFS |
| Timed out junit tests | org.apache.hadoop.mapred.TestClusterMRNotification |
| | org.apache.hadoop.mapred.TestMiniMRClasspath |
| | org.apache.hadoop.mapred.TestMultiFileInputFormat |
| | org.apache.hadoop.mapred.TestMRCJCFileInputFormat |
| | org.apache.hadoop.mapred.TestClusterMapReduceTestCase |
| | org.apache.hadoop.mapred.TestMRTimelineEventHandling |
| | org.apache.hadoop.mapred.TestJobCounters |
| | org.apache.hadoop.mapred.TestJobName |
| | org.apache.hadoop.mapred.TestJobCleanup |
| | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| | org.apache.hadoop.mapred.TestNetworkedJob |
| |
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273839#comment-16273839
]
Hadoop QA commented on MAPREDUCE-6994:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
22s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
13s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-mapreduce-project/hadoop-mapreduce-client hadoop-mapreduce-project
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 16s{color}
| {color:red} hadoop-mapreduce-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-mapreduce-client-uploader in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 19s{color}
| {color:red} hadoop-mapreduce-project in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} |
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273775#comment-16273775 ] Robert Kanter commented on MAPREDUCE-6994: -- +1 on the 009 patch pending Jenkins > Uploader tool for Distributed Cache Deploy code changes > --- > > Key: MAPREDUCE-6994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch, > MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch, > MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch, > MAPREDUCE-6994.008.patch, MAPREDUCE-6994.009.patch > > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[ https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated MAPREDUCE-6994: -- Attachment: MAPREDUCE-6994.009.patch > Uploader tool for Distributed Cache Deploy code changes > --- > > Key: MAPREDUCE-6994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch, > MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch, > MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch, > MAPREDUCE-6994.008.patch, MAPREDUCE-6994.009.patch > > > The proposal is to create a tool that collects all available jars in the > Hadoop classpath and adds them to a single tarball file. It then uploads the > resulting archive to an HDFS directory. This saves the cluster administrator > from having to set this up manually for Distributed Cache Deploy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273741#comment-16273741
]
Miklos Szegedi commented on MAPREDUCE-6994:
---
Thank you for the review [~rkanter]. I will update the patch shortly.
I need the null check to avoid this unit test exception:
{code}
java.lang.NullPointerException
at
org.apache.hadoop.mapred.uploader.FrameworkUploader.buildPackage(FrameworkUploader.java:183)
at
org.apache.hadoop.mapred.uploader.TestFrameworkUploader.testBuildTarBall(TestFrameworkUploader.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
> Uploader tool for Distributed Cache Deploy code changes
> ---
>
> Key: MAPREDUCE-6994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch,
> MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch,
> MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch,
> MAPREDUCE-6994.008.patch
>
>
> The proposal is to create a tool that collects all available jars in the
> Hadoop classpath and adds them to a single tarball file. It then uploads the
> resulting archive to an HDFS directory. This saves the cluster administrator
> from having to set this up manually for Distributed Cache Deploy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6994) Uploader tool for Distributed Cache Deploy code changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273697#comment-16273697
]
Robert Kanter commented on MAPREDUCE-6994:
--
The 008 patch looks good. Just a few trivial things:
- {{beginUpload}} is marked as {{\@VisibleForTesting}} but it's also private
and not used in any tests. We should remove the {{\@VisibleForTesting}}
- Should we set the default replication to 10? I believe that's what's
normally used by the distributed cache.
- In {{testArguments}}, did you mean to pass {{"C"}} for the {{"-blacklist"}}
argument?
- The check if {{targetPath == null}} in {{buildPackage}} seems redundant.
{{buildPackage}} already calls {{beginUpload}}, which sets and validates it.
In fact, instead of making this a class variable, why not simply have
{{beginUpload}} return it or accept it as an argument? It's only ever used in
these two methods.
> Uploader tool for Distributed Cache Deploy code changes
> ---
>
> Key: MAPREDUCE-6994
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6994
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-6994.000.patch, MAPREDUCE-6994.001.patch,
> MAPREDUCE-6994.002.patch, MAPREDUCE-6994.003.patch, MAPREDUCE-6994.004.patch,
> MAPREDUCE-6994.005.patch, MAPREDUCE-6994.006.patch, MAPREDUCE-6994.007.patch,
> MAPREDUCE-6994.008.patch
>
>
> The proposal is to create a tool that collects all available jars in the
> Hadoop classpath and adds them to a single tarball file. It then uploads the
> resulting archive to an HDFS directory. This saves the cluster administrator
> from having to set this up manually for Distributed Cache Deploy.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5124: -- Attachment: MAPREDUCE-5124-branch-2.7.002.patch +1 for the branch-2.8 patch as well. The unit tests failures appear to be unrelated and all pass for me locally with the patch applied. Uploading the 2.7 patch which looks good to me. Assuming Jenkins run looks good as well I'll commit everything tomorrow. > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, > MAPREDUCE-5124-003.patch, MAPREDUCE-5124-CoalescingPOC-1.patch, > MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch, > MAPREDUCE-5124-branch-2.001.patch, MAPREDUCE-5124-branch-2.002.patch, > MAPREDUCE-5124-branch-2.7.001.patch, MAPREDUCE-5124-branch-2.7.002.patch, > MAPREDUCE-5124-branch-2.7.002.patch, MAPREDUCE-5124-branch-2.8.001.patch, > MAPREDUCE-5124-branch-2.8.001.patch, MAPREDUCE-5124-branch-2.8.002.patch, > MAPREDUCE-5124-branch-2.8.002.patch, MAPREDUCE-5124-branch-2.9.001.patch, > MAPREDUCE-5124-branch-2.9.002.patch, MAPREDUCE-5124-proto.2.txt, > MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[
https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273595#comment-16273595
]
Hadoop QA commented on MAPREDUCE-5124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client
generated 0 new + 365 unchanged - 2 fixed = 365 total (was 367) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
23s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}155m 50s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}194m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-mapreduce-client-jobclient:4 |
| Timed out junit tests | org.apache.hadoop.mapred.TestClusterMapReduceTestCase
|
| | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| | org.apache.hadoop.mapreduce.TestChild |
| | org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:c2d96dd |
| JIRA Issue | MAPREDUCE-5124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900069/MAPREDUCE-5124-branch-2.8.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1f4c1359b5c3 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.8 / eacccf1 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| Unreaped Processes Log |
[jira] [Updated] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5124: -- Attachment: MAPREDUCE-5124-branch-2.8.002.patch Thanks for updating the patches! +1 for the latest branch-2 patch. The checkstyle nit is unnecessary. The unit test failures appear to be unrelated, as they pass for me locally with the patch applied. Uploading the branch-2.8 patch again to get a Jenkins run on it. > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, > MAPREDUCE-5124-003.patch, MAPREDUCE-5124-CoalescingPOC-1.patch, > MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch, > MAPREDUCE-5124-branch-2.001.patch, MAPREDUCE-5124-branch-2.002.patch, > MAPREDUCE-5124-branch-2.7.001.patch, MAPREDUCE-5124-branch-2.7.002.patch, > MAPREDUCE-5124-branch-2.8.001.patch, MAPREDUCE-5124-branch-2.8.001.patch, > MAPREDUCE-5124-branch-2.8.002.patch, MAPREDUCE-5124-branch-2.8.002.patch, > MAPREDUCE-5124-branch-2.9.001.patch, MAPREDUCE-5124-branch-2.9.002.patch, > MAPREDUCE-5124-proto.2.txt, MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[
https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273074#comment-16273074
]
Hadoop QA commented on MAPREDUCE-5124:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} hadoop-mapreduce-project_hadoop-mapreduce-client
generated 0 new + 401 unchanged - 2 fixed = 401 total (was 403) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 1 new +
248 unchanged - 5 fixed = 249 total (was 253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
0s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 36s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-mapreduce-client-jobclient:2 |
| Timed out junit tests | org.apache.hadoop.mapred.TestClusterMapReduceTestCase
|
| | org.apache.hadoop.mapred.TestMRIntermediateDataEncryption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | MAPREDUCE-5124 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900029/MAPREDUCE-5124-branch-2.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 393cce7f2752 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 5bb6329 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java |
[jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272770#comment-16272770 ] Peter Bacsko commented on MAPREDUCE-5124: - Updated patches - Javadoc & checkstyle errors were hopefully eliminated. > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, > MAPREDUCE-5124-003.patch, MAPREDUCE-5124-CoalescingPOC-1.patch, > MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch, > MAPREDUCE-5124-branch-2.001.patch, MAPREDUCE-5124-branch-2.002.patch, > MAPREDUCE-5124-branch-2.7.001.patch, MAPREDUCE-5124-branch-2.7.002.patch, > MAPREDUCE-5124-branch-2.8.001.patch, MAPREDUCE-5124-branch-2.8.001.patch, > MAPREDUCE-5124-branch-2.8.002.patch, MAPREDUCE-5124-branch-2.9.001.patch, > MAPREDUCE-5124-branch-2.9.002.patch, MAPREDUCE-5124-proto.2.txt, > MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5124) AM lacks flow control for task events
[ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-5124: Attachment: MAPREDUCE-5124-branch-2.002.patch MAPREDUCE-5124-branch-2.8.002.patch MAPREDUCE-5124-branch-2.9.002.patch MAPREDUCE-5124-branch-2.7.002.patch > AM lacks flow control for task events > - > > Key: MAPREDUCE-5124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Jason Lowe >Assignee: Peter Bacsko > Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, > MAPREDUCE-5124-003.patch, MAPREDUCE-5124-CoalescingPOC-1.patch, > MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch, > MAPREDUCE-5124-branch-2.001.patch, MAPREDUCE-5124-branch-2.002.patch, > MAPREDUCE-5124-branch-2.7.001.patch, MAPREDUCE-5124-branch-2.7.002.patch, > MAPREDUCE-5124-branch-2.8.001.patch, MAPREDUCE-5124-branch-2.8.001.patch, > MAPREDUCE-5124-branch-2.8.002.patch, MAPREDUCE-5124-branch-2.9.001.patch, > MAPREDUCE-5124-branch-2.9.002.patch, MAPREDUCE-5124-proto.2.txt, > MAPREDUCE-5124-prototype.txt > > > The AM does not have any flow control to limit the incoming rate of events > from tasks. If the AM is unable to keep pace with the rate of incoming > events for a sufficient period of time then it will eventually exhaust the > heap and crash. MAPREDUCE-5043 addressed a major bottleneck for event > processing, but the AM could still get behind if it's starved for CPU and/or > handling a very large job with tens of thousands of active tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
