[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292735#comment-15292735
]
Hadoop QA commented on MAPREDUCE-6703:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 5 new + 642 unchanged - 18 fixed = 647 total (was 660) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s
{color} | {color:red} hadoop-mapreduce-client-core in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 55s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 9s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 6s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.tools.TestCLI |
| | hadoop.mapred.TestMiniMRChildTask |
| | hadoop.mapred.TestLocalMRNotification |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805115/MAPREDUCE-6703.002.patch
|
| JIRA Issue | MAPREDUCE-6703 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux bf52daeb79d4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 21d2b90 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Issue Comment Deleted] (MAPREDUCE-6702) TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-6702:
-
Comment: was deleted
(was: I found an assertion failure in a task log.
{noformat}
2016-05-20 12:49:52,551 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1463716126782_0003_m_00_0: Err
or: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.TestMiniMRChildTask$ExecutionEnvCheckMapClass.configure(TestMiniMRChildTask.java:134)
... 22 more
{noformat}
The log tells that {{executionEnvPathVariable}} is null in the below source
code.
{code}
public void configure(JobConf job) {
String executionEnvPathVariable = System.getenv(Shell.WINDOWS ? "PATH"
: "LD_LIBRARY_PATH");
String hadoopHome = System.getenv("HADOOP_COMMON_HOME");
if (hadoopHome == null) {
hadoopHome = "";
}
String hadoopLibLocation = hadoopHome
+ (Shell.WINDOWS ? "\\bin" : "/lib/native");
assertTrue(executionEnvPathVariable.contains(hadoopLibLocation)); //
NPE occurred!
}
{code})
> TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are
> failing
> --
>
> Key: MAPREDUCE-6702
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6702
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>
> ---
> T E S T S
> ---
> Running org.apache.hadoop.mapred.TestMiniMRChildTask
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 92.48 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.TestMiniMRChildTask
> testTaskEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 21.906 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv(TestMiniMRChildTask.java:472)
> testTaskOldEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 17.452 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
[jira] [Commented] (MAPREDUCE-6702) TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292697#comment-15292697
]
Akira AJISAKA commented on MAPREDUCE-6702:
--
I found an assertion failure in a task log.
{noformat}
2016-05-20 12:49:52,551 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1463716126782_0003_m_00_0: Err
or: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1755)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.TestMiniMRChildTask$ExecutionEnvCheckMapClass.configure(TestMiniMRChildTask.java:134)
... 22 more
{noformat}
The log tells that {{executionEnvPathVariable}} is null in the below source
code.
{code}
public void configure(JobConf job) {
String executionEnvPathVariable = System.getenv(Shell.WINDOWS ? "PATH"
: "LD_LIBRARY_PATH");
String hadoopHome = System.getenv("HADOOP_COMMON_HOME");
if (hadoopHome == null) {
hadoopHome = "";
}
String hadoopLibLocation = hadoopHome
+ (Shell.WINDOWS ? "\\bin" : "/lib/native");
assertTrue(executionEnvPathVariable.contains(hadoopLibLocation)); //
NPE occurred!
}
{code}
> TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are
> failing
> --
>
> Key: MAPREDUCE-6702
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6702
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Affects Versions: 3.0.0-alpha1
>Reporter: Daniel Templeton
>
> ---
> T E S T S
> ---
> Running org.apache.hadoop.mapred.TestMiniMRChildTask
> Tests run: 3, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 92.48 sec <<<
> FAILURE! - in org.apache.hadoop.mapred.TestMiniMRChildTask
> testTaskEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 21.906 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.runTestTaskEnv(TestMiniMRChildTask.java:550)
> at
> org.apache.hadoop.mapred.TestMiniMRChildTask.testTaskEnv(TestMiniMRChildTask.java:472)
> testTaskOldEnv(org.apache.hadoop.mapred.TestMiniMRChildTask) Time elapsed:
> 17.452 sec <<< FAILURE!
> java.lang.AssertionError: The environment checker job failed.
> at org.junit.Assert.fail(Assert.java:88)
>
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292646#comment-15292646 ] zhihai xu commented on MAPREDUCE-6696: -- [~jianhe] thanks for reviewing and committing the patch! > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated MAPREDUCE-6703: --- Attachment: MAPREDUCE-6703.002.patch Attaching updated patch addressing [~jianhe]'s comments.. > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch, MAPREDUCE-6703.002.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292581#comment-15292581 ] Hudson commented on MAPREDUCE-6696: --- SUCCESS: Integrated in Hadoop-trunk-Commit #9829 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9829/]) MAPREDUCE-6696. Add a configuration to limit the number of map tasks (jianhe: rev 21d2b90213e8e188bcac31f9360c5176ac89d083) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestLocalJobSubmission.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-6696: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Target Version/s: 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-2, thanks Zhihai ! > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292555#comment-15292555
]
Hadoop QA commented on MAPREDUCE-6703:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 6s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 20s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 43s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
| Timed out junit tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch
|
| JIRA Issue | MAPREDUCE-6703 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 71f566c497aa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 42c22f7 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292491#comment-15292491
]
Jian He commented on MAPREDUCE-6703:
I see, just few minor comments:
- could you add comments to the newly added config about what this config means
?
- Here, we can just call addOpportunisticResourceRequest and so
addContainerReq method does not need to be refactored.
{code}
maps.put(event.getAttemptID(), request);
addContainerReq(request, ExecutionType.OPPORTUNISTIC);
{code}
> Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
> --
>
> Key: MAPREDUCE-6703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: MAPREDUCE-6703.001.patch
>
>
> YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes
> and specifically OPPORTUNISTIC containers.
> The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to
> provide hints via config to the MR framework as to the number of containers
> it would like to schedule as OPPORTUNISTIC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292402#comment-15292402 ] Konstantinos Karanasos commented on MAPREDUCE-6703: --- Hi [~jianhe], This particular change in the MR AM is for people to start trying out OPPORTUNISTIC containers and distributed scheduling. However, you can find some of our initial experimental results on some internal workloads (consisting of both short and long tasks) in our paper here: http://research.microsoft.com/pubs/244469/mercury-atc2015.pdf. > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292343#comment-15292343 ] Jian He commented on MAPREDUCE-6703: I see. depending on how locality sensitive the MR job is, this may not benefit as much. Wonder whether you have statistics to show how much this improves, or this is mainly for example purpose ? > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292327#comment-15292327 ] Arun Suresh commented on MAPREDUCE-6703: [~jianhe], thanks for taking a look.. So in the first version, we decided to ignore locality (and for that matter, node labels too), since the point of YARN-2877 is to improve allocation latencies at the cost of relaxing some scheduling constraints. The sweet-spot would be for Applications that have large number of really short lived tasks that do not care about locality. Reduce tasks are always guaranteed (since the cost of preempting a reduce task is quite high) and failed and retried map tasks are also always guaranteed.. > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292313#comment-15292313 ] Jian He commented on MAPREDUCE-6703: looks like the locality is ignored for opportunistic containers, does YARN-2877 consider locality for opportunistic containers ? > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292106#comment-15292106 ] Jian He commented on MAPREDUCE-6696: lgtm, +1 > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[
https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292038#comment-15292038
]
Ming Ma commented on MAPREDUCE-5044:
bq. In that case, do we want to call it something like signalsToContainers?
Sounds good. signalsToContainers can take an array of
{{SignalContainerRequest}}, each of which has a list of commands belonging to
the same container. When we decide to add signalsToContainers later, deprecate
signalToContainer and NM will still support signalToContainer until major
upgrade. In that way, we don't need to fix {{required}} issue given only new
signalsToContainers method will use list-based {{SignalContainerRequest}}.
> Have AM trigger jstack on task attempts that timeout before killing them
>
>
> Key: MAPREDUCE-5044
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5044.008.patch, MAPREDUCE-5044.009.patch,
> MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch,
> MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch, MAPREDUCE-5044.v06.patch,
> MAPREDUCE-5044.v07.local.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png,
> Screen Shot 2013-11-12 at 1.06.04 PM.png
>
>
> When an AM expires a task attempt it would be nice if it triggered a jstack
> output via SIGQUIT before killing the task attempt. This would be invaluable
> for helping users debug their hung tasks, especially if they do not have
> shell access to the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[
https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291977#comment-15291977
]
Eric Payne commented on MAPREDUCE-5044:
---
[~mingma], thank you for your reply and explanation.
{quote}
- signalContainers was initially suggested as an ordered list of
signalContainer. So it could include requests from the same container or
requests from different containers. It is true that the only use case we know
of so far is to include requests from the same container.
{quote}
In that case, do we want to call it something like {{signalsToContainers}}? I'm
open for ideas.
{quote}
- Will the required in the protocol buffer definition create any issue if we do
rolling upgrade from 2.8 to 2.9 and the 2.9 MR AM might send a list of
SignalContainerCommandProto to 2.8 NM? Maybe 2.8 NM just discards the message,
not a big deal. Regardless, that is a separate issue that we don't need to
address it here.
{quote}
Yes, this is a concern and something we need to look into more deeply and keep
in mind.
> Have AM trigger jstack on task attempts that timeout before killing them
>
>
> Key: MAPREDUCE-5044
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5044.008.patch, MAPREDUCE-5044.009.patch,
> MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch,
> MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch, MAPREDUCE-5044.v06.patch,
> MAPREDUCE-5044.v07.local.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png,
> Screen Shot 2013-11-12 at 1.06.04 PM.png
>
>
> When an AM expires a task attempt it would be nice if it triggered a jstack
> output via SIGQUIT before killing the task attempt. This would be invaluable
> for helping users debug their hung tasks, especially if they do not have
> shell access to the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291961#comment-15291961 ] zhihai xu commented on MAPREDUCE-6696: -- Also it looks like the check style issue is not a real issue. because all configuration definitions in MRJobConfig use "public static final", this patch follows the same rule. I don't why check style script reported this issue. > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291887#comment-15291887
]
Akira AJISAKA edited comment on MAPREDUCE-6607 at 5/19/16 6:30 PM:
---
Hi [~lewuathe], would you provide a patch for branch-2? The patch cannot be
applied to branch-2.
was (Author: ajisakaa):
Hi [~lewuathe], would you provide a patch for branch-2? The patch does not
apply to branch-2.
> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch,
> MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch, MAPREDUCE-6607.05.patch,
> MAPREDUCE-6607.06.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned
> up:
> * mapreduce.task.files.preserve.failedtask
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter
> was supposed to be used only if that task name matches against the specified
> regular expression.
> {code}
> protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
> }
> {code}
> {code}
> public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
> if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
> LOG.warn("Job Staging directory is null");
> return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " +
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
> }
> } catch(IOException io) {
> LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291887#comment-15291887
]
Akira AJISAKA edited comment on MAPREDUCE-6607 at 5/19/16 6:30 PM:
---
Hi [~lewuathe], would you provide a patch for branch-2? The patch does not
apply to branch-2.
was (Author: ajisakaa):
Hi [~lewuathe], would you provide a patch for branch-2?
> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch,
> MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch, MAPREDUCE-6607.05.patch,
> MAPREDUCE-6607.06.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned
> up:
> * mapreduce.task.files.preserve.failedtask
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter
> was supposed to be used only if that task name matches against the specified
> regular expression.
> {code}
> protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
> }
> {code}
> {code}
> public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
> if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
> LOG.warn("Job Staging directory is null");
> return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " +
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
> }
> } catch(IOException io) {
> LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291887#comment-15291887
]
Akira AJISAKA commented on MAPREDUCE-6607:
--
Hi [~lewuathe], would you provide a patch for branch-2?
> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch,
> MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch, MAPREDUCE-6607.05.patch,
> MAPREDUCE-6607.06.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned
> up:
> * mapreduce.task.files.preserve.failedtask
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter
> was supposed to be used only if that task name matches against the specified
> regular expression.
> {code}
> protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
> }
> {code}
> {code}
> public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
> if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
> LOG.warn("Job Staging directory is null");
> return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " +
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
> }
> } catch(IOException io) {
> LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291885#comment-15291885
]
Hadoop QA commented on MAPREDUCE-6696:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 539 unchanged - 1 fixed = 541 total (was 540) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 10s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804964/MAPREDUCE-6696.004.patch
|
| JIRA Issue | MAPREDUCE-6696 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux d5321a7ae7f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 141873c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6514/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| unit |
[jira] [Commented] (MAPREDUCE-6607) .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or mapreduce.task.files.preserve.filepattern are set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291702#comment-15291702
]
Akira AJISAKA commented on MAPREDUCE-6607:
--
+1, sorry for delay. Committing this.
> .staging dir is not cleaned up if mapreduce.task.files.preserve.failedtask or
> mapreduce.task.files.preserve.filepattern are set
> ---
>
> Key: MAPREDUCE-6607
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6607
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Maysam Yabandeh
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: MAPREDUCE-6607.01.patch, MAPREDUCE-6607.02.patch,
> MAPREDUCE-6607.03.patch, MAPREDUCE-6607.04.patch, MAPREDUCE-6607.05.patch,
> MAPREDUCE-6607.06.patch
>
>
> if either of the following configs are set, then .staging dir is not cleaned
> up:
> * mapreduce.task.files.preserve.failedtask
> * mapreduce.task.files.preserve.filepattern
> The former was supposed to keep only .staging of failed tasks and the latter
> was supposed to be used only if that task name matches against the specified
> regular expression.
> {code}
> protected boolean keepJobFiles(JobConf conf) {
> return (conf.getKeepTaskFilesPattern() != null || conf
> .getKeepFailedTaskFiles());
> }
> {code}
> {code}
> public void cleanupStagingDir() throws IOException {
> /* make sure we clean the staging files */
> String jobTempDir = null;
> FileSystem fs = getFileSystem(getConfig());
> try {
> if (!keepJobFiles(new JobConf(getConfig( {
> jobTempDir = getConfig().get(MRJobConfig.MAPREDUCE_JOB_DIR);
> if (jobTempDir == null) {
> LOG.warn("Job Staging directory is null");
> return;
> }
> Path jobTempDirPath = new Path(jobTempDir);
> LOG.info("Deleting staging directory " +
> FileSystem.getDefaultUri(getConfig()) +
> " " + jobTempDir);
> fs.delete(jobTempDirPath, true);
> }
> } catch(IOException io) {
> LOG.error("Failed to cleanup staging dir " + jobTempDir, io);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[
https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291463#comment-15291463
]
Hadoop QA commented on MAPREDUCE-6380:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: patch
generated 25 new + 38 unchanged - 0 fixed = 63 total (was 38) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 7s
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804935/MAPREDUCE-6380.01.patch
|
| JIRA Issue | MAPREDUCE-6380 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 635def41c9ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 141873c |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6515/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6515/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6515/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> AggregatedLogDeletionService will throw exception when there are some other
> directories in remote-log-dir
> -
>
> Key: MAPREDUCE-6380
> URL:
[jira] [Updated] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6696: - Attachment: MAPREDUCE-6696.004.patch > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch, MAPREDUCE-6696.004.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15291366#comment-15291366 ] zhihai xu commented on MAPREDUCE-6696: -- The test failures are not related to my change. It is already reported at https://issues.apache.org/jira/browse/MAPREDUCE-6702 > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated MAPREDUCE-6380: -- Status: Patch Available (was: Open) > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Assignee: Kai Sasaki >Priority: Trivial > Attachments: MAPREDUCE-6380.01.patch > > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated MAPREDUCE-6380: -- Attachment: MAPREDUCE-6380.01.patch > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Assignee: Kai Sasaki >Priority: Trivial > Attachments: MAPREDUCE-6380.01.patch > > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6380) AggregatedLogDeletionService will throw exception when there are some other directories in remote-log-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki reassigned MAPREDUCE-6380: - Assignee: Kai Sasaki > AggregatedLogDeletionService will throw exception when there are some other > directories in remote-log-dir > - > > Key: MAPREDUCE-6380 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6380 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Reporter: Zhang Wei >Assignee: Kai Sasaki >Priority: Trivial > > AggregatedLogDeletionService will throw FileNotFoundException when there are > some extraneous directories put in remote-log-dir. The deletion function will > try to listStatus against the "extraneous-dir + suffix" dir. I think it > would be better if the function can ignore these directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6700) Jobhistory server attempt and task table not loading maps/reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved MAPREDUCE-6700.
--
Resolution: Implemented
This was commit miss in branch-2.8 for the issue YARN-3840. It is fixed by re
opening the YARN-3840.
Closing as Implemented..
Thanks [~bibinchundatt] for finding this issue in regression. I really
appreciate it:-) It would have been downstream defect.
Thanks [~varun_saxena] for finding root cause and providing rebase patch in
YARN-3840 soon.
> Jobhistory server attempt and task table not loading maps/reduce
>
>
> Key: MAPREDUCE-6700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
>
> Browser
> ===
> Chrome
> Steps to reproduce
> ==
> # Submit mapreduce application with 20 maps
> # Wait till completion of mapreduce application
> # Check maps attempts page
> {{jobhistory/attempts/job_1463446678437_0003/m/SUCCESSFUL}}
> and {{jobhistory/tasks/job_1463446678437_0003/m}} page
> Actual
> =
> Table not loading.
> Sort based on any column other than attempt contents are loaded.
> Column 0 is of *natural sorting* and not working. So waiting for ever to be
> sorted.
> {noformat}
> SCRIPT438: Object doesn't support property or method 'natural-asc'
> jquery.dataTables.min.js, line 86 character 179
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290696#comment-15290696
]
Hadoop QA commented on MAPREDUCE-6703:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 11 new + 640 unchanged - 18 fixed = 651 total (was 658) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 0s {color} |
{color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 24s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 46s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.tools.TestCLI |
| | hadoop.mapred.TestMiniMRChildTask |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804646/MAPREDUCE-6703.001.patch
|
| JIRA Issue | MAPREDUCE-6703 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cf46f41e5cf7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1597630 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290683#comment-15290683
]
Hadoop QA commented on MAPREDUCE-6696:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 29s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 539 unchanged - 1 fixed = 541 total (was 540) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 31s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestMiniMRChildTask |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12804872/MAPREDUCE-6696.003.patch
|
| JIRA Issue | MAPREDUCE-6696 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 11759b34e3cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1597630 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6513/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| unit |
[jira] [Commented] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290576#comment-15290576 ] Bibin A Chundatt commented on MAPREDUCE-: - [~brahmareddy] Could you check all threads are starting are same time?IIUC the barrier case also we need to handle. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6666) Support MultiThreads in a Map and Distribution of files in NNBench
[ https://issues.apache.org/jira/browse/MAPREDUCE-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula updated MAPREDUCE-: Attachment: MAPREDUCE--02.patch uploading the latest patch. > Support MultiThreads in a Map and Distribution of files in NNBench > -- > > Key: MAPREDUCE- > URL: https://issues.apache.org/jira/browse/MAPREDUCE- > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula > Attachments: MAPREDUCE--01.patch, MAPREDUCE--02.patch > > > Support Distribution of files to multiple directories generated by NNBench. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
