[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148838#comment-14148838
]
Zhijie Shen commented on MAPREDUCE-5831:
+1, will commit it
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5831:
---
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2. Thanks, Junping!
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14148847#comment-14148847
]
Hudson commented on MAPREDUCE-5831:
---
FAILURE: Integrated in Hadoop-trunk-Commit #6118 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6118/])
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
Contributed by Junping Du. (zjshen: rev
662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6109) Fix minor typo in distcp -p usage text
[
https://issues.apache.org/jira/browse/MAPREDUCE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149044#comment-14149044
]
Hudson commented on MAPREDUCE-6109:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/692/])
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw)
(aw: rev f614cb71d47a34a5e7ccb1057f4f61d3cd179f55)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
Fix minor typo in distcp -p usage text
--
Key: MAPREDUCE-6109
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6109
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 2.6.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6109.001.patch
In the distcp usage for -p there needs to be a space added after -p:
{quote}
the -p flag.Refer to the DistCp documentation for
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6093) minor distcp doc edits
[ https://issues.apache.org/jira/browse/MAPREDUCE-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149050#comment-14149050 ] Hudson commented on MAPREDUCE-6093: --- FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/692/]) MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) (aw: rev 6c543084238992351a4ab677eae23695763dc1d4) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm * hadoop-mapreduce-project/CHANGES.txt minor distcp doc edits -- Key: MAPREDUCE-6093 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6093 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp, documentation Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Fix For: 2.6.0 Attachments: MAPREDUCE-6093.001.patch Minor edits to DistCp.md.vm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149056#comment-14149056 ] Hudson commented on MAPREDUCE-6078: --- FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/692/]) MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang) (aw: rev 6c3cebda3f3662af9db46c1bbfe22543ac18ed57) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include/gtest/gtest.h * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest-all.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest_main.cc native-task: fix gtest build on macosx -- Key: MAPREDUCE-6078 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-6078.v1.patch, MAPREDUCE-6078.v2.patch Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 separate gtest compile from nttest in order to surpress compile warnings, but it forget to add addition compile flags added to nttest is also required for gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149055#comment-14149055
]
Hudson commented on MAPREDUCE-5831:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #692 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/692/])
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
Contributed by Junping Du. (zjshen: rev
662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149196#comment-14149196
]
Hudson commented on MAPREDUCE-5831:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/])
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
Contributed by Junping Du. (zjshen: rev
662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6109) Fix minor typo in distcp -p usage text
[
https://issues.apache.org/jira/browse/MAPREDUCE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149184#comment-14149184
]
Hudson commented on MAPREDUCE-6109:
---
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/])
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw)
(aw: rev f614cb71d47a34a5e7ccb1057f4f61d3cd179f55)
*
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
* hadoop-mapreduce-project/CHANGES.txt
Fix minor typo in distcp -p usage text
--
Key: MAPREDUCE-6109
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6109
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 2.6.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6109.001.patch
In the distcp usage for -p there needs to be a space added after -p:
{quote}
the -p flag.Refer to the DistCp documentation for
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6093) minor distcp doc edits
[ https://issues.apache.org/jira/browse/MAPREDUCE-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149191#comment-14149191 ] Hudson commented on MAPREDUCE-6093: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/]) MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) (aw: rev 6c543084238992351a4ab677eae23695763dc1d4) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm * hadoop-mapreduce-project/CHANGES.txt minor distcp doc edits -- Key: MAPREDUCE-6093 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6093 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp, documentation Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Fix For: 2.6.0 Attachments: MAPREDUCE-6093.001.patch Minor edits to DistCp.md.vm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149197#comment-14149197 ] Hudson commented on MAPREDUCE-6078: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1883 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1883/]) MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang) (aw: rev 6c3cebda3f3662af9db46c1bbfe22543ac18ed57) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest-all.cc * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include/gtest/gtest.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest_main.cc native-task: fix gtest build on macosx -- Key: MAPREDUCE-6078 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-6078.v1.patch, MAPREDUCE-6078.v2.patch Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 separate gtest compile from nttest in order to surpress compile warnings, but it forget to add addition compile flags added to nttest is also required for gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6093) minor distcp doc edits
[ https://issues.apache.org/jira/browse/MAPREDUCE-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149270#comment-14149270 ] Hudson commented on MAPREDUCE-6093: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/]) MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) (aw: rev 6c543084238992351a4ab677eae23695763dc1d4) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/DistCp.md.vm minor distcp doc edits -- Key: MAPREDUCE-6093 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6093 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp, documentation Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Trivial Fix For: 2.6.0 Attachments: MAPREDUCE-6093.001.patch Minor edits to DistCp.md.vm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6109) Fix minor typo in distcp -p usage text
[
https://issues.apache.org/jira/browse/MAPREDUCE-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149264#comment-14149264
]
Hudson commented on MAPREDUCE-6109:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/])
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw)
(aw: rev f614cb71d47a34a5e7ccb1057f4f61d3cd179f55)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
Fix minor typo in distcp -p usage text
--
Key: MAPREDUCE-6109
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6109
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 2.6.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6109.001.patch
In the distcp usage for -p there needs to be a space added after -p:
{quote}
the -p flag.Refer to the DistCp documentation for
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6078) native-task: fix gtest build on macosx
[ https://issues.apache.org/jira/browse/MAPREDUCE-6078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149276#comment-14149276 ] Hudson commented on MAPREDUCE-6078: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/]) MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang) (aw: rev 6c3cebda3f3662af9db46c1bbfe22543ac18ed57) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include/gtest/gtest.h * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/CMakeLists.txt * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest-all.cc * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/gtest_main.cc native-task: fix gtest build on macosx -- Key: MAPREDUCE-6078 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6078 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Trivial Fix For: 3.0.0 Attachments: MAPREDUCE-6078.v1.patch, MAPREDUCE-6078.v2.patch Try compile the HEAD code in macos but failed, looks like MAPREDUCE-5977 separate gtest compile from nttest in order to surpress compile warnings, but it forget to add addition compile flags added to nttest is also required for gtest build, this patch fix this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5831) Old MR client is not compatible with new MR application
[
https://issues.apache.org/jira/browse/MAPREDUCE-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149275#comment-14149275
]
Hudson commented on MAPREDUCE-5831:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1908 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1908/])
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
Contributed by Junping Du. (zjshen: rev
662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FileSystemCounterGroup.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestCounters.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/counters/FrameworkCounterGroup.java
Old MR client is not compatible with new MR application
---
Key: MAPREDUCE-5831
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5831
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mr-am
Affects Versions: 2.2.0, 2.3.0
Reporter: Zhijie Shen
Assignee: Junping Du
Priority: Blocker
Fix For: 2.6.0
Attachments: MAPREDUCE-5831-v2.patch, MAPREDUCE-5831-v3.patch,
MAPREDUCE-5831.patch
Recently, we saw the following scenario:
1. The user setup a cluster of hadoop 2.3., which contains YARN 2.3 and MR
2.3.
2. The user client on a machine that MR 2.2 is installed and in the classpath.
Then, when the user submitted a simple wordcount job, he saw the following
message:
{code}
16:00:41,027 INFO main mapreduce.Job:1345 - map 100% reduce 100%
16:00:41,036 INFO main mapreduce.Job:1356 - Job job_1396468045458_0006
completed successfully
16:02:20,535 WARN main mapreduce.JobRunner:212 - Cannot start job
[wordcountJob]
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.mapreduce.JobCounter.MB_MILLIS_REDUCES
at java.lang.Enum.valueOf(Enum.java:236)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.valueOf(FrameworkCounterGroup.java:148)
at
org.apache.hadoop.mapreduce.counters.FrameworkCounterGroup.findCounter(FrameworkCounterGroup.java:182)
at
org.apache.hadoop.mapreduce.counters.AbstractCounters.findCounter(AbstractCounters.java:154)
at
org.apache.hadoop.mapreduce.TypeConverter.fromYarn(TypeConverter.java:240)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:370)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1289)
. . .
{code}
The problem is that the wordcount job was running on one or more than one
nodes of the YARN cluster, where MR 2.3 libs were installed, and
JobCounter.MB_MILLIS_REDUCES is available in the counters. On the other side,
due to the classpath setting, the client was likely to run with MR 2.2 libs.
After the client retrieved the counters from MR AM, it tried to construct the
Counter object with the received counter name. Unfortunately, the enum didn't
exist in the client's classpath. Therefore, No enum constant exception is
thrown here.
JobCounter.MB_MILLIS_REDUCES is brought to MR2 via MAPREDUCE-5464 since
Hadoop 2.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6094) TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149716#comment-14149716
]
Chris Douglas commented on MAPREDUCE-6094:
--
+1
TestMRCJCFileInputFormat.testAddInputPath() fails on trunk
--
Key: MAPREDUCE-6094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6094
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Sangjin Lee
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-6094.patch
{noformat}
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.624 sec
FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat
testAddInputPath(org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat)
Time elapsed: 0.886 sec ERROR!
java.io.IOException: No FileSystem for scheme: s3
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2583)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2590)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2629)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2611)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)
at
org.apache.hadoop.mapreduce.lib.input.TestMRCJCFileInputFormat.testAddInputPath(TestMRCJCFileInputFormat.java:55)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149720#comment-14149720
]
Zhijie Shen commented on MAPREDUCE-5933:
bq. 1. TimelineClient should do nothing in the case where the ATS is not
enabled. Or did you mean that users may want to disable posting to the ATS for
a specific job, even when the ATS is enabled?
We can either do per-frame work or per-job config. Depending on YARN's
system-wide config makes all kinds of applications enable or disable the client
simultaneously. There's another ticket: MAPREDUCE-6018. We can handle it
separately.
bq. 2. I think we should revisit this after YARN-2517 is done. No reason to
duplicate work here.
Make sense.
bq. 3. The current behavior is to throw a YarnRuntimeException when there's a
problem processing the events for the JHS,
That's true. Let's keep to it.
bq. 4. I think we want to publish each event as it comes in. This should make
the Job available in the ATS sooner than it is in the JHS.
Good point. If the job has a lot of events, we may not need to wait until job
finish to flush all of them, but we can periodically flush the outstanding
events. Again, this may be handled by YARN-2517. I did some investigation: it
seems that we only record the final counters, but not the realtime ones, which
should be much traffic. I'm not sure if it exists other high frequent events.
Any idea?
bq. 5. The countersToJSON method already turns the Counters into JSON so the
ATS can display them.
Yeah, my question is whether we really want to make counters' details
visualization-friendly via timeline read APIs, while the JHS will query this
data, reconstruct counter objects and show them there. If we don't, the easiest
way is to write counters into bytes when storing, and recover them from bytes
when retrieving in.
bq. 6. As before, I was trying to mirror what has been done for the JHS. I'm
publishing to the ATS the same events and information that the JHS gets.
It seems that you published all the events related to job summary, which may
not be enough. Before processing the summary, all events except
NormalizedResourceEvent is going to be written into a job history file.
{code}
HistoryEvent historyEvent = event.getHistoryEvent();
if (! (historyEvent instanceof NormalizedResourceEvent)) {
mi.writeEvent(historyEvent);
}
processEventForJobSummary(event.getHistoryEvent(), mi.getJobSummary(),
{code}
You can take a look at HistoryFileManager and JobHistoryParser. It seems that
while the events in a job history file are used to recover a job's history
info, while a summary file is going to be used to log a job summary at JHS.
JobHistoryParser may tell you all the events that JHS wants to use.
Enable MR AM to post history events to the timeline server
--
Key: MAPREDUCE-5933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mr-am
Reporter: Zhijie Shen
Assignee: Robert Kanter
Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch,
MAPREDUCE-5933.patch, mr_timelineserver_response.txt
Nowadays, MR AM collects the history events and writes it to HDFS for JHS to
source. With the timeline server, MR AM can put these events there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6073) Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags
[
https://issues.apache.org/jira/browse/MAPREDUCE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-6073:
Resolution: Fixed
Fix Version/s: 2.6.0
Status: Resolved (was: Patch Available)
+1 lgtm.
Will commit to branch-2 and trunk.
Thanks!
Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default
should be moved into description tags
--
Key: MAPREDUCE-6073
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6073
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6073.1.patch
Currently, description of mapreduce.job.speculative.slowtaskthreshold in
mapred-default is put outside of description tags. We should move it into
description tags.
{code}
property
namemapreduce.job.speculative.slowtaskthreshold/name
value1.0/valueThe number of standard deviations by which a task's
ave progress-rates must be lower than the average of all running tasks'
for the task to be considered too slow.
description
/description
/property
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6073) Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags
[
https://issues.apache.org/jira/browse/MAPREDUCE-6073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149764#comment-14149764
]
Hudson commented on MAPREDUCE-6073:
---
FAILURE: Integrated in Hadoop-trunk-Commit #6122 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6122/])
MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold in
mapred-default should be moved into description tags (Tsuyoshi OZAWA via aw)
(aw: rev 1147b9ae16186ef4637269f615cfa87f16c1b5a8)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default
should be moved into description tags
--
Key: MAPREDUCE-6073
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6073
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
Priority: Trivial
Fix For: 2.6.0
Attachments: MAPREDUCE-6073.1.patch
Currently, description of mapreduce.job.speculative.slowtaskthreshold in
mapred-default is put outside of description tags. We should move it into
description tags.
{code}
property
namemapreduce.job.speculative.slowtaskthreshold/name
value1.0/valueThe number of standard deviations by which a task's
ave progress-rates must be lower than the average of all running tasks'
for the task to be considered too slow.
description
/description
/property
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6072) Update INSTALL document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149767#comment-14149767 ] Allen Wittenauer commented on MAPREDUCE-6072: - I'm of mixed feelings on this patch. While it is certainly desirable to point people to the web docs for installation, specifically to current is tricky because in the case of an incompatible change, those docs may be wrong. An alternative idea: Just remove the file. I know! Crazy idea! But bear with me: INSTALL inside of the hadoop-mapreduce-project tree is completely the wrong place for this information. The other trees don't have similar docs. We also tell people how to build the documentation inside BUILDING.txt. Users are going to see that first. So let's just remove and solve the problem. There probably should be a pointer inside BUILDING.txt that says Look for these HTML files, after you built the docs or something. Update INSTALL document --- Key: MAPREDUCE-6072 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6072 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: MAPREDUCE-6072.2.patch, MAPREDUCE-6072.patch ./hadoop-mapreduce-project/INSTALL has become stale. The document shows * svn as SCM, now we use git * ant to compile hadoop-mapreduce-examples The document should be updated to link BUILDING.txt and web docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
Jian Fang created MAPREDUCE-6111: Summary: Hadoop users' staging directories should be under a user folder Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.5.0 Reporter: Jian Fang Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Attachment: MAPREDUCE-6111.patch Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0, 2.4.1 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6112) Negative shuffle time on History UI
Andrew Or created MAPREDUCE-6112: Summary: Negative shuffle time on History UI Key: MAPREDUCE-6112 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6112 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1 Reporter: Andrew Or Attachments: Screen Shot 2014-09-26 at 11.31.21 AM.png Just running a simple pi job, with 10 mappers and 100 iterations. See screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6112) Negative shuffle time on History UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated MAPREDUCE-6112: - Attachment: Screen Shot 2014-09-26 at 11.31.21 AM.png Negative shuffle time on History UI --- Key: MAPREDUCE-6112 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6112 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1 Reporter: Andrew Or Attachments: Screen Shot 2014-09-26 at 11.31.21 AM.png Just running a simple pi job, with 10 mappers and 100 iterations. See screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Status: Patch Available (was: Open) Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.5.0 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[
https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149820#comment-14149820
]
Hadoop QA commented on MAPREDUCE-6111:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671505/MAPREDUCE-6111.patch
against trunk revision 1147b9a.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common:
org.apache.hadoop.mapreduce.v2.util.TestMRApps
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4918//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4918//console
This message is automatically generated.
Hadoop users' staging directories should be under a user folder
---
Key: MAPREDUCE-6111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.5.0, 2.4.1
Reporter: Jian Fang
Attachments: MAPREDUCE-6111.patch
Right now, Hadoop puts all users' staging directories under
/tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for
user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for
other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold
finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it
difficult to track all users' folders without adding extra logic to exclude
other known folders.
As a result, we should move all users' folders to a user sub-folder, i.e.,
/tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder
becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5988) Fix dead links to the javadocs in mapreduce project
[ https://issues.apache.org/jira/browse/MAPREDUCE-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-5988: Status: Patch Available (was: Open) Fix dead links to the javadocs in mapreduce project --- Key: MAPREDUCE-5988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5988 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.1 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5988.2.patch, MAPREDUCE-5988.patch In http://hadoop.apache.org/docs/r2.4.1/api/allclasses-frame.html, some classes are listed, but not documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5988) Fix dead links to the javadocs in mapreduce project
[ https://issues.apache.org/jira/browse/MAPREDUCE-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-5988: Status: Open (was: Patch Available) Fix dead links to the javadocs in mapreduce project --- Key: MAPREDUCE-5988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5988 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.1 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5988.2.patch, MAPREDUCE-5988.patch In http://hadoop.apache.org/docs/r2.4.1/api/allclasses-frame.html, some classes are listed, but not documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5796) Use current version of the archive name in DistributedCacheDeploy document
[
https://issues.apache.org/jira/browse/MAPREDUCE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-5796:
Resolution: Fixed
Fix Version/s: 2.6.0
Status: Resolved (was: Patch Available)
+1 lgtm.
Committing to trunk and branch-2.
Thanks!
Use current version of the archive name in DistributedCacheDeploy document
--
Key: MAPREDUCE-5796
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5796
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Fix For: 2.6.0
Attachments: MAPREDUCE-5796.patch
The archive name is {{hadoop-mapreduce-2.1.1.tar.gz}} in
DistributedCacheDeploy document but Hadoop 2.1.1 is not released. It should
be fixed to {{hadoop-mapreduce-$\{project.version\}.tar.gz}} to show the
current version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5796) Use current version of the archive name in DistributedCacheDeploy document
[
https://issues.apache.org/jira/browse/MAPREDUCE-5796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149901#comment-14149901
]
Hudson commented on MAPREDUCE-5796:
---
FAILURE: Integrated in Hadoop-trunk-Commit #6126 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6126/])
MAPREDUCE-5796. Use current version of the archive name in
DistributedCacheDeploy document (Akira AJISAKA via aw) (aw: rev
b40f433cf8ff5f6d93d7376909e53de36e6b52cb)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/DistributedCacheDeploy.apt.vm
Use current version of the archive name in DistributedCacheDeploy document
--
Key: MAPREDUCE-5796
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5796
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: documentation
Affects Versions: 2.3.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Fix For: 2.6.0
Attachments: MAPREDUCE-5796.patch
The archive name is {{hadoop-mapreduce-2.1.1.tar.gz}} in
DistributedCacheDeploy document but Hadoop 2.1.1 is not released. It should
be fixed to {{hadoop-mapreduce-$\{project.version\}.tar.gz}} to show the
current version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Attachment: MAPREDUCE-6111.patch Upload patch with fix for unit test. Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0, 2.4.1 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch, MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Status: Open (was: Patch Available) Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.5.0 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Attachment: (was: MAPREDUCE-6111.patch) Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0, 2.4.1 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[ https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian Fang updated MAPREDUCE-6111: - Status: Patch Available (was: Open) Resubmit patch with updated unit test Hadoop users' staging directories should be under a user folder --- Key: MAPREDUCE-6111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.5.0 Reporter: Jian Fang Attachments: MAPREDUCE-6111.patch Right now, Hadoop puts all users' staging directories under /tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it difficult to track all users' folders without adding extra logic to exclude other known folders. As a result, we should move all users' folders to a user sub-folder, i.e., /tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6072) Update INSTALL document
[ https://issues.apache.org/jira/browse/MAPREDUCE-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14149971#comment-14149971 ] Akira AJISAKA commented on MAPREDUCE-6072: -- Thank Allen for the comment. I agree with you that the doc should be removed. What do you think, [~ozawa]? Update INSTALL document --- Key: MAPREDUCE-6072 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6072 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Attachments: MAPREDUCE-6072.2.patch, MAPREDUCE-6072.patch ./hadoop-mapreduce-project/INSTALL has become stale. The document shows * svn as SCM, now we use git * ant to compile hadoop-mapreduce-examples The document should be updated to link BUILDING.txt and web docs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5945) Update the description of GenericOptionsParser -jt option
[ https://issues.apache.org/jira/browse/MAPREDUCE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-5945: Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available) +1 lgtm! Committed to branch-2 and trunk. Thanks! Update the description of GenericOptionsParser -jt option - Key: MAPREDUCE-5945 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5945 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Fix For: 2.6.0 Attachments: MAPREDUCE-5945.patch Now -jt option is used to specify the address of ResourceManager but document says -jt option specifies JobTracker. The document should be updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[
https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150015#comment-14150015
]
Hadoop QA commented on MAPREDUCE-6111:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671540/MAPREDUCE-6111.patch
against trunk revision b40f433.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4919//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4919//console
This message is automatically generated.
Hadoop users' staging directories should be under a user folder
---
Key: MAPREDUCE-6111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.5.0, 2.4.1
Reporter: Jian Fang
Attachments: MAPREDUCE-6111.patch
Right now, Hadoop puts all users' staging directories under
/tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for
user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for
other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold
finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it
difficult to track all users' folders without adding extra logic to exclude
other known folders.
As a result, we should move all users' folders to a user sub-folder, i.e.,
/tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder
becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5945) Update the description of GenericOptionsParser -jt option
[ https://issues.apache.org/jira/browse/MAPREDUCE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150030#comment-14150030 ] Hudson commented on MAPREDUCE-5945: --- FAILURE: Integrated in Hadoop-trunk-Commit #6127 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6127/]) MAPREDUCE-5945. Update the description of GenericOptionsParser -jt option (Akira AJISAKA via aw) (aw: rev c7c8e388a953b904b24d3343e5b069b0984244e0) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/pipes/TestPipeApplication.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm Update the description of GenericOptionsParser -jt option - Key: MAPREDUCE-5945 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5945 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.4.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Labels: newbie Fix For: 2.6.0 Attachments: MAPREDUCE-5945.patch Now -jt option is used to specify the address of ResourceManager but document says -jt option specifies JobTracker. The document should be updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-6087: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2, thanks [~ajisakaa] ! MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong Key: MAPREDUCE-6087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Labels: newbie Attachments: MAPREDUCE-6087.2.patch, MAPREDUCE-6087.patch The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS now has double prefix as yarn.app.mapreduce. + yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6087) MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-6087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150309#comment-14150309 ] Hudson commented on MAPREDUCE-6087: --- FAILURE: Integrated in Hadoop-trunk-Commit #6131 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6131/]) MAPREDUCE-6087. Fixed wrong config name of MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS. Contributed by Akira AJISAKA (jianhe: rev 5f16c98ad66b562760d5d947151a38f14875d21a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS config name is wrong Key: MAPREDUCE-6087 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6087 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.3.0 Reporter: Jian He Assignee: Akira AJISAKA Labels: newbie Attachments: MAPREDUCE-6087.2.patch, MAPREDUCE-6087.patch The config name for MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS now has double prefix as yarn.app.mapreduce. + yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
