[jira] [Resolved] (MAPREDUCE-6132) Rumen unable to accept hdfs as scheme
[ https://issues.apache.org/jira/browse/MAPREDUCE-6132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA resolved MAPREDUCE-6132. -- Resolution: Not a Problem Closing. If you disagree, please reopen with a reason. Rumen unable to accept hdfs as scheme - Key: MAPREDUCE-6132 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6132 Project: Hadoop Map/Reduce Issue Type: Bug Components: tools/rumen Affects Versions: 2.4.1 Reporter: Mayank Mishra Priority: Minor while running, java -cp hadoop-2.4.1/share/hadoop/common/hadoop-common-2.4.1.jar:hadoop-2.4.1/share/hadoop/tools/lib/hadoop-rumen-2.4.1.jar:hadoop-2.4.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:hadoop-2.4.1/share/hadoop/common/lib/commons-cli-1.2.jar:hadoop-2.4.1/share/hadoop/common/lib/commons-configuration-1.6.jar:hadoop-2.4.1/share/hadoop/common/lib/commons-lang-2.6.jar:hadoop-2.4.1/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:hadoop-2.4.1/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:hadoop-2.4.1/share/hadoop/tools/lib/guava-11.0.2.jar:hadoop-2.4.1/share/hadoop/tools/lib/guava-11.0.2.jar:hadoop-2.4.1/share/hadoop/tools/lib/commons-collections-3.2.1.jar:hadoop-2.4.1/share/hadoop/common/lib/hadoop-auth-2.4.1.jar:hadoop-2.4.1/share/hadoop/common/lib/slf4j-api-1.7.5.jar:hadoop-2.4.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.4.1.jar:hadoop-2.4.1/share/hadoop/common/lib/log4j-1.2.17.jar:hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:hadoop-2.4.1/share/hadoop/common/lib/log4j-1.2.17.jar org.apache.hadoop.tools.rumen.TraceBuilder file:///pathto/rumen/jobjars/job-trace.json file:///pathto/rumen/jobjars/topology hdfs://path to jhist file We are getting, java.io.IOException: No FileSystem for scheme: hdfs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.tools.rumen.TraceBuilder$MyOptions.processInputArgument(TraceBuilder.java:134) at org.apache.hadoop.tools.rumen.TraceBuilder$MyOptions.init(TraceBuilder.java:91) at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:206) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:186) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
Akira AJISAKA created MAPREDUCE-6243: Summary: Fix findbugs warnings in hadoop-rumen Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Priority: Minor There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6243: - Attachment: findbugs.xml Attaching the detail of the warnings. Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Priority: Minor Attachments: findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6243: - Labels: newbie (was: ) Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Priority: Minor Labels: newbie Attachments: findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304622#comment-14304622
]
Hadoop QA commented on MAPREDUCE-6240:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696345/MAPREDUCE-6240-gera.001.patch
against trunk revision b73e776.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.TestLargeSort
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
The following test timeouts occurred in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
org.apache.hadoop.mapred.TestMerge
org.apache.hadoop.mapreduce.v2.TestUberAM
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5151//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5151//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5151//console
This message is automatically generated.
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-6240-gera.001.patch, MAPREDUCE-6240.1.patch
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please
[jira] [Assigned] (MAPREDUCE-6243) Fix findbugs warnings in hadoop-rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-6243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned MAPREDUCE-6243: --- Assignee: Masatake Iwasaki Fix findbugs warnings in hadoop-rumen - Key: MAPREDUCE-6243 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6243 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tools/rumen Affects Versions: 2.6.0 Reporter: Akira AJISAKA Assignee: Masatake Iwasaki Priority: Minor Labels: newbie Attachments: findbugs.xml There are 7 findbugs warnings in hadoop-rumen modules. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303369#comment-14303369
]
Hadoop QA commented on MAPREDUCE-5718:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694551/MAPREDUCE-5718.v2.patch
against trunk revision 8cb4731.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestMRAppMaster
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5147//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5147//console
This message is automatically generated.
MR job will fail after commit fail
--
Key: MAPREDUCE-5718
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.3.0, 2.6.0
Reporter: Karthik Kambatla
Assignee: Yang Hao
Fix For: 2.6.0
Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch
when any of this happens:
* While testing RM HA, if the RM fails over while an MR AM is in the middle
of a commit,
* When testing preempting, if the MR AM fails over during the middle of a
commit
the subsequent AM gets spawned but dies with a diagnostic message - We
crashed durring a commit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303424#comment-14303424 ] Hudson commented on MAPREDUCE-6143: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2044 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2044/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303382#comment-14303382 ] Hudson commented on MAPREDUCE-6143: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #94 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/94/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303105#comment-14303105 ] Hudson commented on MAPREDUCE-6143: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #93 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/93/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/CHANGES.txt add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303316#comment-14303316 ] Hudson commented on MAPREDUCE-6143: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #90 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/90/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303482#comment-14303482 ] Siddhartha Azad commented on MAPREDUCE-4052: Any luck? I am using CDH5 and having the same issue submitting from Windows. Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303561#comment-14303561 ] bc Wong commented on MAPREDUCE-4052: Which minor version are you using? This patch is included in CDH 5.2. You should file a jira on issues.cloudera.org for CDH specific problems. Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6223) TestJobConf#testNegativeValueForTaskVmem failures
[
https://issues.apache.org/jira/browse/MAPREDUCE-6223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303466#comment-14303466
]
Varun Saxena commented on MAPREDUCE-6223:
-
Findbugs as mentioned earlier will be addressed by MAPREDUCE-6225 and test
failure is unrelated
TestJobConf#testNegativeValueForTaskVmem failures
-
Key: MAPREDUCE-6223
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6223
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Gera Shegalov
Assignee: Varun Saxena
Attachments: MAPREDUCE-6223.001.patch, MAPREDUCE-6223.002.patch
{code}
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.328 sec
FAILURE! - in org.apache.hadoop.conf.TestJobConf
testNegativeValueForTaskVmem(org.apache.hadoop.conf.TestJobConf) Time
elapsed: 0.089 sec FAILURE!
java.lang.AssertionError: expected:1024 but was:-1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.conf.TestJobConf.testNegativeValueForTaskVmem(TestJobConf.java:111)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303481#comment-14303481 ] Siddhartha Azad commented on MAPREDUCE-4052: I am using CDH5 while submitting from Windows to Linux. The job gets submitted and starts running. There are split files created (during the map phase I believe) but then there is an error and the client exits. If submitted from linux to linux the job runs fine. Is this patch available in the CDH5 version or does it need to be explicitly applied? Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303609#comment-14303609 ] Stephan Drescher commented on MAPREDUCE-6241: - SUCCESS: After replacing on line 788 #ifdef(USE_HARDWARE_CRC32C) with #if defined(USE_HARDWARE_CRC32C) defined(__GNUC__) (defined(__amd64__) || defined(__i386)) (repeating the conditional from the top) compilation phase passes successfully. SUCCESS: Hadoop distro was built ( package -Pdist,native) for ppc64 as (32-bit version) completely. Next, I will continue with testing. This bug was issued concerning a compilation problem. Hence, I will change the state to resolved. Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC Key: MAPREDUCE-6241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux gcc (Debian 4.9.1-19) protobuf 2.6.1 OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) OpenJDK Zero VM (build 24.65-b04, interpreted mode) source was cloned (and updated) from Apache-Hadoop's git repository Reporter: Stephan Drescher Assignee: Binglin Chang Priority: Minor Labels: features Attachments: MAPREDUCE-6241.001.patch Issue when using assembler code for performance optimization on the powerpc platform (compiled for 32bit) mvn compile -Pnative -DskipTests [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux -isystem /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed [exec] make[2]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/nativetask.dir/all' failed [exec] make[1]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] Makefile:76: recipe for target 'all' failed [exec] /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: In function ‘void NativeTask::init_cpu_support_flag()’: /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: error: impossible register constraint in ‘asm’ -- popl %%ebx : =a (eax), [ebx] =r(ebx), =c(ecx), =d(edx) : a (eax_in) : cc); -- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303560#comment-14303560 ] bc Wong commented on MAPREDUCE-4052: Which minor version are you using? This patch is included in CDH 5.2. You should file a jira on issues.cloudera.org for CDH specific problems. Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303132#comment-14303132 ] Hudson commented on MAPREDUCE-6143: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #827 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/827/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * hadoop-mapreduce-project/CHANGES.txt add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6241: - Attachment: MAPREDUCE-6241.001.patch This patch makes Checksum.cc more compatible for compiling, just like code in hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC Key: MAPREDUCE-6241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux gcc (Debian 4.9.1-19) protobuf 2.6.1 OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) OpenJDK Zero VM (build 24.65-b04, interpreted mode) source was cloned (and updated) from Apache-Hadoop's git repository Reporter: Stephan Drescher Assignee: Binglin Chang Priority: Minor Labels: features Attachments: MAPREDUCE-6241.001.patch Issue when using assembler code for performance optimization on the powerpc platform (compiled for 32bit) mvn compile -Pnative -DskipTests [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux -isystem /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed [exec] make[2]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/nativetask.dir/all' failed [exec] make[1]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] Makefile:76: recipe for target 'all' failed [exec] /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: In function ‘void NativeTask::init_cpu_support_flag()’: /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: error: impossible register constraint in ‘asm’ -- popl %%ebx : =a (eax), [ebx] =r(ebx), =c(ecx), =d(edx) : a (eax_in) : cc); -- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: MAPREDUCE-6232.v2.patch Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[
https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303198#comment-14303198
]
Hadoop QA commented on MAPREDUCE-6241:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696164/MAPREDUCE-6241.001.patch
against trunk revision 8cb4731.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5145//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5145//console
This message is automatically generated.
Native compilation fails for Checksum.cc due to an incompatibility of
assembler register constraint for PowerPC
Key: MAPREDUCE-6241
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build
Affects Versions: 3.0.0
Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux
gcc (Debian 4.9.1-19)
protobuf 2.6.1
OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2)
OpenJDK Zero VM (build 24.65-b04, interpreted mode)
source was cloned (and updated) from Apache-Hadoop's git repository
Reporter: Stephan Drescher
Assignee: Binglin Chang
Priority: Minor
Labels: features
Attachments: MAPREDUCE-6241.001.patch
Issue when using assembler code for performance optimization on the powerpc
platform (compiled for 32bit)
mvn compile -Pnative -DskipTests
[exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY
-fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
-I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native
-I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux
-isystem
/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include
-o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c
/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
[exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target
'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed
[exec] make[2]: Leaving directory
'/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native'
[exec] CMakeFiles/Makefile2:95: recipe for target
'CMakeFiles/nativetask.dir/all' failed
[exec] make[1]: Leaving directory
'/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native'
[exec] Makefile:76: recipe for
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: (was: MAPREDUCE-6232.v2.patch) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Attachment: MAPREDUCE-6232.v2.patch Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6232) Task state is running when all task attempts fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-6232: Fix Version/s: 2.6.0 Target Version/s: 2.6.0 Status: Patch Available (was: Open) Task state is running when all task attempts fail - Key: MAPREDUCE-6232 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 2.6.0 Reporter: Yang Hao Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch, TaskImpl.new.png, TaskImpl.normal.png, result.pdf When task attempts fails, the task's state is still running. A clever way is to check the task attempts's state, if none of the attempts is running, then the task state should not be running -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-6241: - Assignee: Binglin Chang Status: Patch Available (was: Open) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC Key: MAPREDUCE-6241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux gcc (Debian 4.9.1-19) protobuf 2.6.1 OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) OpenJDK Zero VM (build 24.65-b04, interpreted mode) source was cloned (and updated) from Apache-Hadoop's git repository Reporter: Stephan Drescher Assignee: Binglin Chang Priority: Minor Labels: features Issue when using assembler code for performance optimization on the powerpc platform (compiled for 32bit) mvn compile -Pnative -DskipTests [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux -isystem /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed [exec] make[2]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/nativetask.dir/all' failed [exec] make[1]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] Makefile:76: recipe for target 'all' failed [exec] /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: In function ‘void NativeTask::init_cpu_support_flag()’: /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: error: impossible register constraint in ‘asm’ -- popl %%ebx : =a (eax), [ebx] =r(ebx), =c(ecx), =d(edx) : a (eax_in) : cc); -- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang moved HADOOP-11537 to MAPREDUCE-6241: --- Component/s: (was: build) build Target Version/s: 3.0.0 (was: 3.0.0) Affects Version/s: (was: 3.0.0) 3.0.0 Key: MAPREDUCE-6241 (was: HADOOP-11537) Project: Hadoop Map/Reduce (was: Hadoop Common) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC Key: MAPREDUCE-6241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0 Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux gcc (Debian 4.9.1-19) protobuf 2.6.1 OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) OpenJDK Zero VM (build 24.65-b04, interpreted mode) source was cloned (and updated) from Apache-Hadoop's git repository Reporter: Stephan Drescher Priority: Minor Labels: features Issue when using assembler code for performance optimization on the powerpc platform (compiled for 32bit) mvn compile -Pnative -DskipTests [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux -isystem /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed [exec] make[2]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] CMakeFiles/Makefile2:95: recipe for target 'CMakeFiles/nativetask.dir/all' failed [exec] make[1]: Leaving directory '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' [exec] Makefile:76: recipe for target 'all' failed [exec] /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: In function ‘void NativeTask::init_cpu_support_flag()’: /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: error: impossible register constraint in ‘asm’ -- popl %%ebx : =a (eax), [ebx] =r(ebx), =c(ecx), =d(edx) : a (eax_in) : cc); -- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao updated MAPREDUCE-5718: Fix Version/s: 2.6.0 Labels: (was: ha) Affects Version/s: 2.6.0 Status: Patch Available (was: Reopened) MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.6.0, 2.3.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6143) add configuration for mapreduce speculative execution in MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303293#comment-14303293 ] Hudson commented on MAPREDUCE-6143: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2025 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2025/]) MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) (rkanter: rev 8acc5e9b4b3fea8b418b3526c15022c8a9fedd56) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java add configuration for mapreduce speculative execution in MR2 - Key: MAPREDUCE-6143 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6143 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 2.7.0 Attachments: MAPREDUCE-6143.000.patch, MAPREDUCE-6143.001.patch, MAPREDUCE-6143.002.patch add configuration for mapreduce speculative execution in MR2. Currently mapreduce.job.speculative.speculativecap and mapreduce.job.speculative.slownodethreshold are not used for MR2 mapreduce speculative execution any more. We should make the following hardcode constants in DefaultSpeculator configurable for MR2 Map Reduce speculative execution: private static final long SOONEST_RETRY_AFTER_NO_SPECULATE = 1000L * 1L; private static final long SOONEST_RETRY_AFTER_SPECULATE = 1000L * 15L; private static final double PROPORTION_RUNNING_TASKS_SPECULATABLE = 0.1; private static final double PROPORTION_TOTAL_TASKS_SPECULATABLE = 0.01; private static final int MINIMUM_ALLOWED_SPECULATIVE_TASKS = 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6232) Task state is running when all task attempts fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303244#comment-14303244
]
Hadoop QA commented on MAPREDUCE-6232:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696173/MAPREDUCE-6232.v2.patch
against trunk revision 8cb4731.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1191 javac
compiler warnings (more than the trunk's current 1187 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-maven-plugins.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5146//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5146//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5146//artifact/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5146//console
This message is automatically generated.
Task state is running when all task attempts fail
-
Key: MAPREDUCE-6232
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6232
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 2.6.0
Reporter: Yang Hao
Assignee: Yang Hao
Fix For: 2.6.0
Attachments: MAPREDUCE-6232.patch, MAPREDUCE-6232.v2.patch,
TaskImpl.new.png, TaskImpl.normal.png, result.pdf
When task attempts fails, the task's state is still running. A clever way is
to check the task attempts's state, if none of the attempts is running, then
the task state should not be running
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6237) DBRecordReader is not thread safe
[ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303715#comment-14303715 ] Tsuyoshi OZAWA commented on MAPREDUCE-6237: --- [~rkannan82] Thank you for the contribution. I prefer to keep the method getConnection since it's public method. Also, TestDbClasses shouldn't be changed. About the connection pool, I'd like to +1 to do on separate JIRA. DBRecordReader is not thread safe - Key: MAPREDUCE-6237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0 Reporter: Kannan Rajah Assignee: Kannan Rajah Attachments: mapreduce-6237.patch DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader. This is not a good idea. We should be creating separate connection. If performance is a concern, then we should be using connection pooling instead. I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object for each DBRecordReader. So can we just change DBInputFormat to create new Connection every time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443. Any reason for caching the connection? We observed this issue in a customer setup where they were reading data from MySQL using Pig. As per customer, the query is returning two records which causes Pig to create two instances of DBRecordReader. These two instances are sharing the database connection instance. The first DBRecordReader runs to extract the first record from MySQL just fine, but then closes the shared connection instance. When the second DBRecordReader runs, it tries to execute a query to retrieve the second record on the closed shared connection instance, which fail. If we set mapred.map.tasks to 1, the query will be successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303706#comment-14303706
]
Gera Shegalov commented on MAPREDUCE-6240:
--
Hi [~kamrul], I actually ran into this problem with my recent project, and have
a fix that I could submit later today.
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:449)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:372)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:370)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:379)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1185)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:927)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303763#comment-14303763 ] Siddhartha Azad commented on MAPREDUCE-4052: I am using CDH 5.2.0 jars. The job gets submitted to linux from windows but doesn't complete. Same job works fine from Linux. I have the following property settings in mapred-site.xml: property namemapreduce.app-submission.cross-platform/name valuetrue/value /property Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Fix For: 2.4.0 Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6242) Progress report log is incredibly excessive in application master
[ https://issues.apache.org/jira/browse/MAPREDUCE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli moved YARN-3130 to MAPREDUCE-6242: -- Component/s: (was: applications) applicationmaster Affects Version/s: (was: 2.4.0) 2.4.0 Key: MAPREDUCE-6242 (was: YARN-3130) Project: Hadoop Map/Reduce (was: Hadoop YARN) Progress report log is incredibly excessive in application master - Key: MAPREDUCE-6242 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6242 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.0 Reporter: Jian Fang We saw incredibly excessive logs in application master for a long running one with many task attempts. The log write rate is around 1MB/sec in some cases. Most of the log entries were from the progress report such as the following ones. 2015-02-03 17:46:14,321 INFO [IPC Server handler 56 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.15605757 2015-02-03 17:46:17,581 INFO [IPC Server handler 2 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.4108217 2015-02-03 17:46:20,426 INFO [IPC Server handler 0 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_02_0 is : 0.06634143 2015-02-03 17:46:20,807 INFO [IPC Server handler 4 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.6506 2015-02-03 17:46:21,013 INFO [IPC Server handler 6 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_01_0 is : 0.21723115 Looks like the report interval is controlled by a hard-coded variable PROGRESS_INTERVAL as 3 seconds in class org.apache.hadoop.mapred.Task. We should allow users to set the appropriate progress interval for their applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5988) Fix dead links to the javadocs in mapreduce project
[ https://issues.apache.org/jira/browse/MAPREDUCE-5988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-5988: - Target Version/s: 2.7.0 (was: 2.6.0) Fix dead links to the javadocs in mapreduce project --- Key: MAPREDUCE-5988 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5988 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.1 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Priority: Minor Attachments: MAPREDUCE-5988.2.patch, MAPREDUCE-5988.patch In http://hadoop.apache.org/docs/r2.4.1/api/allclasses-frame.html, some classes are listed, but not documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304172#comment-14304172
]
Akira AJISAKA commented on MAPREDUCE-5800:
--
Committed this to trunk and branch-2. Thanks [~qwertymaniac] for the review!
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Fix For: 2.7.0
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5800:
-
Resolution: Fixed
Fix Version/s: 2.7.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Fix For: 2.7.0
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6237) DBRecordReader is not thread safe
[ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304294#comment-14304294 ] Kannan Rajah commented on MAPREDUCE-6237: - Fixed the patch to retain getConnection method. Reverted the changes to TestDbClasses. DBRecordReader is not thread safe - Key: MAPREDUCE-6237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0 Reporter: Kannan Rajah Assignee: Kannan Rajah Attachments: mapreduce-6237.patch, mapreduce-6237.patch DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader. This is not a good idea. We should be creating separate connection. If performance is a concern, then we should be using connection pooling instead. I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object for each DBRecordReader. So can we just change DBInputFormat to create new Connection every time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443. Any reason for caching the connection? We observed this issue in a customer setup where they were reading data from MySQL using Pig. As per customer, the query is returning two records which causes Pig to create two instances of DBRecordReader. These two instances are sharing the database connection instance. The first DBRecordReader runs to extract the first record from MySQL just fine, but then closes the shared connection instance. When the second DBRecordReader runs, it tries to execute a query to retrieve the second record on the closed shared connection instance, which fail. If we set mapred.map.tasks to 1, the query will be successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6237) DBRecordReader is not thread safe
[ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304291#comment-14304291 ] Kannan Rajah commented on MAPREDUCE-6237: - Fixed the patch to retain getConnection method. Reverted the changes to TestDbClasses. DBRecordReader is not thread safe - Key: MAPREDUCE-6237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0 Reporter: Kannan Rajah Assignee: Kannan Rajah Attachments: mapreduce-6237.patch, mapreduce-6237.patch DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader. This is not a good idea. We should be creating separate connection. If performance is a concern, then we should be using connection pooling instead. I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object for each DBRecordReader. So can we just change DBInputFormat to create new Connection every time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443. Any reason for caching the connection? We observed this issue in a customer setup where they were reading data from MySQL using Pig. As per customer, the query is returning two records which causes Pig to create two instances of DBRecordReader. These two instances are sharing the database connection instance. The first DBRecordReader runs to extract the first record from MySQL just fine, but then closes the shared connection instance. When the second DBRecordReader runs, it tries to execute a query to retrieve the second record on the closed shared connection instance, which fail. If we set mapred.map.tasks to 1, the query will be successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[ https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304178#comment-14304178 ] Hudson commented on MAPREDUCE-5800: --- FAILURE: Integrated in Hadoop-trunk-Commit #6995 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6995/]) MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors. (aajisaka) (aajisaka: rev bd69fb2d44403e930d1fc0868ed1dd2a49dd9659) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/ControlledJob.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/dancing/DistributedPentomino.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Grep.java * hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/mapreduce/TestStreamXmlRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/LargeSorter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/MultipleOutputs.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRChildTask.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomWriter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/RandomTextWriterJob.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/BaileyBorweinPlouffe.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/aggregate/ValueAggregatorJob.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomWriter.java * hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/GridmixJob.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/MultiFileWordCount.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/WordStandardDeviation.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/InputSampler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/InputSampler.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/db/DBInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/jobcontrol/Job.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/WordMedian.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMiniMRClientCluster.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/join/Parser.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/pi/DistSum.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/DelegatingInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/CombineFileInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/testjar/UserNamePermission.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/QuasiMonteCarlo.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Sort.java * hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestCompressionEmulationUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/SecondarySort.java * hadoop-mapreduce-project/CHANGES.txt *
[jira] [Commented] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304174#comment-14304174
]
Mohammad Kamrul Islam commented on MAPREDUCE-6240:
--
I also prepared a patch as well.
I will upload mine too. Then we can combine.
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:449)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:372)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:370)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:379)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1185)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:927)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6237) DBRecordReader is not thread safe
[ https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kannan Rajah updated MAPREDUCE-6237: Attachment: mapreduce-6237.patch DBRecordReader is not thread safe - Key: MAPREDUCE-6237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.5.0 Reporter: Kannan Rajah Assignee: Kannan Rajah Attachments: mapreduce-6237.patch, mapreduce-6237.patch DBInputFormat.createDBRecorder is reusing JDBC connections across instances of DBRecordReader. This is not a good idea. We should be creating separate connection. If performance is a concern, then we should be using connection pooling instead. I looked at DBOutputFormat.getRecordReader. It actually creates a new Connection object for each DBRecordReader. So can we just change DBInputFormat to create new Connection every time? The connection reuse code was added as part of connection leak bug in MAPREDUCE-1443. Any reason for caching the connection? We observed this issue in a customer setup where they were reading data from MySQL using Pig. As per customer, the query is returning two records which causes Pig to create two instances of DBRecordReader. These two instances are sharing the database connection instance. The first DBRecordReader runs to extract the first record from MySQL just fine, but then closes the shared connection instance. When the second DBRecordReader runs, it tries to execute a query to retrieve the second record on the closed shared connection instance, which fail. If we set mapred.map.tasks to 1, the query will be successful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated MAPREDUCE-5800:
-
Attachment: MAPREDUCE-5800.2.patch
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303935#comment-14303935
]
Akira AJISAKA commented on MAPREDUCE-5800:
--
Thanks [~qwertymaniac] for the comment. Resubmitting.
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Kamrul Islam updated MAPREDUCE-6240:
-
Attachment: MAPREDUCE-6240.1.patch
[~jira.shegalov] this is my patch. Please put yours. We can definitely combine
both.
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-6240.1.patch
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:449)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:372)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:370)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:379)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1185)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:927)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6237) DBRecordReader is not thread safe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304363#comment-14304363
]
Hadoop QA commented on MAPREDUCE-6237:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696317/mapreduce-6237.patch
against trunk revision b73e776.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5149//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5149//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5149//console
This message is automatically generated.
DBRecordReader is not thread safe
-
Key: MAPREDUCE-6237
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6237
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.5.0
Reporter: Kannan Rajah
Assignee: Kannan Rajah
Attachments: mapreduce-6237.patch, mapreduce-6237.patch
DBInputFormat.createDBRecorder is reusing JDBC connections across instances
of DBRecordReader. This is not a good idea. We should be creating separate
connection. If performance is a concern, then we should be using connection
pooling instead.
I looked at DBOutputFormat.getRecordReader. It actually creates a new
Connection object for each DBRecordReader. So can we just change
DBInputFormat to create new Connection every time? The connection reuse code
was added as part of connection leak bug in MAPREDUCE-1443. Any reason for
caching the connection?
We observed this issue in a customer setup where they were reading data from
MySQL using Pig. As per customer, the query is returning two records which
causes Pig to create two instances of DBRecordReader. These two instances are
sharing the database connection instance. The first DBRecordReader runs to
extract the first record from MySQL just fine, but then closes the shared
connection instance. When the second DBRecordReader runs, it tries to execute
a query to retrieve the second record on the closed shared connection
instance, which fail. If we set
mapred.map.tasks to 1, the query will be successful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Kamrul Islam updated MAPREDUCE-6240:
-
Status: Patch Available (was: Open)
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-6240.1.patch
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:449)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:372)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:370)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.oozie.service.HadoopAccessorService.createJobClient(HadoopAccessorService.java:379)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.createJobClient(JavaActionExecutor.java:1185)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:927)
... 10 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304063#comment-14304063
]
Harsh J commented on MAPREDUCE-5800:
Went over the changes again, and they still look good to me. I also ran a few
of the modified tests cause Jenkins didn't run them. +1, please commit, thanks
again!
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5800) Use Job#getInstance instead of deprecated constructors
[
https://issues.apache.org/jira/browse/MAPREDUCE-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304018#comment-14304018
]
Hadoop QA commented on MAPREDUCE-5800:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696278/MAPREDUCE-5800.2.patch
against trunk revision 21d80b3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5148//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5148//console
This message is automatically generated.
Use Job#getInstance instead of deprecated constructors
--
Key: MAPREDUCE-5800
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5800
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.3.0, 2.6.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-5800.2.patch, MAPREDUCE-5800.2.patch,
MAPREDUCE-5800.patch
There're some methods calling deprecated constructors such as {{new Job()}},
which causes javac warnings.
We should use {{Job.getInstance()}} to get an instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5718) MR job will fail after commit fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304041#comment-14304041 ] Karthik Kambatla commented on MAPREDUCE-5718: - [~yanghaogn] - initially, I was also trying to delete the startCommitFile if there is not corresponding endFile. However, we can't do that for reasons Jason described here - https://issues.apache.org/jira/browse/MAPREDUCE-5718?focusedCommentId=13872189page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13872189 MR job will fail after commit fail -- Key: MAPREDUCE-5718 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5718 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0, 2.6.0 Reporter: Karthik Kambatla Assignee: Yang Hao Fix For: 2.6.0 Attachments: MAPREDUCE-5718.v2.patch, mr-5718-0.patch when any of this happens: * While testing RM HA, if the RM fails over while an MR AM is in the middle of a commit, * When testing preempting, if the MR AM fails over during the middle of a commit the subsequent AM gets spawned but dies with a diagnostic message - We crashed durring a commit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6242) Progress report log is incredibly excessive in application master
[ https://issues.apache.org/jira/browse/MAPREDUCE-6242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena reassigned MAPREDUCE-6242: --- Assignee: Varun Saxena Progress report log is incredibly excessive in application master - Key: MAPREDUCE-6242 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6242 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.4.0 Reporter: Jian Fang Assignee: Varun Saxena We saw incredibly excessive logs in application master for a long running one with many task attempts. The log write rate is around 1MB/sec in some cases. Most of the log entries were from the progress report such as the following ones. 2015-02-03 17:46:14,321 INFO [IPC Server handler 56 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.15605757 2015-02-03 17:46:17,581 INFO [IPC Server handler 2 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.4108217 2015-02-03 17:46:20,426 INFO [IPC Server handler 0 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_02_0 is : 0.06634143 2015-02-03 17:46:20,807 INFO [IPC Server handler 4 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_00_0 is : 0.6506 2015-02-03 17:46:21,013 INFO [IPC Server handler 6 on 37661] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1422985365246_0001_m_01_0 is : 0.21723115 Looks like the report interval is controlled by a hard-coded variable PROGRESS_INTERVAL as 3 seconds in class org.apache.hadoop.mapred.Task. We should allow users to set the appropriate progress interval for their applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-6240:
-
Attachment: MAPREDUCE-6240-gera.001.patch
Hi [~kamrul], my patch is similar. It chains exceptions like this:
{code}
java.io.IOException: Cannot initialize Cluster. Please check your configuration
for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:127)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at
org.apache.hadoop.mapreduce.TestClientProtocolProviderImpls.testClusterException(TestClientProtocolProviderImpls.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:254)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:149)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: Failed to use
org.apache.hadoop.mapred.YarnClientProtocolProvider
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
... 22 more
Caused by: java.lang.RuntimeException: Error in instantiating YarnClient
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:140)
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:122)
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:112)
at
org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95)
... 22 more
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException:
fs.AbstractFileSystem.nosuchfs.impl=null: No AbstractFileSystem configured for
scheme: nosuchfs
at
org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:161)
at
org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:324)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:321)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at
org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:321)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:438)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:464)
at org.apache.hadoop.mapred.YARNRunner.init(YARNRunner.java:138)
... 26 more
{code}
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-6240-gera.001.patch, MAPREDUCE-6240.1.patch
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log
[jira] [Commented] (MAPREDUCE-6240) Hadoop client displays confusing error message
[
https://issues.apache.org/jira/browse/MAPREDUCE-6240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304460#comment-14304460
]
Hadoop QA commented on MAPREDUCE-6240:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12696338/MAPREDUCE-6240.1.patch
against trunk revision b73e776.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5150//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5150//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5150//console
This message is automatically generated.
Hadoop client displays confusing error message
--
Key: MAPREDUCE-6240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mohammad Kamrul Islam
Assignee: Mohammad Kamrul Islam
Attachments: MAPREDUCE-6240-gera.001.patch, MAPREDUCE-6240.1.patch
Hadoop client often throws exception with java.io.IOException: Cannot
initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
This is a misleading and generic message for any cluster initialization
problem. It takes a lot of debugging hours to identify the root cause. The
correct error message could resolve this problem quickly.
In one such instance, Oozie log showed the following exception while the
root cause was CNF that Hadoop client didn't return in the exception.
{noformat}
JA009: Cannot initialize Cluster. Please check your configuration for
mapreduce.framework.name and the correspond server addresses.
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:392)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:979)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1134)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:281)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
at
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your
configuration for mapreduce.framework.name and the correspond server
addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:82)
at org.apache.hadoop.mapreduce.Cluster.init(Cluster.java:75)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:449)
at
org.apache.oozie.service.HadoopAccessorService$1.run(HadoopAccessorService.java:372)
at
