[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934708#comment-13934708
]
Hadoop QA commented on MAPREDUCE-4052:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634647/MAPREDUCE-4052.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4424//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4424//console
This message is automatically generated.
Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop
cluster.
---
Key: MAPREDUCE-4052
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Affects Versions: 0.23.1, 2.2.0
Environment: client on the Windows, the the cluster on the suse
Reporter: xieguiming
Assignee: Jian He
Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch,
MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch,
MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch,
MAPREDUCE-4052.patch
when I use the eclipse on the windows to submit the job. and the
applicationmaster throw the exception:
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
The reasion is :
class Apps addToEnvironment function, use the
private static final String SYSTEM_PATH_SEPARATOR =
System.getProperty(path.separator);
and will result the MRApplicationMaster classpath use the ; separator.
I suggest that nodemanger do the replace.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[ https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikola Vujic updated MAPREDUCE-5791: Attachment: MAPREDUCE-5791.patch I have submitted a new patch, fixed according to your comments. Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently Key: MAPREDUCE-5791 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Nikola Vujic Assignee: Nikola Vujic Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using transferTo method from a FileChannel to transfer data from a disk to socket. This is performing slow in Windows, slower than in Linux. The reason is that transferTo method for the java.nio is issuing 32K IO requests all the time. In Windows, these 32K transfers are not optimal and we don't get the best performance form the underlying IO subsystem. In order to achieve better performance when reading from the drives, we need to read data in bigger chunks, 512K for example. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5456) TestFetcher.testCopyFromHostExtraBytes is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934869#comment-13934869 ] Hudson commented on MAPREDUCE-5456: --- FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/509/]) MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577313) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java TestFetcher.testCopyFromHostExtraBytes is missing - Key: MAPREDUCE-5456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5456.patch, MAPREDUCE-5456.patch I noticed that the test to verify the fix from MAPREDUCE-5308 was deleted by MAPREDUCE-5194. It looks like an accidental deletion from an upmerge. We should reinstate this unit test. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5713) InputFormat and JobConf JavaDoc Fixes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934868#comment-13934868 ] Hudson commented on MAPREDUCE-5713: --- FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/509/]) MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577239) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java InputFormat and JobConf JavaDoc Fixes - Key: MAPREDUCE-5713 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5713 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 1.2.1, 2.2.0 Reporter: Ben Robie Assignee: Chen He Priority: Trivial Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5713.patch, hadoop-10210.patch Original Estimate: 0.5h Remaining Estimate: 0.5h https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/InputFormat.html Instead of record boundaries are to respected Should be record boundaries are to be respected https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/JobConf.html Instead of some parameters interact subtly rest of the framework Should be some parameters interact subtly with the rest of the framework -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5794) SliveMapper always uses default FileSystem.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934874#comment-13934874 ] Hudson commented on MAPREDUCE-5794: --- FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/509/]) MAPREDUCE-5794. SliveMapper always uses default FileSystem. (Contributed by szetszwo) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577281) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/DummyInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveMapper.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SlivePartitioner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveReducer.java SliveMapper always uses default FileSystem. --- Key: MAPREDUCE-5794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5794 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor Fix For: 3.0.0, 2.4.0 Attachments: m5794_20140311.patch Similar to MAPREDUCE-5780, SliveMapper should use the test path to get FileSystem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934876#comment-13934876 ] Hudson commented on MAPREDUCE-5765: --- FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/509/]) MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577188) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Priority: Minor Labels: documentation Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5765.patch, MAPREDUCE-5765.patch wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5789) Average Reduce time is incorrect on Job Overview page
[ https://issues.apache.org/jira/browse/MAPREDUCE-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934879#comment-13934879 ] Hudson commented on MAPREDUCE-5789: --- FAILURE: Integrated in Hadoop-Yarn-trunk #509 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/509/]) MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page. Contributed by Rushabh S Shah (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577202) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java Average Reduce time is incorrect on Job Overview page - Key: MAPREDUCE-5789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5789 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.10, 2.3.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Fix For: 0.23.11, 2.4.0 Attachments: patch-MapReduce-5789-v2.patch, patch-MapReduce-5789.patch The Average Reduce time displayed on the job overview page is incorrect. Previously Reduce time was calculated as difference between finishTime and shuffleFinishTime. It should be difference of finishTime and sortFinishTime -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5791) Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not read disks efficiently
[
https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934881#comment-13934881
]
Hadoop QA commented on MAPREDUCE-5791:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634688/MAPREDUCE-5791.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4426//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4426//console
This message is automatically generated.
Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not
read disks efficiently
Key: MAPREDUCE-5791
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Nikola Vujic
Assignee: Nikola Vujic
Attachments: MAPREDUCE-5791.patch, MAPREDUCE-5791.patch
transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using
transferTo method from a FileChannel to transfer data from a disk to socket.
This is performing slow in Windows, slower than in Linux. The reason is that
transferTo method for the java.nio is issuing 32K IO requests all the time.
In Windows, these 32K transfers are not optimal and we don't get the best
performance form the underlying IO subsystem. In order to achieve better
performance when reading from the drives, we need to read data in bigger
chunks, 512K for example.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5713) InputFormat and JobConf JavaDoc Fixes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935022#comment-13935022 ] Hudson commented on MAPREDUCE-5713: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1701 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1701/]) MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577239) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java InputFormat and JobConf JavaDoc Fixes - Key: MAPREDUCE-5713 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5713 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 1.2.1, 2.2.0 Reporter: Ben Robie Assignee: Chen He Priority: Trivial Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5713.patch, hadoop-10210.patch Original Estimate: 0.5h Remaining Estimate: 0.5h https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/InputFormat.html Instead of record boundaries are to respected Should be record boundaries are to be respected https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/JobConf.html Instead of some parameters interact subtly rest of the framework Should be some parameters interact subtly with the rest of the framework -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5456) TestFetcher.testCopyFromHostExtraBytes is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935023#comment-13935023 ] Hudson commented on MAPREDUCE-5456: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1701 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1701/]) MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577313) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java TestFetcher.testCopyFromHostExtraBytes is missing - Key: MAPREDUCE-5456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5456.patch, MAPREDUCE-5456.patch I noticed that the test to verify the fix from MAPREDUCE-5308 was deleted by MAPREDUCE-5194. It looks like an accidental deletion from an upmerge. We should reinstate this unit test. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935030#comment-13935030 ] Hudson commented on MAPREDUCE-5765: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1701 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1701/]) MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577188) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Priority: Minor Labels: documentation Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5765.patch, MAPREDUCE-5765.patch wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5789) Average Reduce time is incorrect on Job Overview page
[ https://issues.apache.org/jira/browse/MAPREDUCE-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935033#comment-13935033 ] Hudson commented on MAPREDUCE-5789: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1701 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1701/]) MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page. Contributed by Rushabh S Shah (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577202) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java Average Reduce time is incorrect on Job Overview page - Key: MAPREDUCE-5789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5789 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.10, 2.3.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Fix For: 0.23.11, 2.4.0 Attachments: patch-MapReduce-5789-v2.patch, patch-MapReduce-5789.patch The Average Reduce time displayed on the job overview page is incorrect. Previously Reduce time was calculated as difference between finishTime and shuffleFinishTime. It should be difference of finishTime and sortFinishTime -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5792) When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error
[
https://issues.apache.org/jira/browse/MAPREDUCE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935044#comment-13935044
]
Jason Lowe commented on MAPREDUCE-5792:
---
bq. It becomes very hard to find AM logs for a failed AM because clicking the
logs link from the RM page takes you to the NM it executed on and with log
aggregation, the logs get pushed to HDFS very quickly, and then the NM just
throws an error that the container doesn't exist.
This is clearly a bug. We run with log aggregation and routinely have users
debugging failed AM startups by clicking on the AM log links. The link goes to
the NM which re-directs to the history server and it shows the logs. If this
isn't working then there's either a regression or the cluster isn't configured
properly. Is yarn.log.server.url configured properly so the NM can redirect to
the log server after logs have been aggregated?
When mapreduce.jobhistory.intermediate-done-dir isn't writable, application
fails with generic error
Key: MAPREDUCE-5792
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5792
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Travis Thompson
Assignee: Mohammad Kamrul Islam
When trying to run an application and the permissions are wrong on
{{mapreduce.jobhistory.intermediate-done-dir}}, the MapReduce AM fails with a
non-descriptive error message:
{noformat}
Application application_1394227890066_0004 failed 2 times due to AM Container
for appattempt_1394227890066_0004_02 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
main : command provided 1
main : user is tthompso
main : requested yarn user is tthompso
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
{noformat}
When permissions are corrected on this dir, applications are able to run.
There should probably be some sort of check on this dir before launching the
AM so a more meaningful error message can be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5792) When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error
[
https://issues.apache.org/jira/browse/MAPREDUCE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935060#comment-13935060
]
Jason Lowe commented on MAPREDUCE-5792:
---
bq. So maybe the better fix here is to get the RM to pull the logs off of HDFS
instead of linking to the NM?
The problem with this approach is that the RM may have difficulty knowing when
log aggregation has completed to know whether it should continue referencing
the NM or redirect to the log server.
bq. I'm not sure who's supposed to be handling log viewing besides the JHS
which is specific to M/R jobs.
The JHS can serve logs even for non-MR jobs. It was a hack to provide an
aggregated log server before one existed. Now in recent 2.x I believe the YARN
Application History/Timeline Server can serve up logs as well. On our 0.23
clusters we are using the JHS to serve up aggregated logs, and
yarn.log.server.url is configured to
{noformat}http://jhs-server-name:port/jobhistory/nmlogs{noformat}
When mapreduce.jobhistory.intermediate-done-dir isn't writable, application
fails with generic error
Key: MAPREDUCE-5792
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5792
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Travis Thompson
Assignee: Mohammad Kamrul Islam
When trying to run an application and the permissions are wrong on
{{mapreduce.jobhistory.intermediate-done-dir}}, the MapReduce AM fails with a
non-descriptive error message:
{noformat}
Application application_1394227890066_0004 failed 2 times due to AM Container
for appattempt_1394227890066_0004_02 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
main : command provided 1
main : user is tthompso
main : requested yarn user is tthompso
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
{noformat}
When permissions are corrected on this dir, applications are able to run.
There should probably be some sort of check on this dir before launching the
AM so a more meaningful error message can be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5751:
--
Resolution: Fixed
Fix Version/s: 2.4.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Sangjin! I committed this to trunk, branch-2, and branch-2.4.
MR app master fails to start in some cases if mapreduce.job.classloader is
true
---
Key: MAPREDUCE-5751
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: mapreduce-5751.patch, mapreduce-5751.patch
If mapreduce.job.classloader is set to true, and the MR client includes a
jetty jar in its libjars or job jar, the MR app master fails to start. A
typical stack trace we get is as follows:
{noformat}
java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration
cannot be cast to org.mortbay.jetty.webapp.Configuration
at
org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
{noformat}
This happens because as part of the MR app master start the jetty classes are
loaded normally through the app classloader, but WebAppContext tries to load
the specific Configuration class via the thread context classloader (which
had been set to the user job classloader).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5456) TestFetcher.testCopyFromHostExtraBytes is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935102#comment-13935102 ] Hudson commented on MAPREDUCE-5456: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/]) MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason Lowe via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577313) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java TestFetcher.testCopyFromHostExtraBytes is missing - Key: MAPREDUCE-5456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5456.patch, MAPREDUCE-5456.patch I noticed that the test to verify the fix from MAPREDUCE-5308 was deleted by MAPREDUCE-5194. It looks like an accidental deletion from an upmerge. We should reinstate this unit test. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5794) SliveMapper always uses default FileSystem.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935107#comment-13935107 ] Hudson commented on MAPREDUCE-5794: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/]) MAPREDUCE-5794. SliveMapper always uses default FileSystem. (Contributed by szetszwo) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577281) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/DummyInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveMapper.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SlivePartitioner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/slive/SliveReducer.java SliveMapper always uses default FileSystem. --- Key: MAPREDUCE-5794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5794 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor Fix For: 3.0.0, 2.4.0 Attachments: m5794_20140311.patch Similar to MAPREDUCE-5780, SliveMapper should use the test path to get FileSystem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5713) InputFormat and JobConf JavaDoc Fixes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935101#comment-13935101 ] Hudson commented on MAPREDUCE-5713: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/]) MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577239) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/InputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java InputFormat and JobConf JavaDoc Fixes - Key: MAPREDUCE-5713 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5713 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 1.2.1, 2.2.0 Reporter: Ben Robie Assignee: Chen He Priority: Trivial Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5713.patch, hadoop-10210.patch Original Estimate: 0.5h Remaining Estimate: 0.5h https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/InputFormat.html Instead of record boundaries are to respected Should be record boundaries are to be respected https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/JobConf.html Instead of some parameters interact subtly rest of the framework Should be some parameters interact subtly with the rest of the framework -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5789) Average Reduce time is incorrect on Job Overview page
[ https://issues.apache.org/jira/browse/MAPREDUCE-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935112#comment-13935112 ] Hudson commented on MAPREDUCE-5789: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/]) MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page. Contributed by Rushabh S Shah (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577202) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java Average Reduce time is incorrect on Job Overview page - Key: MAPREDUCE-5789 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5789 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.10, 2.3.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Fix For: 0.23.11, 2.4.0 Attachments: patch-MapReduce-5789-v2.patch, patch-MapReduce-5789.patch The Average Reduce time displayed on the job overview page is incorrect. Previously Reduce time was calculated as difference between finishTime and shuffleFinishTime. It should be difference of finishTime and sortFinishTime -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935099#comment-13935099
]
Hudson commented on MAPREDUCE-5751:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #5328 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5328/])
MAPREDUCE-5751. MR app master fails to start in some cases if
mapreduce.job.classloader is true. Contributed by Sangjin Lee (jlowe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577554)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
MR app master fails to start in some cases if mapreduce.job.classloader is
true
---
Key: MAPREDUCE-5751
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: mapreduce-5751.patch, mapreduce-5751.patch
If mapreduce.job.classloader is set to true, and the MR client includes a
jetty jar in its libjars or job jar, the MR app master fails to start. A
typical stack trace we get is as follows:
{noformat}
java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration
cannot be cast to org.mortbay.jetty.webapp.Configuration
at
org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
{noformat}
This happens because as part of the MR app master start the jetty classes are
loaded normally through the app classloader, but WebAppContext tries to load
the specific Configuration class via the thread context classloader (which
had been set to the user job classloader).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935109#comment-13935109 ] Hudson commented on MAPREDUCE-5765: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1726 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1726/]) MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577188) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Priority: Minor Labels: documentation Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5765.patch, MAPREDUCE-5765.patch wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5688) TestStagingCleanup fails intermittently with JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5688:
--
Fix Version/s: (was: 2.5.0)
2.4.0
Hadoop Flags: Reviewed
I pulled this into branch-2.4 as well.
TestStagingCleanup fails intermittently with JDK7
-
Key: MAPREDUCE-5688
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5688
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Mit Desai
Assignee: Mit Desai
Labels: java7
Fix For: 3.0.0, 2.4.0
Attachments: MAPREDUCE-5688-v2.patch, MAPREDUCE-5688-v2.patch,
MAPREDUCE-5688.patch
Due to random ordering ordering in JDK7, the test
TestStagingCleanup#testDeletionofStagingOnKillLastTry is failing
{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.231 sec
FAILURE!
test(org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup) Time elapsed:
3882 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:349)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:159)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1399)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.testDeletionofStagingOnKillLastTry(TestStagingCleanup.java:239)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.test(TestStagingCleanup.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Falied if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935268#comment-13935268 ] Xuan Gong commented on MAPREDUCE-5795: -- added one more field:errorHappenedShutDown in AMStartedEvent Job should be marked as Falied if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: MAPREDUCE-5795.1.patch If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5795) Job should be marked as Falied if it is recovered from commit.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935311#comment-13935311 ] Xuan Gong commented on MAPREDUCE-5795: -- for -1 javac, every fields in AMStart have marked as deprecated. I added one more field: field:errorHappenedShutDown which will add two more deprecated warning Job should be marked as Falied if it is recovered from commit. -- Key: MAPREDUCE-5795 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5795 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Xuan Gong Assignee: Xuan Gong Attachments: MAPREDUCE-5795.1.patch If Resource manager is restarted when a job is in commit state, The job is not able to recovered after RM restart and it is marked as Killed. The job status should be Failed instead killed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5751) MR app master fails to start in some cases if mapreduce.job.classloader is true
[
https://issues.apache.org/jira/browse/MAPREDUCE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935324#comment-13935324
]
Sangjin Lee commented on MAPREDUCE-5751:
Thanks Jason! Much appreciated.
MR app master fails to start in some cases if mapreduce.job.classloader is
true
---
Key: MAPREDUCE-5751
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5751
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Fix For: 2.4.0
Attachments: mapreduce-5751.patch, mapreduce-5751.patch
If mapreduce.job.classloader is set to true, and the MR client includes a
jetty jar in its libjars or job jar, the MR app master fails to start. A
typical stack trace we get is as follows:
{noformat}
java.lang.ClassCastException: org.mortbay.jetty.webapp.WebInfConfiguration
cannot be cast to org.mortbay.jetty.webapp.Configuration
at
org.mortbay.jetty.webapp.WebAppContext.loadConfigurations(WebAppContext.java:890)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:462)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:676)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:208)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:151)
at
org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1040)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1307)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1303)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1259)
{noformat}
This happens because as part of the MR app master start the jetty classes are
loaded normally through the app classloader, but WebAppContext tries to load
the specific Configuration class via the thread context classloader (which
had been set to the user job classloader).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5769) Unregistration to RM should not be called if AM is crashed before registering with RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935349#comment-13935349 ] Jason Lowe commented on MAPREDUCE-5769: --- Sorry for the delay, finally getting back to this. +1 lgtm. Committing this. Unregistration to RM should not be called if AM is crashed before registering with RM - Key: MAPREDUCE-5769 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5769 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Rohith Assignee: Rohith Attachments: MAPREDUCE-5769.1.patch, MAPREDUCE-5769.2.patch Got the scenario from YARN-1752, where unregistration of application master is called before registering with RM. There should be mechanism to identify isApplicationMasaterRegistered before calling unregistration (finishApplicationMaster). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5769) Unregistration to RM should not be called if AM is crashed before registering with RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5769: -- Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Rohith! I committed this to trunk, branch-2, and branch-2.4. Unregistration to RM should not be called if AM is crashed before registering with RM - Key: MAPREDUCE-5769 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5769 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Rohith Assignee: Rohith Fix For: 2.4.0 Attachments: MAPREDUCE-5769.1.patch, MAPREDUCE-5769.2.patch Got the scenario from YARN-1752, where unregistration of application master is called before registering with RM. There should be mechanism to identify isApplicationMasaterRegistered before calling unregistration (finishApplicationMaster). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5769) Unregistration to RM should not be called if AM is crashed before registering with RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935402#comment-13935402 ] Hudson commented on MAPREDUCE-5769: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5330 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5330/]) MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed before registering with RM. Contributed by Rohith (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577647) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java Unregistration to RM should not be called if AM is crashed before registering with RM - Key: MAPREDUCE-5769 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5769 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Rohith Assignee: Rohith Fix For: 2.4.0 Attachments: MAPREDUCE-5769.1.patch, MAPREDUCE-5769.2.patch Got the scenario from YARN-1752, where unregistration of application master is called before registering with RM. There should be mechanism to identify isApplicationMasaterRegistered before calling unregistration (finishApplicationMaster). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5570: -- Attachment: patch-MapReduce-5570-v2.patch Thanks Jason for the comments. Incorporated them in the new patch. Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Rushabh S Shah Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5570: -- Status: Patch Available (was: Open) Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 2.1.1-beta, 0.23.9 Reporter: Jason Lowe Assignee: Rushabh S Shah Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5570: -- Status: Open (was: Patch Available) Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 2.1.1-beta, 0.23.9 Reporter: Jason Lowe Assignee: Rushabh S Shah Attachments: patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5797) The elapsed time for tasks in a failed job that were never started can be way off.
Rushabh S Shah created MAPREDUCE-5797: - Summary: The elapsed time for tasks in a failed job that were never started can be way off. Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) The elapsed time for tasks in a failed job that were never started can be way off.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Attachment: patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. --- Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5792) When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error
[
https://issues.apache.org/jira/browse/MAPREDUCE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935522#comment-13935522
]
Travis Thompson commented on MAPREDUCE-5792:
Let me double check {{yarn.log.server.url}}, I think you're on to something.
When mapreduce.jobhistory.intermediate-done-dir isn't writable, application
fails with generic error
Key: MAPREDUCE-5792
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5792
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Travis Thompson
Assignee: Mohammad Kamrul Islam
When trying to run an application and the permissions are wrong on
{{mapreduce.jobhistory.intermediate-done-dir}}, the MapReduce AM fails with a
non-descriptive error message:
{noformat}
Application application_1394227890066_0004 failed 2 times due to AM Container
for appattempt_1394227890066_0004_02 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
main : command provided 1
main : user is tthompso
main : requested yarn user is tthompso
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
{noformat}
When permissions are corrected on this dir, applications are able to run.
There should probably be some sort of check on this dir before launching the
AM so a more meaningful error message can be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) The elapsed time for tasks in a failed job that were never started can be way off.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Status: Patch Available (was: Open) Add a new check in javascript if the returned date is '-1'. If it is then return N/A. Minor changes to Times.java also and added a test case to confirm that. The elapsed time for tasks in a failed job that were never started can be way off. --- Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) The elapsed time for tasks in a failed job is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Summary: The elapsed time for tasks in a failed job is wrong (was: The elapsed time for tasks in a failed job that were never started can be way off. ) The elapsed time for tasks in a failed job is wrong - Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Summary: Elapsed time for failed tasks that never started is wrong (was: The elapsed time for tasks in a failed job is wrong ) Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[
https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1393#comment-1393
]
Hadoop QA commented on MAPREDUCE-5570:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634804/patch-MapReduce-5570-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4427//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4427//console
This message is automatically generated.
Map task attempt with fetch failure has incorrect attempt finish time
-
Key: MAPREDUCE-5570
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.9, 2.1.1-beta
Reporter: Jason Lowe
Assignee: Rushabh S Shah
Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch
If a map task attempt is retroactively failed due to excessive fetch failures
reported by reducers then the attempt's finish time is set to the time the
task was retroactively failed rather than when the task attempt completed.
This causes the map task attempt to appear to have run for much longer than
it actually did.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935557#comment-13935557 ] Jason Lowe commented on MAPREDUCE-5570: --- +1 lgtm, committing this. Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Rushabh S Shah Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[
https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935562#comment-13935562
]
Hadoop QA commented on MAPREDUCE-5797:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634809/patch-MapReduce-5797.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4428//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4428//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4428//console
This message is automatically generated.
Elapsed time for failed tasks that never started is wrong
Key: MAPREDUCE-5797
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, webapps
Affects Versions: 0.23.9
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Attachments: patch-MapReduce-5797.patch
The elapsed time for tasks in a failed job that were never
started can be way off. It looks like we're marking the start time as the
beginning of the epoch (i.e.: start time = -1) but the finish time is when the
task was marked as failed when the whole job failed. That causes the
calculated elapsed time of the task to be a ridiculous number of hours.
Tasks that fail without any attempts shouldn't have start/finish/elapsed
times.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5570: -- Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Rushabh! I committed this to trunk, branch-2, and branch-2.4. Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Rushabh S Shah Fix For: 2.4.0 Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5570) Map task attempt with fetch failure has incorrect attempt finish time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935619#comment-13935619 ] Hudson commented on MAPREDUCE-5570: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5331 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5331/]) MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt finish time. Contributed by Rushabh S Shah (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1577692) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java Map task attempt with fetch failure has incorrect attempt finish time - Key: MAPREDUCE-5570 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5570 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.9, 2.1.1-beta Reporter: Jason Lowe Assignee: Rushabh S Shah Fix For: 2.4.0 Attachments: patch-MapReduce-5570-v2.patch, patch-MapReduce-5570.patch If a map task attempt is retroactively failed due to excessive fetch failures reported by reducers then the attempt's finish time is set to the time the task was retroactively failed rather than when the task attempt completed. This causes the map task attempt to appear to have run for much longer than it actually did. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5648) Allow user-specified diagnostics for killed tasks and jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5648: - Target Version/s: 2.4.0 Allow user-specified diagnostics for killed tasks and jobs -- Key: MAPREDUCE-5648 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5648 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, mr-am, mrv2 Affects Versions: 2.2.0 Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-5648.v01.patch, MAPREDUCE-5648.v02.patch, MAPREDUCE-5648.v03.patch, MAPREDUCE-5648.v04.patch, Screen Shot 2013-11-23 at 11.12.15 AM.png Our users and tools want to be able to supply additional custom diagnostic messages to mapreduce ClientProtocol killTask. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[ https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5044: - Target Version/s: 2.4.0 Have AM trigger jstack on task attempts that timeout before killing them Key: MAPREDUCE-5044 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Gera Shegalov Attachments: MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch, MAPREDUCE-5044.v04.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png, Screen Shot 2013-11-12 at 1.06.04 PM.png When an AM expires a task attempt it would be nice if it triggered a jstack output via SIGQUIT before killing the task attempt. This would be invaluable for helping users debug their hung tasks, especially if they do not have shell access to the nodes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Status: Open (was: Patch Available) Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Attachment: patch-MapReduce-5797-v2.patch Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated MAPREDUCE-5797: -- Status: Patch Available (was: Open) Added Apache License Agreement TestTimes.java Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-4052: --- Status: Open (was: Patch Available) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 2.2.0, 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-4052: --- Status: Patch Available (was: Open) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 2.2.0, 0.23.1 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-4052: --- Attachment: MAPREDUCE-4052.8.patch New patch did a couple of renames, and added public/unstable for each newly added config. Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster. --- Key: MAPREDUCE-4052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.1, 2.2.0 Environment: client on the Windows, the the cluster on the suse Reporter: xieguiming Assignee: Jian He Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch, MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch, MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch, MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch when I use the eclipse on the windows to submit the job. and the applicationmaster throw the exception: Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/v2/app/MRAppMaster Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapreduce.v2.app.MRAppMaster at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. The reasion is : class Apps addToEnvironment function, use the private static final String SYSTEM_PATH_SEPARATOR = System.getProperty(path.separator); and will result the MRApplicationMaster classpath use the ; separator. I suggest that nodemanger do the replace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[
https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935752#comment-13935752
]
Hadoop QA commented on MAPREDUCE-5797:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634852/patch-MapReduce-5797-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4429//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4429//console
This message is automatically generated.
Elapsed time for failed tasks that never started is wrong
Key: MAPREDUCE-5797
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, webapps
Affects Versions: 0.23.9
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch
The elapsed time for tasks in a failed job that were never
started can be way off. It looks like we're marking the start time as the
beginning of the epoch (i.e.: start time = -1) but the finish time is when the
task was marked as failed when the whole job failed. That causes the
calculated elapsed time of the task to be a ridiculous number of hours.
Tasks that fail without any attempts shouldn't have start/finish/elapsed
times.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5792) When mapreduce.jobhistory.intermediate-done-dir isn't writable, application fails with generic error
[
https://issues.apache.org/jira/browse/MAPREDUCE-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935786#comment-13935786
]
Travis Thompson commented on MAPREDUCE-5792:
So I didn't have {{yarn.log.server.url}} set, and after setting that log
redirection off the NM works, yay. It would still be nice to have a more
meaningful message make it back to the RM though, because right now you have to
go to the GUI to find the error message, the console just shows the generic
shell exception message.
When mapreduce.jobhistory.intermediate-done-dir isn't writable, application
fails with generic error
Key: MAPREDUCE-5792
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5792
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 2.3.0
Reporter: Travis Thompson
Assignee: Mohammad Kamrul Islam
When trying to run an application and the permissions are wrong on
{{mapreduce.jobhistory.intermediate-done-dir}}, the MapReduce AM fails with a
non-descriptive error message:
{noformat}
Application application_1394227890066_0004 failed 2 times due to AM Container
for appattempt_1394227890066_0004_02 exited with exitCode: 1 due to:
Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
main : command provided 1
main : user is tthompso
main : requested yarn user is tthompso
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
{noformat}
When permissions are corrected on this dir, applications are able to run.
There should probably be some sort of check on this dir before launching the
AM so a more meaningful error message can be thrown.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-4052) Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935898#comment-13935898
]
Hadoop QA commented on MAPREDUCE-4052:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634855/MAPREDUCE-4052.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4430//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4430//console
This message is automatically generated.
Windows eclipse cannot submit job from Windows client to Linux/Unix Hadoop
cluster.
---
Key: MAPREDUCE-4052
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4052
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Affects Versions: 0.23.1, 2.2.0
Environment: client on the Windows, the the cluster on the suse
Reporter: xieguiming
Assignee: Jian He
Attachments: MAPREDUCE-4052-0.patch, MAPREDUCE-4052.1.patch,
MAPREDUCE-4052.2.patch, MAPREDUCE-4052.3.patch, MAPREDUCE-4052.4.patch,
MAPREDUCE-4052.5.patch, MAPREDUCE-4052.6.patch, MAPREDUCE-4052.7.patch,
MAPREDUCE-4052.8.patch, MAPREDUCE-4052.patch
when I use the eclipse on the windows to submit the job. and the
applicationmaster throw the exception:
Exception in thread main java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
The reasion is :
class Apps addToEnvironment function, use the
private static final String SYSTEM_PATH_SEPARATOR =
System.getProperty(path.separator);
and will result the MRApplicationMaster classpath use the ; separator.
I suggest that nodemanger do the replace.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935965#comment-13935965 ] Jonathan Eagles commented on MAPREDUCE-5797: [~shahrs87], in general I like it. Couple of minor nits. *Hadoop uses the java oracle coding standards roughly, with the exceptions that tabs are expanded and we use 2 spaces to indent. I highly recommend the indentation and white space sections for gotchas. For example keywords followed by a parenthesis should have a space in between. http://wiki.apache.org/hadoop/HowToContribute *Of course, all things have exceptions like maintaining the edited files current formatting, however wrong. *Separate out the test cases and name them appropriate to describe the feature under test. Example, testNegativeStartTime, testNegativeEndTime. This way when it fails, the feature that is broken is immediately evident. Sorry for the overly long comment. Welcome to the Apache Hadoop community. We are happy to have you onboard. Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
