[jira] [Updated] (MAPREDUCE-5601) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5601: -- Resolution: Fixed Fix Version/s: 2.3.0 Release Note: I just committed this Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails -- Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5601.patch, MAPREDUCE-5601.patch, MAPREDUCE-5601.patch When a reducer initiates a fetch request, it does not know whether it will be able to fit the fetched data in memory. The first part of the response tells how much data will be coming. If space is not currently available, the reduce will abandon its request and try again later. When this occurs, the ShuffleHandler still fadvises the file region as DONTNEED. Meaning that the next time it's asked for, it will definitely be read from disk, even if it happened to be in the page cache before the request. I noticed this when trying to figure out why my job was doing so much more disk IO in MR2 than in MR1. When I turned the fadvise stuff off, I found that disk reads went to nearly 0 on machines that had enough memory to fit map outputs into the page cache. I then straced the NodeManager and noticed that there were over four times as many fadvise DONTNEED calls as map-reduce pairs. Further logging showed the same map outputs being fetched about this many times. This is a regression from MR1, which only did the fadvise DONTNEED after all the bytes were transferred. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5601) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811121#comment-13811121 ] Hudson commented on MAPREDUCE-5601: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4685 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4685/]) MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when fetch fails (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1537855) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java ShuffleHandler fadvises file regions as DONTNEED even when fetch fails -- Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5601.patch, MAPREDUCE-5601.patch, MAPREDUCE-5601.patch When a reducer initiates a fetch request, it does not know whether it will be able to fit the fetched data in memory. The first part of the response tells how much data will be coming. If space is not currently available, the reduce will abandon its request and try again later. When this occurs, the ShuffleHandler still fadvises the file region as DONTNEED. Meaning that the next time it's asked for, it will definitely be read from disk, even if it happened to be in the page cache before the request. I noticed this when trying to figure out why my job was doing so much more disk IO in MR2 than in MR1. When I turned the fadvise stuff off, I found that disk reads went to nearly 0 on machines that had enough memory to fit map outputs into the page cache. I then straced the NodeManager and noticed that there were over four times as many fadvise DONTNEED calls as map-reduce pairs. Further logging showed the same map outputs being fetched about this many times. This is a regression from MR1, which only did the fadvise DONTNEED after all the bytes were transferred. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5601) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5601: -- Release Note: (was: I just committed this) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails -- Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5601.patch, MAPREDUCE-5601.patch, MAPREDUCE-5601.patch When a reducer initiates a fetch request, it does not know whether it will be able to fit the fetched data in memory. The first part of the response tells how much data will be coming. If space is not currently available, the reduce will abandon its request and try again later. When this occurs, the ShuffleHandler still fadvises the file region as DONTNEED. Meaning that the next time it's asked for, it will definitely be read from disk, even if it happened to be in the page cache before the request. I noticed this when trying to figure out why my job was doing so much more disk IO in MR2 than in MR1. When I turned the fadvise stuff off, I found that disk reads went to nearly 0 on machines that had enough memory to fit map outputs into the page cache. I then straced the NodeManager and noticed that there were over four times as many fadvise DONTNEED calls as map-reduce pairs. Further logging showed the same map outputs being fetched about this many times. This is a regression from MR1, which only did the fadvise DONTNEED after all the bytes were transferred. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5601) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811230#comment-13811230 ] Hudson commented on MAPREDUCE-5601: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1596 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1596/]) MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when fetch fails (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1537855) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java ShuffleHandler fadvises file regions as DONTNEED even when fetch fails -- Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5601.patch, MAPREDUCE-5601.patch, MAPREDUCE-5601.patch When a reducer initiates a fetch request, it does not know whether it will be able to fit the fetched data in memory. The first part of the response tells how much data will be coming. If space is not currently available, the reduce will abandon its request and try again later. When this occurs, the ShuffleHandler still fadvises the file region as DONTNEED. Meaning that the next time it's asked for, it will definitely be read from disk, even if it happened to be in the page cache before the request. I noticed this when trying to figure out why my job was doing so much more disk IO in MR2 than in MR1. When I turned the fadvise stuff off, I found that disk reads went to nearly 0 on machines that had enough memory to fit map outputs into the page cache. I then straced the NodeManager and noticed that there were over four times as many fadvise DONTNEED calls as map-reduce pairs. Further logging showed the same map outputs being fetched about this many times. This is a regression from MR1, which only did the fadvise DONTNEED after all the bytes were transferred. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5186) mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-5186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811244#comment-13811244 ] Jason Lowe commented on MAPREDUCE-5186: --- TestUberAM timeout is unrelated, see MAPREDUCE-5481. mapreduce.job.max.split.locations causes some splits created by CombineFileInputFormat to fail -- Key: MAPREDUCE-5186 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5186 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 2.0.4-alpha, 2.2.0 Reporter: Sangjin Lee Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE-5186v1.patch, MAPREDUCE-5186v2.patch, MAPREDUCE-5186v3.patch, MAPREDUCE-5186v3.patch CombineFileInputFormat can easily create splits that can come from many different locations (during the last pass of creating global splits). However, we observe that this often runs afoul of the mapreduce.job.max.split.locations check that's done by JobSplitWriter. The default value for mapreduce.job.max.split.locations is 10, and with any decent size cluster, CombineFileInputFormat creates splits that are well above this limit. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5601) ShuffleHandler fadvises file regions as DONTNEED even when fetch fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811250#comment-13811250 ] Hudson commented on MAPREDUCE-5601: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1570 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1570/]) MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when fetch fails (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1537855) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java ShuffleHandler fadvises file regions as DONTNEED even when fetch fails -- Key: MAPREDUCE-5601 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5601 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.2.0 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.3.0 Attachments: MAPREDUCE-5601.patch, MAPREDUCE-5601.patch, MAPREDUCE-5601.patch When a reducer initiates a fetch request, it does not know whether it will be able to fit the fetched data in memory. The first part of the response tells how much data will be coming. If space is not currently available, the reduce will abandon its request and try again later. When this occurs, the ShuffleHandler still fadvises the file region as DONTNEED. Meaning that the next time it's asked for, it will definitely be read from disk, even if it happened to be in the page cache before the request. I noticed this when trying to figure out why my job was doing so much more disk IO in MR2 than in MR1. When I turned the fadvise stuff off, I found that disk reads went to nearly 0 on machines that had enough memory to fit map outputs into the page cache. I then straced the NodeManager and noticed that there were over four times as many fadvise DONTNEED calls as map-reduce pairs. Further logging showed the same map outputs being fetched about this many times. This is a regression from MR1, which only did the fadvise DONTNEED after all the bytes were transferred. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5481) Uber job reducers hang waiting to shuffle map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5481: -- Summary: Uber job reducers hang waiting to shuffle map outputs (was: TestUberAM timeout) This is not localized to just TestUberAM, as all uber jobs with reducers will currently hang like this. I believe the test is This appears to have been broken when MAPREDUCE-434 was committed, as that changed the way reducers locate map outputs when not running via normal containers. Updating the summary to better reflect the issue. Uber job reducers hang waiting to shuffle map outputs - Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0 Reporter: Jason Lowe Assignee: Xuan Gong Priority: Blocker Attachments: syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-434) LocalJobRunner limited to single reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811324#comment-13811324 ] Jason Lowe commented on MAPREDUCE-434: -- It looks like Arun's earlier comment about porting this to the uber AM case was missed, as this change has broken any ubertask job that has reducers. See MAPREDUCE-5481. [~sandyr] could you take a look? LocalJobRunner limited to single reducer Key: MAPREDUCE-434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-434 Project: Hadoop Map/Reduce Issue Type: Improvement Environment: local job tracker Reporter: Yoram Arnon Assignee: Aaron Kimball Priority: Minor Fix For: 2.3.0 Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.3.patch, MAPREDUCE-434.4.patch, MAPREDUCE-434.5.patch, MAPREDUCE-434.6.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1. This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (MAPREDUCE-5481) Uber job reducers hang waiting to shuffle map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811316#comment-13811316 ] Jason Lowe edited comment on MAPREDUCE-5481 at 11/1/13 3:13 PM: This is not localized to just TestUberAM, as all uber jobs with reducers will currently hang like this. This appears to have been broken when MAPREDUCE-434 was committed, since it changed the way reducers locate map outputs when not running via normal containers. Updating the summary to better reflect the issue. was (Author: jlowe): This is not localized to just TestUberAM, as all uber jobs with reducers will currently hang like this. I believe the test is This appears to have been broken when MAPREDUCE-434 was committed, as that changed the way reducers locate map outputs when not running via normal containers. Updating the summary to better reflect the issue. Uber job reducers hang waiting to shuffle map outputs - Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0 Reporter: Jason Lowe Assignee: Xuan Gong Priority: Blocker Attachments: syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Uber job reducers hang waiting to shuffle map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811406#comment-13811406 ] Xuan Gong commented on MAPREDUCE-5481: -- Yes, It is not only broke the TestUberAM. Under current trunk version, we can not even run the applications under the Uber mode. Uber job reducers hang waiting to shuffle map outputs - Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0 Reporter: Jason Lowe Assignee: Xuan Gong Priority: Blocker Attachments: syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-434) LocalJobRunner limited to single reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811417#comment-13811417 ] Sandy Ryza commented on MAPREDUCE-434: -- Will take a look today. LocalJobRunner limited to single reducer Key: MAPREDUCE-434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-434 Project: Hadoop Map/Reduce Issue Type: Improvement Environment: local job tracker Reporter: Yoram Arnon Assignee: Aaron Kimball Priority: Minor Fix For: 2.3.0 Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.3.patch, MAPREDUCE-434.4.patch, MAPREDUCE-434.5.patch, MAPREDUCE-434.6.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch, MAPREDUCE-434.patch when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1. This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5392) mapred job -history all command throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinichi Yamashita updated MAPREDUCE-5392:
--
Attachment: MAPREDUCE-5392.patch
mapred job -history all command throws IndexOutOfBoundsException
--
Key: MAPREDUCE-5392
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5392
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0, 2.0.5-alpha, 2.2.0
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch, MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch, MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch
When I use an all option by mapred job -history comamnd, the following
exceptions are displayed and do not work.
{code}
Exception in thread main java.lang.StringIndexOutOfBoundsException: String
index out of range: -3
at java.lang.String.substring(String.java:1875)
at
org.apache.hadoop.mapreduce.util.HostUtil.convertTrackerNameToHostName(HostUtil.java:49)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.getTaskLogsUrl(HistoryViewer.java:459)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.printAllTaskAttempts(HistoryViewer.java:235)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.print(HistoryViewer.java:117)
at org.apache.hadoop.mapreduce.tools.CLI.viewHistory(CLI.java:472)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:313)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1233)
{code}
This is because a node name recorded in History file is not given tracker_.
Therefore it makes modifications to be able to read History file even if a
node name is not given by tracker_.
In addition, it fixes the URL of displayed task log.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5392) mapred job -history all command throws IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811521#comment-13811521
]
Hadoop QA commented on MAPREDUCE-5392:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12611639/MAPREDUCE-5392.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4165//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4165//console
This message is automatically generated.
mapred job -history all command throws IndexOutOfBoundsException
--
Key: MAPREDUCE-5392
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5392
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0, 2.0.5-alpha, 2.2.0
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita
Priority: Minor
Fix For: 3.0.0
Attachments: MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch, MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch, MAPREDUCE-5392.patch, MAPREDUCE-5392.patch,
MAPREDUCE-5392.patch
When I use an all option by mapred job -history comamnd, the following
exceptions are displayed and do not work.
{code}
Exception in thread main java.lang.StringIndexOutOfBoundsException: String
index out of range: -3
at java.lang.String.substring(String.java:1875)
at
org.apache.hadoop.mapreduce.util.HostUtil.convertTrackerNameToHostName(HostUtil.java:49)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.getTaskLogsUrl(HistoryViewer.java:459)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.printAllTaskAttempts(HistoryViewer.java:235)
at
org.apache.hadoop.mapreduce.jobhistory.HistoryViewer.print(HistoryViewer.java:117)
at org.apache.hadoop.mapreduce.tools.CLI.viewHistory(CLI.java:472)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:313)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1233)
{code}
This is because a node name recorded in History file is not given tracker_.
Therefore it makes modifications to be able to read History file even if a
node name is not given by tracker_.
In addition, it fixes the URL of displayed task log.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5604) TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max path length
[
https://issues.apache.org/jira/browse/MAPREDUCE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-5604:
--
Component/s: (was: client)
Hadoop Flags: Reviewed
+1 patch looks good.
Since this only changes a test, the build failure is obviously unrelated.
TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max
path length
---
Key: MAPREDUCE-5604
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5604
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: MAPREDUCE-5604.1.patch
The test uses the full class name as a component of the
{{yarn.nodemanager.local-dirs}} setting for a {{MiniMRYarnCluster}}. This
causes container launch to fail when trying to access files at a path longer
than the maximum of 260 characters.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5604) TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max path length
[
https://issues.apache.org/jira/browse/MAPREDUCE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811574#comment-13811574
]
Hudson commented on MAPREDUCE-5604:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #4686 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4686/])
MAPREDUCE-5604. TestMRAMWithNonNormalizedCapabilities fails on Windows due to
exceeding max path length. Contributed by Chris Nauroth. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1538035)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRAMWithNonNormalizedCapabilities.java
TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max
path length
---
Key: MAPREDUCE-5604
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5604
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: MAPREDUCE-5604.1.patch
The test uses the full class name as a component of the
{{yarn.nodemanager.local-dirs}} setting for a {{MiniMRYarnCluster}}. This
causes container launch to fail when trying to access files at a path longer
than the maximum of 260 characters.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He reassigned MAPREDUCE-5052: -- Assignee: Chen He Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5604) TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max path length
[
https://issues.apache.org/jira/browse/MAPREDUCE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-5604:
-
Resolution: Fixed
Fix Version/s: 2.2.1
3.0.0
Status: Resolved (was: Patch Available)
I've committed this to trunk, branch-2, and branch-2.2. Thanks to Chuan and
Nicholas for the code reviews.
TestMRAMWithNonNormalizedCapabilities fails on Windows due to exceeding max
path length
---
Key: MAPREDUCE-5604
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5604
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Fix For: 3.0.0, 2.2.1
Attachments: MAPREDUCE-5604.1.patch
The test uses the full class name as a component of the
{{yarn.nodemanager.local-dirs}} setting for a {{MiniMRYarnCluster}}. This
causes container launch to fail when trying to access files at a path longer
than the maximum of 260 characters.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Uber job reducers hang waiting to shuffle map outputs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13811748#comment-13811748 ] Sandy Ryza commented on MAPREDUCE-5481: --- It looks like, after MAPREDUCE-434, allowing Uber jobs to use multiple reducers will be pretty straightforward. Will post a patch soon that does so. Uber job reducers hang waiting to shuffle map outputs - Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0 Reporter: Jason Lowe Assignee: Xuan Gong Priority: Blocker Attachments: syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console -- This message was sent by Atlassian JIRA (v6.1#6144)
