[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated MAPREDUCE-6377: Attachment: Sorting Issue.png JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Priority: Minor Attachments: Sorting Issue.png Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
Bibin A Chundatt created MAPREDUCE-6377: --- Summary: JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Priority: Minor Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565608#comment-14565608 ] Craig Welch commented on MAPREDUCE-6302: [~kasha] this looks good to me overall, it would be great to have this to avoid deadlocks as such, at least as a last resort. I agree with [~wangda] that it would be good to be able to disable this with a value like -1, but I think you're setting the default to not be disabled (looks like you have 5 minutes) is the way to go - by default I think it is best to have this active. I actually think seconds is the right granularity here, the time factor of relevance for this sort of activity is really not in the MS range. I also think the naming you have for the configuration option is fine, esp. in contrast to the other (existing) delay it makes sense. Do you think you could add a test or two? deadlock in a job between map and reduce cores allocation -- Key: MAPREDUCE-6302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: mai shurong Assignee: Karthik Kambatla Priority: Critical Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz, log.txt, mr-6302-prelim.patch, queue_with_max163cores.png, queue_with_max263cores.png, queue_with_max333cores.png I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue. I think there is the similar issue for memory of a queue . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565665#comment-14565665 ] Karthik Kambatla commented on MAPREDUCE-6302: - I haven't had a chance to follow up on this. If anyone else wants to follow up, please feel free. I ll try to pick this up late next week, if no one else gets to it. deadlock in a job between map and reduce cores allocation -- Key: MAPREDUCE-6302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: mai shurong Assignee: Karthik Kambatla Priority: Critical Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz, log.txt, mr-6302-prelim.patch, queue_with_max163cores.png, queue_with_max263cores.png, queue_with_max333cores.png I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue. I think there is the similar issue for memory of a queue . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu reassigned MAPREDUCE-6377: Assignee: zhihai xu JHS sorting on state column not working in webUi Key: MAPREDUCE-6377 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.7.0 Environment: 2 NM, JHS Reporter: Bibin A Chundatt Assignee: zhihai xu Priority: Minor Attachments: Sorting Issue.png Steps to reproduce 1. Install and setup HA cluster with JHS 2.Create state in in JHS where few jobs are killed and Success Check sorting State in JHS WebUI Actual = Sorting on state column not working in JHS Expected == Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[
https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565818#comment-14565818
]
zhihai xu commented on MAPREDUCE-6377:
--
Hi [~bibinchundatt], thanks for reporting this issue.
I can work on this issue. It looks like the change in MAPREDUCE-5052 caused
this issue.
In
[MAPREDUCE-5052|https://github.com/apache/hadoop/commit/ee5351bf22e113d39db3839432bc7d6c743cc736],
Submit Time is added at the first column and all columns should be shifted
by one. But {{HsView#jobsTableInit}} is not changed to match the new layout.
{code}
private String jobsTableInit() {
return tableInit().
append(, 'aaData': jobsTableData).
append(, bDeferRender: true).
append(, bProcessing: true).
// Sort by id upon page load
append(, aaSorting: [[2, 'desc']]).
append(, aoColumnDefs:[).
// Maps Total, Maps Completed, Reduces Total and Reduces Completed
append({'sType':'numeric', 'bSearchable': false, 'aTargets': [ 7, 8,
9, 10 ] }).
append(]}).
toString();
}
{code}
Based on the above code, you can see column State which is 7 now is
mistakenly considered as column Maps Total.
JHS sorting on state column not working in webUi
Key: MAPREDUCE-6377
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.7.0
Environment: 2 NM, JHS
Reporter: Bibin A Chundatt
Assignee: zhihai xu
Priority: Minor
Attachments: Sorting Issue.png
Steps to reproduce
1. Install and setup HA cluster with JHS
2.Create state in in JHS where few jobs are killed and Success
Check sorting State in JHS WebUI
Actual
=
Sorting on state column not working in JHS
Expected
==
Sorting on state column should be working
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6378) dependency loop in hadoop-streaming
Allen Wittenauer created MAPREDUCE-6378: --- Summary: dependency loop in hadoop-streaming Key: MAPREDUCE-6378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Running mvn install in the hadoop-tools/hadoop-streaming directory results in a dependency loop. See comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6378) convergence error in hadoop-streaming during mvn install
[ https://issues.apache.org/jira/browse/MAPREDUCE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6378: Description: Running mvn install in the hadoop-tools/hadoop-streaming directory results in a convergence error. See comments. (was: Running mvn install in the hadoop-tools/hadoop-streaming directory results in a dependency loop. See comments.) convergence error in hadoop-streaming during mvn install Key: MAPREDUCE-6378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Running mvn install in the hadoop-tools/hadoop-streaming directory results in a convergence error. See comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6378) dependency loop in hadoop-streaming
[
https://issues.apache.org/jira/browse/MAPREDUCE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565230#comment-14565230
]
Allen Wittenauer commented on MAPREDUCE-6378:
-
I'm seeing:
{code}
[WARNING]
Dependency convergence error for org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
paths to dependency are:
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-hs:3.0.0-20150528.180838-4920
+-org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdfs:3.0.0-20150529.025134-5802
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdfs:3.0.0-20150529.025134-5802
[WARNING]
Dependency convergence error for
org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-20150528.180833-5003 paths
to dependency are:
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-20150528.180833-5003
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-hs:3.0.0-20150528.180838-4920
+-org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-SNAPSHOT
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence
failed with message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
paths to dependency are:
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-hs:3.0.0-20150528.180838-4920
+-org.apache.hadoop:hadoop-hdfs:3.0.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdfs:3.0.0-20150529.025134-5802
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-hdfs:3.0.0-20150529.025134-5802
,
Dependency convergence error for
org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-20150528.180833-5003 paths
to dependency are:
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-20150528.180833-5003
and
+-org.apache.hadoop:hadoop-streaming:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-mapreduce-client-hs:3.0.0-20150528.180838-4920
+-org.apache.hadoop:hadoop-mapreduce-client-app:3.0.0-SNAPSHOT
{code}
dependency loop in hadoop-streaming
---
Key: MAPREDUCE-6378
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Allen Wittenauer
Running mvn install in the hadoop-tools/hadoop-streaming directory results in
a dependency loop. See comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6378) convergence error in hadoop-streaming during mvn install
[ https://issues.apache.org/jira/browse/MAPREDUCE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6378: Summary: convergence error in hadoop-streaming during mvn install (was: dependency loop in hadoop-streaming) convergence error in hadoop-streaming during mvn install Key: MAPREDUCE-6378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Running mvn install in the hadoop-tools/hadoop-streaming directory results in a dependency loop. See comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6378) convergence error in hadoop-streaming during mvn install
[ https://issues.apache.org/jira/browse/MAPREDUCE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565256#comment-14565256 ] Chris Nauroth commented on MAPREDUCE-6378: -- Hi [~aw]. This looks like unfortunate bad timing of different snapshot versions of dependencies getting pushed into the remote repo in between the builds of the individual sub-modules. Running Maven in offline mode would resolve this by using only the local repo, containing local builds of 3.0.0-SNAPSHOT, which wouldn't be subject to changes happening in a remote repo. I just tried a build of current trunk, and it worked. I did a full mvn install from the root of the source tree, and then I did a separate mvn install of hadoop-streaming. convergence error in hadoop-streaming during mvn install Key: MAPREDUCE-6378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Running mvn install in the hadoop-tools/hadoop-streaming directory results in a convergence error. See comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6378) convergence error in hadoop-streaming during mvn install
[ https://issues.apache.org/jira/browse/MAPREDUCE-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14565273#comment-14565273 ] Allen Wittenauer commented on MAPREDUCE-6378: - bq. I did a full mvn install from the root of the source tree I think that's the difference. I'm not doing that. convergence error in hadoop-streaming during mvn install Key: MAPREDUCE-6378 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6378 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Allen Wittenauer Running mvn install in the hadoop-tools/hadoop-streaming directory results in a convergence error. See comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
