[jira] [Created] (MAPREDUCE-6816) Progress bars in Web UI always at 100%
Shen Yinjie created MAPREDUCE-6816:
--
Summary: Progress bars in Web UI always at 100%
Key: MAPREDUCE-6816
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6816
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: webapps
Affects Versions: 3.0.0-alpha2
Reporter: Shen Yinjie
Assignee: Shen Yinjie
YARN web UI always shows progress bars at 100% (see screenshot, progress of the
reduce step is roughly at 40.00%). I opened the HTML source code to check (also
see screenshot), and it seems the problem is that it uses "%%" ,which cannot be
recognized. This is due to the output of{code}StringUtils.format(){code}
contains '%', while in *Block.java use {code}join{code}add an extra '%'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713679#comment-15713679 ] Hudson commented on MAPREDUCE-6787: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10927 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10927/]) MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job (rkanter: rev c87b3a448a00df97149a4e93a8c39d9ad0268bdb) * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebApp.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/Router.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppController.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/ConfBlock.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApp.java > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6787.004.patch, MAPREDUCE-6787.004.patch, > job_1478210774848_0001.xml, mapreduce6787.001.patch, mapreduce6787.002.patch, > mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6787: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Thanks [~haibochen]. Committed to trunk and branch-2! > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: MAPREDUCE-6787.004.patch, MAPREDUCE-6787.004.patch, > job_1478210774848_0001.xml, mapreduce6787.001.patch, mapreduce6787.002.patch, > mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6787) Allow job_conf.xml to be downloadable on the job overview page in JHS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713653#comment-15713653 ] Robert Kanter commented on MAPREDUCE-6787: -- +1 > Allow job_conf.xml to be downloadable on the job overview page in JHS > - > > Key: MAPREDUCE-6787 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6787 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: MAPREDUCE-6787.004.patch, MAPREDUCE-6787.004.patch, > job_1478210774848_0001.xml, mapreduce6787.001.patch, mapreduce6787.002.patch, > mapreduce6787.002.patch, mapreduce6787.003.patch > > > The job overview page in JHS provides the path to the job.xml file, but it is > not a link that users can click on to download the job xml file directly from > JHS. We could provide a download link in JHS for better usability. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713641#comment-15713641
]
Hadoop QA commented on MAPREDUCE-6815:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 20s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12841387/MAPREDUCE-6815.01.patch
|
| JIRA Issue | MAPREDUCE-6815 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b54a70d9fe3b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2d77dc7 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6824/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6824/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix flaky TestKill.testKillTask()
> -
>
> Key: MAPREDUCE-6815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: MAPREDUCE-6815.01.patch
>
>
> Error Message
> Job state is not correct (timedout) expected: but was:
>
[jira] [Updated] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6815: -- Attachment: MAPREDUCE-6815.01.patch > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: MAPREDUCE-6815.01.patch > > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6815: -- Status: Patch Available (was: Open) > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > Attachments: MAPREDUCE-6815.01.patch > > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6442) Stack trace is missing when error occurs in client protocol provider's constructor
[ https://issues.apache.org/jira/browse/MAPREDUCE-6442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6442: -- Fix Version/s: 2.8.0 > Stack trace is missing when error occurs in client protocol provider's > constructor > -- > > Key: MAPREDUCE-6442 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6442 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6442.2.patch, MAPREDUCE-6442.patch > > > when provider creation fail dump the stack trace rather than just print out > the message -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713547#comment-15713547 ] Haibo Chen edited comment on MAPREDUCE-6815 at 12/2/16 12:39 AM: - bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still in SetUp state, sending task_kill event will eventually cause the job to fail as show above. We could wait on the internal state instead of external state. was (Author: haibochen): bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still in SetUp state, sending task_kill event will eventually cause the job to fail as show above > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713561#comment-15713561 ] Haibo Chen commented on MAPREDUCE-6815: --- bq. 2016-11-23 10:08:07,726 ERROR [AsyncDispatcher event handler] impl.TaskImpl (TaskImpl.java:handle(656)) - Can't handle this event at current state for task_0__m_00 org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: T_SCHEDULE at KILLED at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:654) at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:101) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:1475) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:1469) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) On the other hand, A T_SCHEDULE event on a killed task is invalid, and will cause the job to fail as well. We could add a new transition from killed to killed state upon a T_SCHEDULE event (basically ignoring it ) > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713547#comment-15713547 ] Haibo Chen edited comment on MAPREDUCE-6815 at 12/2/16 12:35 AM: - bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still in SetUp state, sending task_kill event will eventually cause the job to fail as show above was (Author: haibochen): bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still SetUp state, sending task_kill event will eventually cause the job to fail as show above > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6439) AM may fail instead of retrying if RM shuts down during the allocate call
[
https://issues.apache.org/jira/browse/MAPREDUCE-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6439:
--
Fix Version/s: 2.8.0
> AM may fail instead of retrying if RM shuts down during the allocate call
> -
>
> Key: MAPREDUCE-6439
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6439
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Anubhav Dhoot
>Assignee: Anubhav Dhoot
>Priority: Critical
> Fix For: 2.8.0, 2.7.2, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6439.001.patch, MAPREDUCE-6439.002.patch
>
>
> We are seeing cases where MR AM gets a YarnRuntimeException thats thrown in
> RM and gets sent back to AM causing it to think that it has exhausted the
> number of retries. Copying the error which causes the heartbeat thread to
> quit.
> {noformat}
> 2015-07-25 20:07:27,346 ERROR [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Error
> communicating with RM: java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:245)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:469)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:240)
> ... 11 more
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.InterruptedException
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:245)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:469)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> Caused by: java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
> at
> java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:240)
> ... 11 more
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
[jira] [Comment Edited] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713547#comment-15713547 ] Haibo Chen edited comment on MAPREDUCE-6815 at 12/2/16 12:34 AM: - bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still SetUp state, sending task_kill event will eventually cause the job to fail as show above was (Author: haibochen): bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still SetUp state, sending task_kill event will eventually cause the job to fail as show above > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713547#comment-15713547 ] Haibo Chen commented on MAPREDUCE-6815: --- bq. 2016-11-23 10:08:07,725 ERROR [AsyncDispatcher event handler] impl.JobImpl (JobImpl.java:handle(1004)) - Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: JOB_TASK_COMPLETED at SETUP at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:1002) at org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:140) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1465) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1461) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:187) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:113) at java.lang.Thread.run(Thread.java:745) In TestKill.testKillTask(), we do app.waitForState(job, JobState.RUNNING) to wait for the job to be running. But when the job is in running state externally, it can be in either SetUp or Running internally. If the Job is still SetUp state, sending task_kill event will eventually cause the job to fail as show above > Fix flaky TestKill.testKillTask() > - > > Key: MAPREDUCE-6815 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen > > Error Message > Job state is not correct (timedout) expected: but was: > Stacktrace > java.lang.AssertionError: Job state is not correct (timedout) > expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) > at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713539#comment-15713539
]
Haibo Chen commented on MAPREDUCE-6815:
---
STD out
{code}
2016-11-23 10:08:07,209 INFO [main] app.MRAppMaster
(MRAppMaster.java:(269)) - Created MRAppMaster for application
appattempt_0__01
2016-11-23 10:08:07,209 INFO [main] app.MRApp (MRApp.java:(239)) -
PathUsed:
/data/jenkins/workspace/cdh6.x-Hadoop-MR/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/testKill
2016-11-23 10:08:07,254 INFO [main] app.MRAppMaster
(MRAppMaster.java:call(539)) - OutputCommitter set in config null
2016-11-23 10:08:07,254 INFO [main] app.MRAppMaster
(MRAppMaster.java:call(562)) - OutputCommitter is
org.apache.hadoop.mapred.FileOutputCommitter
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.v2.app.MRApp$4
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2016-11-23 10:08:07,256 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2016-11-23 10:08:07,257 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRApp$MRAppContainerAllocator
2016-11-23 10:08:07,257 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRApp$MockContainerLauncher
2016-11-23 10:08:07,259 INFO [main] event.AsyncDispatcher
(AsyncDispatcher.java:register(212)) - Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRApp$1
2016-11-23 10:08:07,260 WARN [main] impl.MetricsConfig
(MetricsConfig.java:loadFirst(127)) - Cannot locate configuration: tried
hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
2016-11-23 10:08:07,261 INFO [main] impl.MetricsSystemImpl
(MetricsSystemImpl.java:startTimer(375)) - Scheduled snapshot period at 10
second(s).
2016-11-23 10:08:07,261 INFO [main] impl.MetricsSystemImpl
(MetricsSystemImpl.java:start(192)) - MRAppMaster metrics system started
2016-11-23 10:08:07,262 INFO [main] impl.JobImpl (JobImpl.java:setup(1531)) -
Adding job token for job_0_ to jobTokenSecretManager
2016-11-23 10:08:07,263 WARN [main] impl.JobImpl (JobImpl.java:setup(1537)) -
Shuffle secret key missing from job credentials. Using job token secret as
shuffle secret.
2016-11-23 10:08:07,264 INFO [main] impl.JobImpl
(JobImpl.java:makeUberDecision(1302)) - Not uberizing job_0_ because: not
enabled;
2016-11-23 10:08:07,264 INFO [main] impl.JobImpl
(JobImpl.java:createMapTasks(1559)) - Input size for job job_0_ = 0. Number
of splits = 2
2016-11-23 10:08:07,264 INFO [main] impl.JobImpl
(JobImpl.java:createReduceTasks(1576)) - Number of reduces for job job_0_ = 0
2016-11-23 10:08:07,265 INFO [main] impl.JobImpl (JobImpl.java:handle(1012)) -
job_0_Job Transitioned from NEW to INITED
2016-11-23 10:08:07,265 INFO [main] app.MRAppMaster
(MRAppMaster.java:serviceStart(1256)) - MRAppMaster launching normal,
non-uberized, multi-container job job_0_.
2016-11-23 10:08:07,265 INFO [main] ipc.CallQueueManager
(CallQueueManager.java:(75)) - Using callQueue: class
java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class
org.apache.hadoop.ipc.DefaultRpcScheduler
2016-11-23 10:08:07,266 INFO [Socket Reader #1 for port 37121] ipc.Server
[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6377: -- Fix Version/s: 2.8.0 > JHS sorting on state column not working in webUi > > > Key: MAPREDUCE-6377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.0 > Environment: 2 NM, JHS >Reporter: Bibin A Chundatt >Assignee: zhihai xu >Priority: Minor > Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, > state_sorted1.pdf, state_sorted2.pdf > > > Steps to reproduce > > 1. Install and setup HA cluster with JHS > 2.Create state in in JHS where few jobs are killed and Success > Check sorting State in JHS WebUI > Actual > = > Sorting on state column not working in JHS > Expected > == > Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6815) Fix flaky TestKill.testKillTask()
Haibo Chen created MAPREDUCE-6815: - Summary: Fix flaky TestKill.testKillTask() Key: MAPREDUCE-6815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0-alpha1 Reporter: Haibo Chen Assignee: Haibo Chen Error Message Job state is not correct (timedout) expected: but was: Stacktrace java.lang.AssertionError: Job state is not correct (timedout) expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.apache.hadoop.mapreduce.v2.app.MRApp.waitForState(MRApp.java:416) at org.apache.hadoop.mapreduce.v2.app.TestKill.testKillTask(TestKill.java:124) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6273) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
[
https://issues.apache.org/jira/browse/MAPREDUCE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6273:
--
Fix Version/s: 2.8.0
> HistoryFileManager should check whether summaryFile exists to avoid
> FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
>
>
> Key: MAPREDUCE-6273
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6273
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Fix For: 2.8.0, 2.7.2, 2.6.3, 3.0.0-alpha1
>
> Attachments: MAPREDUCE-6273.000.patch, MAPREDUCE-6273.001.patch
>
>
> HistoryFileManager should check whether summaryFile exists to avoid
> FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state,
> I saw the following error message:
> {code}
> 2015-02-17 19:13:45,198 ERROR
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to
> move a job to done
> java.io.FileNotFoundException: File does not exist:
> /user/history/done_intermediate/agd_laci-sluice/job_1423740288390_1884.summary
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1878)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1819)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1771)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:85)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1181)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
> at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:230)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457)
> at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:318)
> at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:59)
> at
> org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:621)
> at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:789)
> at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:785)
> at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
> at org.apache.hadoop.fs.FileContext.open(FileContext.java:785)
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getJobSummary(HistoryFileManager.java:953)
> at
>
