[jira] [Resolved] (MAPREDUCE-4743) Job is marking as FAILED and also throwing the Transition exception instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K resolved MAPREDUCE-4743.
--
Resolution: Fixed
It is fixed in the latest code.
Job is marking as FAILED and also throwing the Transition exception instead
of KILLED when issues a KILL command
Key: MAPREDUCE-4743
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4743
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha
Reporter: Devaraj K
Assignee: Devaraj K
{code:xml}
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
T_KILL at SUCCEEDED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:605)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl.handle(TaskImpl.java:89)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:903)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher.handle(MRAppMaster.java:897)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5309) 2.0.4 JobHistoryParser can't parse certain failed job history files generated by 2.0.3 history server
Vrushali C created MAPREDUCE-5309:
-
Summary: 2.0.4 JobHistoryParser can't parse certain failed job
history files generated by 2.0.3 history server
Key: MAPREDUCE-5309
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5309
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.0.4-alpha
Reporter: Vrushali C
When the 2.0.4 JobHistoryParser tries to parse a job history file generated by
hadoop 2.0.3, the jobhistoryparser throws as an error as
java.lang.ClassCastException: org.apache.avro.generic.GenericData$Array cannot
be cast to org.apache.hadoop.mapreduce.jobhistory.JhCounters
at
org.apache.hadoop.mapreduce.jobhistory.TaskAttemptUnsuccessfulCompletion.put(TaskAttemptUnsuccessfulCompletion.java:58)
at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:138)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:129)
at
org.apache.hadoop.mapreduce.jobhistory.EventReader.getNextEvent(EventReader.java:93)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:111)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:156)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:142)
at
com.twitter.somepackage.Test20JobHistoryParsing.testFileAvro(Test20JobHistoryParsing.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Test code and the job history file are attached.
Test code:
package com.twitter.somepackagel;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser;
import org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.JobInfo;
import org.junit.Test;
import org.apache.hadoop.yarn.YarnException;
public class Test20JobHistoryParsing {
@Test
public void testFileAvro() throws IOException
{
Path local_path2 = new Path(/tmp/job_2_0_3-KILLED.jhist);
JobHistoryParser parser2 = new JobHistoryParser(FileSystem.getLocal(new
Configuration()), local_path2);
try {
JobInfo ji2 = parser2.parse();
System.out.println( job info: + ji2.getJobname() +
+ ji2.getFinishedMaps() +
+ ji2.getTotalMaps() +
+ ji2.getJobId() ) ;
}
catch (IOException e) {
throw new YarnException(Could not load history file
+ local_path2.getName(), e);
}
}
}
This seems to stem from the fix in
[jira] [Resolved] (MAPREDUCE-5252) Fair scheduler should use SchedulerUtils.normalizeRequest
[ https://issues.apache.org/jira/browse/MAPREDUCE-5252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5252. --- Resolution: Not A Problem This was fixed in YARN-326 Fair scheduler should use SchedulerUtils.normalizeRequest - Key: MAPREDUCE-5252 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5252 Project: Hadoop Map/Reduce Issue Type: Improvement Components: scheduler Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor The capacity scheduler and the fifo scheduler use the same normalizeRequest in SchedulerUtils. The fair scheduler has its own version of this method that does exactly the same thing. It should use the common one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5310) MRAM should not normalize allocation request capabilities
Alejandro Abdelnur created MAPREDUCE-5310: - Summary: MRAM should not normalize allocation request capabilities Key: MAPREDUCE-5310 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5310 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur The MRAM is assuming knowledge of the scheduler internals to normalize allocation request capabilities. Per discussions in YARN-689 and YARN-769 it should not do that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5182) LineRecordReader#getProgress throwing IOException breaks compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-5182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza resolved MAPREDUCE-5182. --- Resolution: Won't Fix LineRecordReader#getProgress throwing IOException breaks compatibility -- Key: MAPREDUCE-5182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5182 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2, 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza This has been in trunk for a while (since MAPREDUCE-773), but was only introduced into branch-1 in July. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5312) TestRMNMInfo is failing
Alejandro Abdelnur created MAPREDUCE-5312:
-
Summary: TestRMNMInfo is failing
Key: MAPREDUCE-5312
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5312
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Alejandro Abdelnur
2 test methods are failing:
{code}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 13.904 sec
FAILURE!
testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 198
sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
testRMNMInfoMissmatch(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time
elapsed: 146 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfoMissmatch(TestRMNMInfo.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at
