[
https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184157#comment-13184157
]
Karam Singh commented on MAPREDUCE-3656:
----------------------------------------
For org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
NM log says :
{code}
2012-01-10 14:36:03,779 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=<jobuser>
IP=<ip_address_of_RM> OPERATION=Stop Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1326205390797_0002
CONTAINERID=container_1326205390797_0002_01_000001
2012-01-10 14:36:03,780 INFO 2012-01-10 14:36:03,785 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Processing container_1326205390797_0002_01_000001 of type KILL_CONTAINER
{code}
RM log says :
{code}
2012-01-10 14:33:14,505 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Processing container_1326205390797_0002_01_000001 of type LAUNCHED
2012-01-10 14:33:14,505 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1326205390797_0002_01_000001 Container Transitioned from ACQUIRED to
RUNNING
2012-01-10 14:36:03,846 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
Processing container_1326205390797_0002_01_000001 of type KILL
2012-01-10 14:36:03,846 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
container_1326205390797_0002_01_000001 Container Transitioned from RUNNING to
KILLED
2012-01-10 14:36:03,846 INFO
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp: Completed
container: container_1326205390797_0002_01_000001 in state: KILLED event:KILL
{code}
Stack trace of InvalidStateTransitonException :
{code}
2012-01-10 14:34:39,289 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Processing
job_1326205390797_0002 of type JOB_TASK_COMPLETED
2012-01-10 14:34:39,289 INFO [AsyncDispatcher event
handler]org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Processing
attempt_1326205390797_0002_r_000154_0 of type TA_UPDATE
2012-01-10 14:34:39,289 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Processing
job_1326205390797_0002 of type JOB_COUNTER_UPDATE
2012-01-10 14:34:39,290 ERROR [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event at
current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
JOB_COUNTER_UPDATE at ERROR
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:699)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:112)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:849)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:845)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
at java.lang.Thread.run(Thread.java:619)
{code}
And at the when AM log say :
{code}
2012-01-10 14:36:03,755 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002.summary_tmp
to
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002.summary
2012-01-10 14:36:03,755 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002_conf.xml_tmp
to
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002_conf.xml
2012-01-10 14:36:03,756 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to
done:
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002-1326205993489-jobuser-sorter-1326206079288-0-0-ERROR-default.jhist_tmp
to
hdfs://NN/mapred/history/done_intermediate/jobuser/job_1326205390797_0002-1326205993489-jobuser-sorter-1326206079288-0-0-ERROR-default.jhist
2012-01-10 14:36:03,756 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped
JobHistoryEventHandler. super.stop()
2012-01-10 14:36:03,756 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.service.AbstractService: Service:JobHistoryEventHandler
is stopped.
2012-01-10 14:36:03,756 ERROR [ContainerLauncher Event Handler]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Returning,
interrupted : java.lang.InterruptedException
2012-01-10 14:36:03,759 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.service.AbstractService:
Service:org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl is
stopped.
2012-01-10 14:36:03,759 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.service.AbstractService:
Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
is stopped.
2012-01-10 14:36:03,760 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Allocated thread
interrupted. Returning.
2012-01-10 14:36:03,760 ERROR [Thread-27]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
Returning,interrupted : java.lang.InterruptedException
2012-01-10 14:36:03,761 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job
diagnostics to Invalid event TA_UPDATE on
TaskAttempt attempt_1326205390797_0002_m_007534_0
Invalid event JOB_COUNTER_UPDATE on Job job_1326205390797_0002
Invalid event JOB_COUNTER_UPDATE on Job job_1326205390797_0002
Invalid event JOB_COUNTER_UPDATE on Job job_1326205390797_0002
Invalid event JOB_COUNTER_UPDATE on Job job_1326205390797_0002
Invalid event JOB_COUNTER_UPDATE on Job job_1326205390797_0002
{code}
However WebUI of RM or HistoryServer of JobClient does not show any diagnostics
messages.
> Sort job on 350 scale is consistently failing with latest MRV2 code
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.1
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently
> failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM
> container.
> But at INFO level from RM log not able find why RM is killing AM when job is
> not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
