[
https://issues.apache.org/jira/browse/MAPREDUCE-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634684#comment-16634684
]
Jason Lowe commented on MAPREDUCE-7130:
---------------------------------------
I'm not a Rumen expert and I haven't dug into the Rumen side for this yet. I'm
wondering why Rumen is trying to apply pre-Hadoop 0.21 enums to a version well
past Hadoop 0.21. Maybe the "Pre21" is a misnomer?
It looks like this may have been triggered by MAPREDUCE-5795. Before that it
always emitted "KILLED" as the status of an unsuccessful job completion, but
that code changed it to emit SUCCEEDED in some cases. I agree that SUCCEEDED
has been used for a job status for a really long time now, since before 2011.
So my initial reaction is Rumen shouldn't be applying a pre-Hadoop 21 enum to
this field at all, certainly not for MRAppMaster recovery files which didn't
even exist pre-Hadoop 0.21. It seems like it should be using the JobStatus
enum for this instead. Certainly adding SUCCEEDED to the enum or something
similar would also work, but again I'm wondering why it's using a custom enum
set here.
> Rumen crashes trying to handle MRAppMaster recovery events
> ----------------------------------------------------------
>
> Key: MAPREDUCE-7130
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7130
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
> Reporter: Jonathan Bender
> Priority: Minor
>
> In the event of an MRAppMaster recovery, the Job History file gets an event
> of the following form:
> {code:json}
> {"type":"JOB_KILLED","event":\{"org.apache.hadoop.mapreduce.jobhistory.JobUnsuccessfulCompletion":{"jobid":"job_1532048817013_xxxx","finishTime":1534521962641,"finishedMaps":0,"finishedReduces":0,"jobStatus":"SUCCEEDED","diagnostics":{"string":"Job
> commit succeeded in a prior MRAppMaster attempt before it crashed.
> Recovering."},"failedMaps":0,"failedReduces":0,"killedMaps":0,"killedReduces":0}}}
> {code}
> The issue seems to be around the SUCCEEDED job status for a
> JobUnsuccessfulCompletion:
> https://github.com/apache/hadoop/blob/e0f6ffdbad6f43fd43ec57fb68ebf5275b8b9ba0/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/JobBuilder.java#L609
> Which fails to find the enum here:
> https://github.com/apache/hadoop/blob/e0f6ffdbad6f43fd43ec57fb68ebf5275b8b9ba0/hadoop-tools/hadoop-rumen/src/main/java/org/apache/hadoop/tools/rumen/Pre21JobHistoryConstants.java#L50
> I'm not sure if this is an error with the Rumen parser or if the job history
> file is getting into an invalid state.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
