[
https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260172#comment-13260172
]
Siddharth Seth commented on MAPREDUCE-3921:
-------------------------------------------
Couple of questions and suggestions.
- Does 'node unhealthy' need to be treated differently from
'TooManyFetchFailures' ? Killed versus Failed. This ends up with NodeFailures
not counting towards the limit on task attempts.
- JOB_UPDATED_NODES needs to be handled in the JOB_INIT state. Very small
chance of hitting this.
- Minor: JobImpl.actOnUsableNode can get the task type from the id itself. It
doesn't need to fetch the actual task.
- Minor: "if this attempt is not successful" this comment in JobImpl can be
removed. It's removing an entry from a successfulAttempt index.
- In KilledAfterSuccessTransition - createJobCounterUpdateEventTAFailed should
be createJobCounterUpdateEventTAKilled
- TaskImpl.handleAttemptCompletion - finishedAttempts - this will end up double
counting the same task attempt. It's used in some other transition.
- Does the JobHistoryParser need some more changes - to unset fields which may
have been set previously by Map/ReduceAttemptSuccessfulEvents and
TaskFinishedEvent
- For running tasks - shouldn't running Reduce attempts also be killed ?
- RMContainerAllocator.handleUpdatedNodes - instead of fetching the nodeId via
appContext, job etc - the nodeId can be stored with the AssignedRequest. 1)
getTask, getAttempt require readLocks - can avoid these calls every second. 2)
There's an unlikely race where the nodeId may not be assigned in the
TaskAttempt (if the dispatcher thread is backlogged).
- TaskAttemptId.getNodeId() can be avoided. getContainerManagerAddress can be
used instead.
Not related to this patch.
Does JOB_TASK_COMPLETED need to be handled (ignored) in additional states?
> MR AM should act on the nodes liveliness information when nodes go
> up/down/unhealthy
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3921
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Bikas Saha
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch,
> MAPREDUCE-3921-4.patch, MAPREDUCE-3921-branch-0.23.patch,
> MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch,
> MAPREDUCE-3921.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
