[
https://issues.apache.org/jira/browse/MAPREDUCE-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016357#comment-14016357
]
Wangda Tan commented on MAPREDUCE-5196:
---------------------------------------
Hi [~curino],
While trying to understand this part of change, I've several questions, hope
you could share some idea of it,
my understand of the workflow, please forgive my current ignorance of this code,
1) CheckpointAMPreemptionPolicy will keep track of which containers
(task-attempt) need to be preempted
2) TaskAttemptListener will set AMFeedback.preempted to true, when
AMPreemptionPolicy.isPreempted(TaskID) returns true
3) Task get AMFeedback, and set mustPreempt. Task takes some action to do some
checkpoints, etc. And call umbilical.preempted(taskId, taskStatus)
But a question here, I found the Task do nothing except printing some logs
after mustPreempt set, is it expected behavior?
If it's expected behavior, the task will keep running until it completed or
killed by NM, resource cannot be proactively released by this task. I think we
should call umbilical.preempted when we found mustPreempt is true, correct?
Another question is, I found in Task.java
{code}
public void done(TaskUmbilicalProtocol umbilical,
TaskReporter reporter
) throws IOException, InterruptedException {
updateCounters();
if (taskStatus.getRunState() == TaskStatus.State.PREEMPTED ) {
// If we are preempted, do no output promotion; signal done and exit
committer.commitTask(taskContext);
umbilical.preempted(taskId, taskStatus);
taskDone.set(true);
reporter.stopCommunicationThread();
return;
}
...
}
{code}
It relies on taskStatus.getRunState() == PREEMPTED, but I found nobody set
taskStatus.runState to PREEMPTED. Could you please tell me which part of code
set taskStatus. If nobody sets runState to PREEMPTED, it cannot invoke
umbilical.preempted properly.
Thanks,
Wangda
> CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
> ------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5196
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5196
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-5196.1.patch, MAPREDUCE-5196.2.patch,
> MAPREDUCE-5196.3.patch, MAPREDUCE-5196.patch, MAPREDUCE-5196.patch
>
>
> This JIRA tracks a checkpoint-based AM preemption policy. The policy handles
> propagation of the preemption requests received from the RM to the
> appropriate tasks, and bookeeping of checkpoints. Actual checkpointing of the
> task state is handled in upcoming JIRAs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
