[
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671046#comment-13671046
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-5176:
----------------------------------------------------
bq. BTW I think we are in a great spot to carry along this conversation, since
we have one very specific example of these annotations: @Preemptable for which
we have the entire end-to-end usage scenario (all the preemption in mapreduce
stuff tracked in MAPREDUCE-5189, MAPREDUCE-5192, MAPREDUCE-5194,
MAPREDUCE-5196, MAPREDUCE-5197 and few upcoming ones), and plenty more ideas
coming up from people.
Yeah, clearly for now, this is a MR only thing. We can promote it to YARN when
we see more uses of it.
Even so, seems like here there is the usage of preemption and checkpointing
interchangeably, but it feels like this is better off called @Checkpointable.
If you agree
- some javadoc in the patch also needs to be fixed
- PreemptableFileOutputCommitter name is fine? I don't know for sure.
Still haven't looked at the patch in the context of the whole thing, but had a
quick look at it:
- Call PartialOutputCommitter and PreemptableFileOutputCommitter to be public
but evolving?
- mapreduce/lib/output/FileOutputCommitter: getCommittedTaskPath can be
protected?
- hdfu://user/hadoop/out : Typo? It perhaps doesn't matter, but..
Also some statements, please correct me if I am wrong:
- The clean up of partitial outputs from previous attempts is done by AM/task.
- When the reducer gets the preempt signal, it uses the regular commit api to
commit the output and then for successive commits too
- The current abortTask/cleanupTask won't be enough as it will only delete the
current TaskAttempts output. cleanUpPartialOutputForTask deletes all the
previous tasks' output.
> Preemptable annotations (to support preemption in MR)
> -----------------------------------------------------
>
> Key: MAPREDUCE-5176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.patch
>
>
> Proposing a patch that introduces a new annotation @Preemptable that
> represents to the framework property of user-supplied classes (e.g., Reducer,
> OutputCommiter). The intended semantics is that a tagged class is safe to be
> preempted between invocations.
> (this is in spirit similar to the Output Contracts of [Nephele/PACT |
> https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
