[
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669075#comment-13669075
]
Chris Douglas commented on MAPREDUCE-5176:
------------------------------------------
I'm skeptical of YARN-level annotations for UDFs. While the platform can make
use of that information, surfacing it without framework-specific support seems
awfully ambitious, particularly before we have experience with it. Would others
be OK if we try this in MapReduce, first? If the ideas prove more general, we
can draw some of the related APIs (like checkpointing in MAPREDUCE-5197) into
library code, possibly packaged with YARN.
It'd be great for the annotation to be part of 2.0.5-beta. While it's not
impossible to hide state in other places, adding annotations as in the attached
patch seemed to be the right balance in (a) not preempting {{reduce()}} UDFs by
default (b) adding default behavior for file-based {{OutputFormat}} classes.
Prototype code for the rest of the pipeline is already posted, but it's OK for
the details to be part of a later release. Note that preemption can be disabled
by a configuration flag, which defaults to true.
Any objections or alternative ideas?
> Preemptable annotations (to support preemption in MR)
> -----------------------------------------------------
>
> Key: MAPREDUCE-5176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.patch
>
>
> Proposing a patch that introduces a new annotation @Preemptable that
> represents to the framework property of user-supplied classes (e.g., Reducer,
> OutputCommiter). The intended semantics is that a tagged class is safe to be
> preempted between invocations.
> (this is in spirit similar to the Output Contracts of [Nephele/PACT |
> https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
