[
https://issues.apache.org/jira/browse/MAPREDUCE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065752#comment-14065752
]
Chris Douglas commented on MAPREDUCE-5974:
------------------------------------------
Could this be equivalently implemented as a composite collector using the
existing plugin architecture? Trading off collector implementations at runtime
is a cool idea, but if the criteria are available to {{init}} then they're also
available during submission (excluding availability of local dependencies or
arch restrictions in heterogeneous clusters).
Changing strategies during the collection phase based on the records emitted
seems to have equivalent or better potential, and is covered by the composite
strategy, also.
> Allow map output collector fallback
> -----------------------------------
>
> Key: MAPREDUCE-5974
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5974
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: task
> Affects Versions: 2.6.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: mapreduce-5974.txt
>
>
> Currently we only allow specifying a single MapOutputCollector implementation
> class in a job. It would be nice to allow a comma-separated list of classes:
> we should try each collector implementation in the user-specified order until
> we find one that can be successfully instantiated and initted.
> This is useful for cases where a particular optimized collector
> implementation cannot operate on all key/value types, or requires native
> code. The cluster administrator can configure the cluster to try to use the
> optimized collector and fall back to the default collector.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
