[
https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124318#comment-16124318
]
Daniel Templeton commented on MAPREDUCE-6870:
---------------------------------------------
Good question. The change breaks the current behavior, but only in unusual
cases, and there is nothing that explicitly states the contract is that a job
will remain running until all the mappers complete. In fact, with speculative
execution, it's just the opposite. What you're breaking here is tradition as
opposed to an execution contract, so I would say that it's not meaningfully
incompatible. However, the contract around applications is that we don't break
anything ever if at all possible. So, strictly speaking, this change is
incompatible, and we should mark it as such. To go into branch-2, it will have
to be off by default.
> Add configuration for MR job to finish when all reducers are complete (even
> with unfinished mappers)
> ----------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.6.1
> Reporter: Zhe Zhang
> Assignee: Peter Bacsko
> Fix For: 3.0.0-beta1
>
> Attachments: MAPREDUCE-6870-001.patch, MAPREDUCE-6870-002.patch,
> MAPREDUCE-6870-003.patch, MAPREDUCE-6870-004.patch, MAPREDUCE-6870-005.patch,
> MAPREDUCE-6870-006.patch, MAPREDUCE-6870-007.patch
>
>
> Even with MAPREDUCE-5817, there could still be cases where mappers get
> scheduled before all reducers are complete, but those mappers run for long
> time, even after all reducers are complete. This could hurt the performance
> of large MR jobs.
> In some cases, mappers don't have any materialize-able outcome other than
> providing intermediate data to reducers. In that case, the job owner should
> have the config option to finish the job once all reducers are complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
