[
https://issues.apache.org/jira/browse/IGNITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383758#comment-15383758
]
Vladimir Ozerov commented on IGNITE-3414:
-----------------------------------------
1) Because we want to assign local reducers until some limit is reached, and
only then switch to weight-based distribution. This way when there are only
several reducers, they are likely to be assigned to nodes where some mappers
are run. If we rely only on weights, the same behavior is only possible when
local and remote reducer weights are different. And this will be bad for cases
when there are lots reducers - the imbalance between local and remote nodes
will grow linearly.
2) This is exactly how it works now.
3) Too difficult for a very rare case when there are more than one node on a
single machine.
> Hadoop: Optimize map-reduce job planning.
> -----------------------------------------
>
> Key: IGNITE-3414
> URL: https://issues.apache.org/jira/browse/IGNITE-3414
> Project: Ignite
> Issue Type: Task
> Components: hadoop
> Affects Versions: 1.6
> Reporter: Vladimir Ozerov
> Assignee: Vladimir Ozerov
> Priority: Critical
> Fix For: 1.7
>
>
> Currently Hadoop module has inefficient map-reduce planning engine. In
> particular, it assigns tasks only to affinity nodes. It could lead to
> situation when very huge tasks is processed by a single cluster node, while
> other cluster nodes are idle.
> We should implement configurable map-reduce planner which will be able to
> utilize the whole cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
