[
https://issues.apache.org/jira/browse/IGNITE-3414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366651#comment-15366651
]
Ivan Veselovsky commented on IGNITE-3414:
-----------------------------------------
issue #2: in the end of method
org.apache.ignite.hadoop.mapreduce.IgniteHadoopWeightedMapReducePlanner#assignLocalReducers
it was written
{code}
// Update result map.
if (res > 0)
resMap.put(nodeId, res);
{code}
this code was overwriting previous value assigned to same id. That lead to
lower total number of reducers than was initially requested in the job config.
I suppose, an increment should be there, that is, the old value + res.
> Hadoop: Optimize map-reduce job planning.
> -----------------------------------------
>
> Key: IGNITE-3414
> URL: https://issues.apache.org/jira/browse/IGNITE-3414
> Project: Ignite
> Issue Type: Task
> Components: hadoop
> Affects Versions: 1.6
> Reporter: Vladimir Ozerov
> Assignee: Vladimir Ozerov
> Priority: Critical
> Fix For: 1.7
>
>
> Currently Hadoop module has inefficient map-reduce planning engine. In
> particular, it assigns tasks only to affinity nodes. It could lead to
> situation when very huge tasks is processed by a single cluster node, while
> other cluster nodes are idle.
> We should implement configurable map-reduce planner which will be able to
> utilize the whole cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
