[
https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367813#comment-14367813
]
Gopal V commented on TEZ-145:
-----------------------------
This is a question for [~bikassaha].
There is an combiner edge & vertex manager that needs to go along with this to
which converts all partitions from all local input into one combine processor
(i.e if it finds it has remote fetches to do, it should just forward the DME
events using the pipelined mode of >1 event per-attempt).
To be able to bail-out with a no-op like that, all partitioning through-out has
to be exactly the reducer partition count.
This is the most optimal mode, but this makes everything extra complex.
Assume you have 600 hosts over 30 racks which ran a map-task + 2000 partitions
in the reducer.
The host-level combiner input count is actually 600 x 2000 partitions, which
can be grouped into 600 x m groups - not 2000 groups.
The rack-level combiner input count is actually 30 x 2000 partitions, which can
be grouped into 30 x n groups - not 2000 groups.
Yet, all the inputs are actually always partitioned into 2000 partitions and
the destination task-index is determined by something other than the partition.
So, how practical is that?
> Support a combiner processor that can run non-local to map/reduce nodes
> -----------------------------------------------------------------------
>
> Key: TEZ-145
> URL: https://issues.apache.org/jira/browse/TEZ-145
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch
>
>
> For aggregate operators that can benefit by running in multi-level trees,
> support of being able to run a combiner in a non-local mode would allow
> performance efficiencies to be gained by running a combiner at a rack-level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
