[
https://issues.apache.org/jira/browse/TEZ-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14491654#comment-14491654
]
Gopal V commented on TEZ-145:
-----------------------------
bq. Not sure why pipelining is required for this? Essentially we are
introducing another vertex that is doing some partial grouping.
Pipelining is perhaps the wrong word, but generating multiple partial events
from a single task is the performance improvement.
That allows a locality missed task to go from start to finish without moving
any data across.
Reducers tend to be spread across racks (which might be another thing to tune
for when using a small queue on a big cluster).
Transducers don't have that problem, they're optional - to avoid producing a
cross-rack fetches for an intermediate stage, the AM would have to excise any
task that runs with sub-optimal locality after the runtime-expansion &
scheduling of the downstream with TaskSpecs.
The addition of the state machine to handle partial/final events in order for
Inputs, effectively allows forwarding of the entire "split" as-is with a
final=false flag, making that an entirely local decision in the Task rather
than re-adjusting vertex fan-out during runtime.
> Support a combiner processor that can run non-local to map/reduce nodes
> -----------------------------------------------------------------------
>
> Key: TEZ-145
> URL: https://issues.apache.org/jira/browse/TEZ-145
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Hitesh Shah
> Assignee: Tsuyoshi Ozawa
> Attachments: TEZ-145.2.patch, WIP-TEZ-145-001.patch
>
>
> For aggregate operators that can benefit by running in multi-level trees,
> support of being able to run a combiner in a non-local mode would allow
> performance efficiencies to be gained by running a combiner at a rack-level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
