[
https://issues.apache.org/jira/browse/TEZ-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243956#comment-15243956
]
Ming Ma commented on TEZ-3206:
------------------------------
Another thing for the DataMovementEvent is its impact on AM memory. it appears
the reducer vertex stores DMEs and other events from mappers. So for the
auto-parallelism case where it has many empty partitions, the overall event
buffer size could be large.
In addition, DME's empty partition payload has all empty partitions from a
specific mapper. At the reducer side, it only cares about the partitions it is
responsible for, not partitions for other reducers. Perhaps we can optimize AM
to send only the relevant empty partitions to the reducer. Also for the
auto-parallelism case, instead of sending one DME to the reducer at a time, it
can send batch events similar to {{CompositeDataMovementEvent}}, useful for the
auto-parallelism case.
Anyway, there are couple issues discussed here. I will use this jira to add the
VertexManagerEvent routing from {{UnorderedPartitionedKVWriter}} to Tez AM. We
can discuss the followings in other jiras:
* Use more precise partition stats in VertexManagerEvent.
* Optimize DME routing between AM and downstream vertex.
> Have unordered partitioned KV output send partition stats via
> VertexManagerEvent
> ---------------------------------------------------------------------------------
>
> Key: TEZ-3206
> URL: https://issues.apache.org/jira/browse/TEZ-3206
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Ming Ma
>
> As part of the auto-parallelism feature, ordered partitioned KV output's
> partition stats are sent to ShuffleVertexManager via VertexManagerEvent. But
> this isn't available for unordered partitioned output. Having
> {{UnorderedPartitionedKVWriter}} send partition stats will enable the
> auto-parallelism support for unordered KV or other custom data routing
> mechanisms that depend on partition size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
