[
https://issues.apache.org/jira/browse/TEZ-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710215#comment-15710215
]
Ming Ma commented on TEZ-3222:
------------------------------
Thanks for the clarification!
For the wording for "CompositeRoutedDataMovementEvent", maybe something like "
A convenience class to specify multiple DataMovementEvents which share the same
payload. This event is generated by Edge in Application Master and sent to
downstream vertex input for optimization purpose. It is not to be consumed by
any user code."
For the question of "What if routeCompositeDataMovementEventToDestination
returns null CompositeEventRouteMetadata?". Edge plugin can return null if
there is no route. So it seems Edge should check for it.
> Reduce messaging overhead for auto-reduce parallelism case
> ----------------------------------------------------------
>
> Key: TEZ-3222
> URL: https://issues.apache.org/jira/browse/TEZ-3222
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3222.1.patch, TEZ-3222.2.patch, TEZ-3222.3.patch,
> TEZ-3222.4.patch, TEZ-3222.5.patch, TEZ-3222.6.patch, TEZ-3222.7.patch
>
>
> A dag with 15k x 1000k vertex may auto-reduce to 15k x 1. And while the data
> size is appropriate for 1 task attempt, this results in an increase in task
> attempt message processing of 1000x.
> This jira aims to reduce the message processing in the auto-reduced task
> while keeping the amount of message processing in the AM the same or less.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
