Ming Ma created TEZ-3217:
----------------------------
Summary: Optimize DataMovementEvent routing between AM and
downstream vertex
Key: TEZ-3217
URL: https://issues.apache.org/jira/browse/TEZ-3217
Project: Apache Tez
Issue Type: Improvement
Reporter: Ming Ma
Follow up on TEZ-3206 discussion, we might be able to optimize the DME routing
from AM to downstream vertex, mostly for the auto-parallelism case.
* DME's empty partition payload has all empty partitions from a specific
mapper. At the reducer side, it only cares about the partitions it is
responsible for, not partitions belong to other reducers. Perhaps we can
optimize AM to send only the relevant empty partitions to that reducer.
* Instead of sending one DME to a given reducer at a time, it can batch all
DMEs belonging to a given (mapper, reducer) pair with the common empty
partition payload, similar to CompositeDataMovementEvent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
