Saikat created TEZ-2726:
---------------------------
Summary: Handle invalid number of partitions for SCATTER-GATHER
edge
Key: TEZ-2726
URL: https://issues.apache.org/jira/browse/TEZ-2726
Project: Apache Tez
Issue Type: Improvement
Reporter: Saikat
Encountered an issue where the source vertex has M task and sink vertex has N
tasks (N > M), [e.g. M = 1, N = 3]and the edge is of type SCATTER -GATHER.
This resulted in sink vertex receiving DMEs with non existent targetIds.
The fetchers for the sink vertex tasks then try to retrieve the map outputs and
retrieve invalid headers due to exception in the ShuffleHandler.
Possible fixes:
1. raise proper Tez Exception to indicate this invalid scenario.
2. or write appropriate empty partition bits, for the missing partitions before
sending out the DMEs to sink vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
