[
https://issues.apache.org/jira/browse/TEZ-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700555#comment-14700555
]
Bikas Saha commented on TEZ-2726:
---------------------------------
Still not sure, what the exact sequence of events was for the error. A planning
bug cause empty partitions and somehow Tez handled the empty partitions
erroneously? It will really help if we had logs or some sequence of events that
produced the error. Tez does have some handling for empty partitions but thats
an optimization to not fetch them (since they are empty).
> Handle invalid number of partitions for SCATTER-GATHER edge
> -----------------------------------------------------------
>
> Key: TEZ-2726
> URL: https://issues.apache.org/jira/browse/TEZ-2726
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Saikat
> Assignee: Saikat
>
> Encountered an issue where the source vertex has M task and sink vertex has N
> tasks (N > M), [e.g. M = 1, N = 3]and the edge is of type SCATTER -GATHER.
> This resulted in sink vertex receiving DMEs with non existent targetIds.
> The fetchers for the sink vertex tasks then try to retrieve the map outputs
> and retrieve invalid headers due to exception in the ShuffleHandler.
> Possible fixes:
> 1. raise proper Tez Exception to indicate this invalid scenario.
> 2. or write appropriate empty partition bits, for the missing partitions
> before sending out the DMEs to sink vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
