[
https://issues.apache.org/jira/browse/TEZ-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700005#comment-14700005
]
Hitesh Shah commented on TEZ-2726:
----------------------------------
\cc [~bikassaha]
> Handle invalid number of partitions for SCATTER-GATHER edge
> -----------------------------------------------------------
>
> Key: TEZ-2726
> URL: https://issues.apache.org/jira/browse/TEZ-2726
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Saikat
> Assignee: Saikat
>
> Encountered an issue where the source vertex has M task and sink vertex has N
> tasks (N > M), [e.g. M = 1, N = 3]and the edge is of type SCATTER -GATHER.
> This resulted in sink vertex receiving DMEs with non existent targetIds.
> The fetchers for the sink vertex tasks then try to retrieve the map outputs
> and retrieve invalid headers due to exception in the ShuffleHandler.
> Possible fixes:
> 1. raise proper Tez Exception to indicate this invalid scenario.
> 2. or write appropriate empty partition bits, for the missing partitions
> before sending out the DMEs to sink vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
