[
https://issues.apache.org/jira/browse/TEZ-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700369#comment-14700369
]
Rajesh Balamohan commented on TEZ-2726:
---------------------------------------
[~saikatr] - Is there any repro for this? When you say invalid headers, is it
something like the following? Can you plz provide more info?
{noformat}
org.apache.tez.runtime.library.common.shuffle.impl.Fetcher: Invalid map id
java.lang.IllegalArgumentException: Invalid header received:
W^s??.attempt_1399351577718_4169_1_ partition: 95
{noformat}
If so, are you using "tez.runtime.intermediate-output.compress.codec =
org.apache.hadoop.io.compress.DefaultCodec" ?
> Handle invalid number of partitions for SCATTER-GATHER edge
> -----------------------------------------------------------
>
> Key: TEZ-2726
> URL: https://issues.apache.org/jira/browse/TEZ-2726
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Saikat
> Assignee: Saikat
>
> Encountered an issue where the source vertex has M task and sink vertex has N
> tasks (N > M), [e.g. M = 1, N = 3]and the edge is of type SCATTER -GATHER.
> This resulted in sink vertex receiving DMEs with non existent targetIds.
> The fetchers for the sink vertex tasks then try to retrieve the map outputs
> and retrieve invalid headers due to exception in the ShuffleHandler.
> Possible fixes:
> 1. raise proper Tez Exception to indicate this invalid scenario.
> 2. or write appropriate empty partition bits, for the missing partitions
> before sending out the DMEs to sink vertex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
