[
https://issues.apache.org/jira/browse/TEZ-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947988#comment-15947988
]
Gopal V commented on TEZ-3676:
------------------------------
This looks like a repeat of HIVE-12957
> Desired number of tasks in TezGroupedSplitsInputFormat could be negative
> ------------------------------------------------------------------------
>
> Key: TEZ-3676
> URL: https://issues.apache.org/jira/browse/TEZ-3676
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Ming Ma
>
> RM sometimes returned negative headroom value to Tez AM, which then generated
> negative number of tasks, which will cause the job to fail. We need to look
> into why FairScheduler will return negative headroom, but it will be useful
> to make Tez more robust against this. For example TezGroupedSplitsInputFormat
> can use the value from the original input format in such scenario.
> {noformat}
> , java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
> at
> org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.setDesiredNumberOfSplits(TezGroupedSplitsInputFormat.java:72)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java:442)
> at
> org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:329)
> at
> org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
