Ming Ma created TEZ-3676:
----------------------------
Summary: Desired number of tasks in TezGroupedSplitsInputFormat
could be negative
Key: TEZ-3676
URL: https://issues.apache.org/jira/browse/TEZ-3676
Project: Apache Tez
Issue Type: Bug
Reporter: Ming Ma
RM sometimes returned negative headroom value to Tez AM, which then generated
negative number of tasks, which will cause the job to fail. We need to look
into why FairScheduler will return negative headroom, but it will be useful to
make Tez more robust against this. For example TezGroupedSplitsInputFormat can
use the value from the original input format in such scenario.
{noformat}
, java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
at
org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.setDesiredNumberOfSplits(TezGroupedSplitsInputFormat.java:72)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java:442)
at
org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:329)
at
org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
