[jira] [Created] (TEZ-3402) SplitGrouper: Integer overflow

[Gopal V (JIRA)]

Gopal V created TEZ-3402:
----------------------------

             Summary: SplitGrouper: Integer overflow
                 Key: TEZ-3402
                 URL: https://issues.apache.org/jira/browse/TEZ-3402
             Project: Apache Tez
          Issue Type: Bug
    Affects Versions: 0.8.4
            Reporter: Gopal V


Bad configs triggers integer overflow. This is a 5Tb query which tries to group 
by max-size of 4096.

{code}
    groupedSplits = new ArrayList<GroupedSplitContainer>(desiredNumSplits);
{code}

{code}
diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due 
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed, 
vertex=vertex_1470081722620_0072_3_00 [Map 2], 
java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
        at java.util.ArrayList.<init>(ArrayList.java:156)
        at 
org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
        at 
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
        at 
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
        at 
org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
        at 
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
        at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
        at 
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)