[ https://issues.apache.org/jira/browse/HIVE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dima machlin updated HIVE-6448: ------------------------------- Description: too many union all statements in a single query require massive amount of memory to process. Under default HiveServer Xmx (256) we can run ~50 union alls. It seems to be rather linear as increasing the Xmx to 1024 we can run ~200. The error is : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuilder.append(StringBuilder.java:119) at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453) at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598) at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609) at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47) Query example : select * from ( select count( * ) from default.dual union all select count ( * ) from default.dual union all ...... select count( * ) from default.dual) z was: too many union all statements in a single query require massive amount of memory to process. Under default HiveServer Xmx (256) we can run ~50 union alls. It seems to be rather linear as increasing the Xmx to 1024 we can run ~200. The error is : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuilder.append(StringBuilder.java:119) at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453) at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598) at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609) at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47) Query example : select * from ( select count(*) from default.dual union all select count(*) from default.dual union all ...... select count(*) from default.dual) z > Java heap space pasring a query with many union all > --------------------------------------------------- > > Key: HIVE-6448 > URL: https://issues.apache.org/jira/browse/HIVE-6448 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Query Processor > Affects Versions: 0.11.0 > Environment: > Reporter: dima machlin > > too many union all statements in a single query require massive amount of > memory to process. > Under default HiveServer Xmx (256) we can run ~50 union alls. > It seems to be rather linear as increasing the Xmx to 1024 we can run ~200. > The error is : > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2882) > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) > at java.lang.StringBuilder.append(StringBuilder.java:119) > at > org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453) > at > org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598) > at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609) > at > org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) > at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47) > Query example : > select * > from ( > select count( * ) from default.dual union all > select count ( * ) from default.dual union all > ...... > select count( * ) from default.dual) z -- This message was sent by Atlassian JIRA (v6.1.5#6160)