Hao Zhu created DRILL-3678:
------------------------------
Summary: Plan generating for Drill on Hive takes huge java heap
size
Key: DRILL-3678
URL: https://issues.apache.org/jira/browse/DRILL-3678
Project: Apache Drill
Issue Type: Bug
Components: Query Planning & Optimization
Affects Versions: 1.1.0
Environment: 1.1
Reporter: Hao Zhu
Assignee: Jinfeng Ni
===Env===
Drill 1.0 on Hive 0.13
(Also tested Drill 1.1 and get the same behavior)
8 nodes drill cluster.
Jave heap size is set to 8G and Direct memory is set to 96G on each drillbit.
===Symptom===
This is a Hive parquet partition table which has multi-level partitions.
The Hive table size is several TB with tens of thousands leaf partitions.
When doing a "select * from table limit 10", the query keeps in "pending" state
to generate the SQL plan. And finally the drillbits crashed with java heap OOM.
{code}
java.lang.OutOfMemoryError: Java heap space
at
hive.parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:599)
at
hive.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360)
at
hive.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:100)
at
hive.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:172)
at
hive.parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at
org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:66)
at
org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at
org.apache.drill.exec.store.hive.HiveRecordReader.init(HiveRecordReader.java:246)
at
org.apache.drill.exec.store.hive.HiveRecordReader.<init>(HiveRecordReader.java:138)
at
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:58)
at
org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch(HiveScanBatchCreator.java:34)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:150)
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:173)
at
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:106)
at
org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:81)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:235)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Node ran out of Heap memory, exiting.
java.lang.OutOfMemoryError: Java heap space
{code}
We captured some stacktraces of foreman thread in foreman drillbit, here are 2
times' examples:
{code}
2a482cd9-7fb2-c492-1356-d049e90870c8:foreman id=115 state=RUNNABLE
at org.apache.xerces.dom.DeferredElementNSImpl.synchronizeData(Unknown
Source)
at org.apache.xerces.dom.ElementImpl.getTagName(Unknown Source)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2348)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2234)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2151)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:871)
at
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:421)
at org.apache.drill.exec.store.hive.HiveScan.splitInput(HiveScan.java:178)
at org.apache.drill.exec.store.hive.HiveScan.getSplits(HiveScan.java:167)
at org.apache.drill.exec.store.hive.HiveScan.access$000(HiveScan.java:69)
at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:146)
at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at
org.apache.drill.exec.store.hive.HiveScan.getSplitsWithUGI(HiveScan.java:144)
at org.apache.drill.exec.store.hive.HiveScan.(HiveScan.java:119)
at
org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:78)
at
org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:41)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:53)
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:72)
at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:81)
at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:63)
at
org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:37)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:514)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:211)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:246)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:162)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:903)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:242)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
--------
2a482cd9-7fb2-c492-1356-d049e90870c8:foreman id=115 state=RUNNABLE
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
- locked java.io.InputStreamReader@63f0c79a
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
- locked java.io.InputStreamReader@63f0c79a
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at
javax.xml.parsers.FactoryFinder.findJarServiceProvider(FactoryFinder.java:319)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:255)
at
javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2257)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2234)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2151)
- locked org.apache.hadoop.mapred.JobConf@6ef80e0e
at org.apache.hadoop.conf.Configuration.get(Configuration.java:871)
at
org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:421)
at org.apache.drill.exec.store.hive.HiveScan.splitInput(HiveScan.java:178)
at org.apache.drill.exec.store.hive.HiveScan.getSplits(HiveScan.java:167)
at org.apache.drill.exec.store.hive.HiveScan.access$000(HiveScan.java:69)
at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:146)
at org.apache.drill.exec.store.hive.HiveScan$1.run(HiveScan.java:144)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at
org.apache.drill.exec.store.hive.HiveScan.getSplitsWithUGI(HiveScan.java:144)
at org.apache.drill.exec.store.hive.HiveScan.(HiveScan.java:119)
at
org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:78)
at
org.apache.drill.exec.store.hive.HiveStoragePlugin.getPhysicalScan(HiveStoragePlugin.java:41)
at
org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:53)
at
org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:72)
at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:81)
at org.apache.drill.exec.planner.logical.DrillScanRel.(DrillScanRel.java:63)
at
org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:37)
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:228)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:795)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:316)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.logicalPlanningVolcanoAndLopt(DefaultSqlHandler.java:514)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:211)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:246)
at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:162)
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:178)
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:903)
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:242)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
===Workaround===
We have to increase the JAVA heap size to 24GB and then query can finish
successfully.
Even though the query took about 16mins to generate the plan.
===Asks===
Drill on Hive parquet query should utilize direct memory instead of java heap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
