Hi Drill experts,
While attempting to run the query, "select
<nestedTableName>.<record>.<column1>, <nestedTableName>.<record>.<column2> from
(select flatten(tbl.<nestedTableName>.`array`) <record> from
dfs.`/server/apache-drill-1.0.0/sample-data/myfile` tbl ) as <nestedTableName>;"
I run into an OOM error. It seems to be running out of memory during the
flattening process.
java.lang.RuntimeException: java.sql.SQLException: RESOURCE ERROR: One or more
nodes ran out of memory while executing the query.
Fragment 1:4
[Error Id: 69a33085-d059-4e71-a899-e181ed741715 on localhost:31010]
(org.apache.drill.exec.memory.OutOfMemoryRuntimeException) Failure while
reallocating buffer to 16777216 bytes
org.apache.drill.exec.vector.Float4Vector.reAlloc():192
org.apache.drill.exec.vector.Float4Vector.copyFromSafe():292
org.apache.drill.exec.vector.NullableFloat4Vector.copyFromSafe():301
org.apache.drill.exec.vector.NullableFloat4Vector$TransferImpl.copyValueSafe():266
org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapTransferPair.copyValueSafe():355
org.apache.drill.exec.vector.complex.MapVector$MapTransferPair.copyValueSafe():208
org.apache.drill.exec.vector.complex.MapVector.copyFromSafe():83
org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.copyAsValue():97
org.apache.drill.exec.test.generated.FlattenerGen5.doEval():30
org.apache.drill.exec.test.generated.FlattenerGen5.flattenRecords():93
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.doWork():153
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
org.apache.drill.exec.physical.impl.flatten.FlattenRecordBatch.innerNext():119
org.apache.drill.exec.record.AbstractRecordBatch.next():146
org.apache.drill.exec.record.AbstractRecordBatch.next():105
org.apache.drill.exec.record.AbstractRecordBatch.next():95
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():146
org.apache.drill.exec.record.AbstractRecordBatch.next():105
org.apache.drill.exec.record.AbstractRecordBatch.next():95
org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
org.apache.drill.exec.record.AbstractRecordBatch.next():146
org.apache.drill.exec.physical.impl.BaseRootExec.next():83
org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():95
org.apache.drill.exec.physical.impl.BaseRootExec.next():73
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():259
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():253
java.security.AccessController.doPrivileged():-2
javax.security.auth.Subject.doAs():415
org.apache.hadoop.security.UserGroupInformation.doAs():1556
org.apache.drill.exec.work.fragment.FragmentExecutor.run():253
org.apache.drill.common.SelfCleaningRunnable.run():38
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():745
at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
at
sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:85)
at sqlline.TableOutputFormat.print(TableOutputFormat.java:116)
I've tried enabling the planner.memory.enable_memory_estimation property, but
that didn't help.
I have the following memory settings:
DRILL_MAX_DIRECT_MEMORY="8G"
DRILL_HEAP="4G"
And the file itself is only 1.4 GB. Is there some other way to trade off
performance for reliability? Also, are there some known limitations with
flatten?
Thanks,
Alvin
