We keep running into this issue when trying to issue a query with hashagg disabled. When I look at system memory usage though, drill doesn't seem to be using much of it but still hits this error.
Our environment: - 1 r3.8xl - 1 drillbit version 1.10.0 configured with 4GB of Heap and 230G of Direct - Data stored on S3 is compressed CSV I've tried increasing planner.memory.max_query_memory_per_node to 230G and lowered planner.width.max_per_query to 1 and it still fails. We've applied the patch from this bug in the hopes that it would resolve the issue but it hasn't: https://issues.apache.org/jira/browse/DRILL-5226 Stack Trace: (org.apache.drill.exec.exception.OutOfMemoryException) Unable to allocate buffer of size 16777216 due to memory limit. Current allocation: 8445952 org.apache.drill.exec.memory.BaseAllocator.buffer():220 org.apache.drill.exec.memory.BaseAllocator.buffer():195 org.apache.drill.exec.vector.VarCharVector.reAlloc():425 org.apache.drill.exec.vector.VarCharVector.copyFromSafe():278 org.apache.drill.exec.vector.NullableVarCharVector.copyFromSafe():379 org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.doCopy():22 org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.next():75 org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():602 org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():428 org.apache.drill.exec.record.AbstractRecordBatch.next():162 org.apache.drill.exec.record.AbstractRecordBatch.next():119 org.apache.drill.exec.record.AbstractRecordBatch.next():109 org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():137 org.apache.drill.exec.record.AbstractRecordBatch.next():162 org.apache.drill.exec.physical.impl.BaseRootExec.next():104 org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144 org.apache.drill.exec.physical.impl.BaseRootExec.next():94 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232 org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226 java.security.AccessController.doPrivileged():-2 javax.security.auth.Subject.doAs():422 org.apache.hadoop.security.UserGroupInformation.doAs():1657 org.apache.drill.exec.work.fragment.FragmentExecutor.run():226 org.apache.drill.common.SelfCleaningRunnable.run():38 java.util.concurrent.ThreadPoolExecutor.runWorker():1142 java.util.concurrent.ThreadPoolExecutor$Worker.run():617 java.lang.Thread.run():745 (state=,code=0) Is there something I'm missing here? Any help/direction would be appreciated. Thanks, Nate
