Re: External Sort - Unable to Allocate Buffer error

[Zelaine Fong]

Nate,

The Jira you’ve referenced relates to the new external sort, which is not 
enabled by default, as it is still going through some additional testing.  If 
you’d like to try it to see if it resolves your problem, you’ll need to 
set “sort.external.disable_managed” as follows  in your drill-override.conf 
file:

drill.exec: {
  cluster-id: "drillbits1",
  zk.connect: "localhost:2181",
  sort.external.disable_managed: false
}

and run the following query:

ALTER SESSION SET `exec.sort.disable_managed` = false;

-- Zelaine

On 5/1/17, 7:44 AM, "Nate Butler" <n...@chartbeat.com> wrote:

    We keep running into this issue when trying to issue a query with hashagg
    disabled. When I look at system memory usage though, drill doesn't seem to
    be using much of it but still hits this error.
    
    Our environment:
    
    - 1 r3.8xl
    - 1 drillbit version 1.10.0 configured with 4GB of Heap and 230G of Direct
    - Data stored on S3 is compressed CSV
    
    I've tried increasing planner.memory.max_query_memory_per_node to 230G and
    lowered planner.width.max_per_query to 1 and it still fails.
    
    We've applied the patch from this bug in the hopes that it would resolve
    the issue but it hasn't:
    
    https://issues.apache.org/jira/browse/DRILL-5226
    
    Stack Trace:
    
      (org.apache.drill.exec.exception.OutOfMemoryException) Unable to allocate
    buffer of size 16777216 due to memory limit. Current allocation: 8445952
        org.apache.drill.exec.memory.BaseAllocator.buffer():220
        org.apache.drill.exec.memory.BaseAllocator.buffer():195
        org.apache.drill.exec.vector.VarCharVector.reAlloc():425
        org.apache.drill.exec.vector.VarCharVector.copyFromSafe():278
        org.apache.drill.exec.vector.NullableVarCharVector.copyFromSafe():379
    
    org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.doCopy():22
        org.apache.drill.exec.test.generated.PriorityQueueCopierGen328.next():75
    
    
org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.mergeAndSpill():602
    
    org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext():428
        org.apache.drill.exec.record.AbstractRecordBatch.next():162
        org.apache.drill.exec.record.AbstractRecordBatch.next():119
        org.apache.drill.exec.record.AbstractRecordBatch.next():109
    
    
org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.innerNext():137
        org.apache.drill.exec.record.AbstractRecordBatch.next():162
        org.apache.drill.exec.physical.impl.BaseRootExec.next():104
    
    
org.apache.drill.exec.physical.impl.partitionsender.PartitionSenderRootExec.innerNext():144
        org.apache.drill.exec.physical.impl.BaseRootExec.next():94
        org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():232
        org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():226
        java.security.AccessController.doPrivileged():-2
        javax.security.auth.Subject.doAs():422
        org.apache.hadoop.security.UserGroupInformation.doAs():1657
        org.apache.drill.exec.work.fragment.FragmentExecutor.run():226
        org.apache.drill.common.SelfCleaningRunnable.run():38
        java.util.concurrent.ThreadPoolExecutor.runWorker():1142
        java.util.concurrent.ThreadPoolExecutor$Worker.run():617
        java.lang.Thread.run():745 (state=,code=0)
    
    Is there something I'm missing here? Any help/direction would be
    appreciated.
    
    Thanks,
    Nate