[
https://issues.apache.org/jira/browse/DRILL-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185201#comment-16185201
]
ASF GitHub Bot commented on DRILL-5824:
---------------------------------------
GitHub user Ben-Zvi opened a pull request:
https://github.com/apache/drill/pull/966
DRILL-5824: Retain original memory limit for 1st phase HashAgg with 1
partition
The First Phase Hash Aggr always "fell back" to 1.10 behaviour (i.e.
memory limit up to 10GB) when its number of partitions was 1. However the 1st
phase can always "spill" (i.e. early return some partition). Hence the code was
change for 1st phase (only operator that can spill with 1 partition) to retain
its original memory limit and not "fall back".
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ben-Zvi/drill DRILL-5824
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/966.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #966
----
commit 08f3a5150fdbc326a8e1be6e899f421859f09c37
Author: Ben-Zvi <[email protected]>
Date: 2017-09-29T00:09:53Z
Retain original memory limit for 1st phase HashAgg with 1 partition
----
> 1st phase Hash Aggregate allocates more memory than the limit
> -------------------------------------------------------------
>
> Key: DRILL-5824
> URL: https://issues.apache.org/jira/browse/DRILL-5824
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.11.0
> Reporter: Boaz Ben-Zvi
> Assignee: Boaz Ben-Zvi
> Fix For: 1.12.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The per query memory limit was set to 2G. But the 1st phase hash agg operator
> memory limit set was larger than that:
> AGGR OOM at First Phase. Partitions: 32. Estimated batch size: 4784128.
> values size: 3670016. Output alloc size: 3670016. Planned batches: 1 Memory
> limit: 2680684544 so far allocated: 374341632.
> Fragment 3:0
> [Error Id: b22fe6ad-b805-433c-bae7-c0f60c30bb99 on 10.10.30.168:31010]
> (org.apache.drill.exec.exception.OutOfMemoryException) AGGR OOM at First
> Phase. Partitions: 32. Estimated batch size: 4784128. values size: 3670016.
> Output alloc size: 3670016. Planned batches: 1 Memory limit: 2680684544 so
> far allocated: 374341632.
> org.apache.drill.exec.test.generated.HashAggregatorGen5265.checkGroupAndAggrValues():1350
> org.apache.drill.exec.test.generated.HashAggregatorGen5265.doWork():591
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():169
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():51
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():141
> org.apache.drill.exec.record.AbstractRecordBatch.next():164
> org.apache.drill.exec.physical.impl.BaseRootExec.next():105
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():92
> org.apache.drill.exec.physical.impl.BaseRootExec.next():95
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():234
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():227
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():415
> org.apache.hadoop.security.UserGroupInformation.doAs():1595
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():227
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1145
> java.util.concurrent.ThreadPoolExecutor$Worker.run():615
> java.lang.Thread.run():745
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
