[
https://issues.apache.org/jira/browse/DRILL-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150876#comment-16150876
]
Sorabh Hamirwasia commented on DRILL-5737:
------------------------------------------
With this patch there will be failures in functional test. Sharing the result
below. Reason being this patch by default will enforce memory limit on HashAgg
i.e. if there is not much memory it won't let HashAgg to fallback to old
behavior of using unbounded memory. Below queries earlier were running with
unbounded memory and were hence passing. After this patch is submitted
[[email protected]] will help to update the test's to turn on the option for
HashAgg to fallback to old behavior of using unbounded memory which will again
pass the functional tests below.
*Execution Failures:*
/root/drillAutomation/mapr/framework/resources/Functional/aggregates/tpcds_variants/text/aggregate25.q
/root/drillAutomation/mapr/framework/resources/Functional/aggregates/tpcds_variants/text/aggregate27.q
/root/drillAutomation/mapr/framework/resources/Functional/tpcds/impala/text/q98.q
/root/drillAutomation/mapr/framework/resources/External/mapr-drill-tests/Testcases/customer/transunion/data/tu_7.q
/root/drillAutomation/mapr/framework/resources/Functional/tpcds/impala/parquet/q89.q
/root/drillAutomation/mapr/framework/resources/External/mapr-drill-tests/Testcases/customer/transunion/data/tu_1.q
/root/drillAutomation/mapr/framework/resources/External/mapr-drill-tests/Testcases/customer/transunion/data/tu_2.q
/root/drillAutomation/mapr/framework/resources/Functional/tpcds/impala/json/q89.q
/root/drillAutomation/mapr/framework/resources/Functional/complex/parquet/complex27.q
/root/drillAutomation/mapr/framework/resources/External/mapr-drill-tests/Testcases/customer/fishbowl/data/fbQuery13.q
/root/drillAutomation/mapr/framework/resources/Functional/aggregates/tpcds_variants/text/aggregate26.q
/root/drillAutomation/mapr/framework/resources/External/mapr-drill-tests/Testcases/customer/transunion/data/tu_5.q
/root/drillAutomation/mapr/framework/resources/Functional/tpcds/impala/text/q89.q
----------------------------------------------------------------------------------------------------------------
*Verification Failures:*
/root/drillAutomation/mapr/framework/resources/Functional/partition_pruning/hive/general/data/orc2.q
/root/drillAutomation/mapr/framework/resources/Functional/partition_pruning/hive/general/data/orc1.q
/root/drillAutomation/mapr/framework/resources/Functional/partition_pruning/hive/general/data/orc3.q
*Sample Failure:*
EXECUTION_FAILURE (4.192 s)
/root/drillAutomation/mapr/framework/resources/Functional/aggregates/tpcds_variants/text/aggregate25.q
(connection: 620065054)
[#23510] Query failed:
oadd.org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR:
Not enough memory for internal partitioning and fallback mechanism for HashAgg
to use unbounded memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
> Hash Agg uses more than the allocated memory under certain low memory
> conditions
> --------------------------------------------------------------------------------
>
> Key: DRILL-5737
> URL: https://issues.apache.org/jira/browse/DRILL-5737
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Sorabh Hamirwasia
> Assignee: Sorabh Hamirwasia
> Labels: ready-to-commit
> Fix For: 1.12.0
>
>
> Reported by [~rkins]
> Based on the memory computations drill thinks that there is not sufficient
> memory and falls back to a single partition case. The single partition case
> however does not respect the memory constraints imposed and completes the
> query using ~130MB of memory
> {code:java}
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.memory.max_query_memory_per_node` = 117127360;
> select count(*) from (select max(nulls_col), max(length(nulls_col)),
> max(`filename`) from dfs.`/drill/testdata/hash-agg/data1` group by
> no_nulls_col) d;
> {code}
> Based on analysis by [~ben-zvi] this is by design. When the Hash Aggr Op
> finds that there is not enough memory for at least two partitions, it falls
> back to the pre 1.11 behavior ( using 10GB limit ).
> Solution is to provide a configuration based on which the fallback will be
> either allowed or query will be failed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
