[
https://issues.apache.org/jira/browse/DRILL-5891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Hou updated DRILL-5891:
------------------------------
Description:
Query is:
select count(\*), max(`filename`) from dfs.`/drill/testdata/hash-agg/data1`
group by no_nulls_col, nulls_col;
Error is:
Error: RESOURCE ERROR: Not enough memory for internal partitioning and fallback
mechanism for HashAgg to use unbounded memory is disabled. Either enable
fallback config drill.exec.hashagg.fallback.enabled using Alter session/system
command or increase memory limit for Drillbit
>From drillbit.log:
{noformat}
2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
o.a.d.e.p.i.aggregate.HashAggregator - Incoming sizer: Actual batch schema &
sizes {
no_nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size:
130, data size: 132892)
nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size:
112, data size: 113673)
EXPR$0(type: REQUIRED BIGINT, count: 1023, std size: 8, actual size: 8, data
size: 8184)
EXPR$1(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 18,
data size: 18414)
Records: 1023, Total size: 524288, Data size: 273163, Gross row width: 513,
Net row width: 268, Density: 53%}
2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
o.a.d.e.p.i.aggregate.HashAggregator - 2nd phase. Estimated internal row width:
166 Values row width: 66 batch size: 12779520 memory limit: 63161283 max
column width: 50
2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:3:2] TRACE
o.a.d.e.p.impl.common.HashTable - HT allocated 4784128 for varchar of max width
50
2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:15] INFO
o.a.d.e.p.i.aggregate.HashAggregator - User Error Occurred: Not enough memory
for internal partitioning and fallback mechanism for HashAgg to use unbounded
memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Not enough
memory for internal partitioning and fallback mechanism for HashAgg to use
unbounded memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
{noformat}
I would recommend that we add a log message with the "alter" command to
increase the amount of memory allocated, and how much memory to allocate.
Otherwise, the user may not know what to do.
I would also not suggest enabling "drill.exec.hashagg.fallback.enabled" except
as a last resort.
was:
Query is:
select count(*), max(`filename`) from dfs.`/drill/testdata/hash-agg/data1`
group by no_nulls_col, nulls_col;
Error is:
Error: RESOURCE ERROR: Not enough memory for internal partitioning and fallback
mechanism for HashAgg to use unbounded memory is disabled. Either enable
fallback config drill.exec.hashagg.fallback.enabled using Alter session/system
command or increase memory limit for Drillbit
>From drillbit.log:
{noformat}
2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
o.a.d.e.p.i.aggregate.HashAggregator - Incoming sizer: Actual batch schema &
sizes {
no_nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size:
130, data size: 132892)
nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size:
112, data size: 113673)
EXPR$0(type: REQUIRED BIGINT, count: 1023, std size: 8, actual size: 8, data
size: 8184)
EXPR$1(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 18,
data size: 18414)
Records: 1023, Total size: 524288, Data size: 273163, Gross row width: 513,
Net row width: 268, Density: 53%}
2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
o.a.d.e.p.i.aggregate.HashAggregator - 2nd phase. Estimated internal row width:
166 Values row width: 66 batch size: 12779520 memory limit: 63161283 max
column width: 50
2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:3:2] TRACE
o.a.d.e.p.impl.common.HashTable - HT allocated 4784128 for varchar of max width
50
2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:15] INFO
o.a.d.e.p.i.aggregate.HashAggregator - User Error Occurred: Not enough memory
for internal partitioning and fallback mechanism for HashAgg to use unbounded
memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Not enough
memory for internal partitioning and fallback mechanism for HashAgg to use
unbounded memory is disabled. Either enable fallback config
drill.exec.hashagg.fallback.enabled using Alter session/system command or
increase memory limit for Drillbit
{noformat}
I would recommend that we add a log message with the "alter" command to
increase the amount of memory allocated, and how much memory to allocate.
Otherwise, the user may not know what to do.
I would also not suggest enabling "drill.exec.hashagg.fallback.enabled" except
as a last resort.
> When Drill runs out of memory for a HashAgg, it should tell the user how much
> memory to allocate
> ------------------------------------------------------------------------------------------------
>
> Key: DRILL-5891
> URL: https://issues.apache.org/jira/browse/DRILL-5891
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.11.0
> Reporter: Robert Hou
> Assignee: Pritesh Maker
>
> Query is:
> select count(\*), max(`filename`) from dfs.`/drill/testdata/hash-agg/data1`
> group by no_nulls_col, nulls_col;
> Error is:
> Error: RESOURCE ERROR: Not enough memory for internal partitioning and
> fallback mechanism for HashAgg to use unbounded memory is disabled. Either
> enable fallback config drill.exec.hashagg.fallback.enabled using Alter
> session/system command or increase memory limit for Drillbit
> From drillbit.log:
> {noformat}
> 2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
> o.a.d.e.p.i.aggregate.HashAggregator - Incoming sizer: Actual batch schema &
> sizes {
> no_nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual
> size: 130, data size: 132892)
> nulls_col(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size:
> 112, data size: 113673)
> EXPR$0(type: REQUIRED BIGINT, count: 1023, std size: 8, actual size: 8,
> data size: 8184)
> EXPR$1(type: OPTIONAL VARCHAR, count: 1023, std size: 54, actual size: 18,
> data size: 18414)
> Records: 1023, Total size: 524288, Data size: 273163, Gross row width: 513,
> Net row width: 268, Density: 53%}
> 2017-10-18 13:30:17,135 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:8] TRACE
> o.a.d.e.p.i.aggregate.HashAggregator - 2nd phase. Estimated internal row
> width: 166 Values row width: 66 batch size: 12779520 memory limit: 63161283
> max column width: 50
> 2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:3:2] TRACE
> o.a.d.e.p.impl.common.HashTable - HT allocated 4784128 for varchar of max
> width 50
> 2017-10-18 13:30:17,139 [26184629-3f4c-856a-e99e-97cdf0d29321:frag:1:15] INFO
> o.a.d.e.p.i.aggregate.HashAggregator - User Error Occurred: Not enough
> memory for internal partitioning and fallback mechanism for HashAgg to use
> unbounded memory is disabled. Either enable fallback config
> drill.exec.hashagg.fallback.enabled using Alter session/system command or
> increase memory limit for Drillbit
> org.apache.drill.common.exceptions.UserException: RESOURCE ERROR: Not enough
> memory for internal partitioning and fallback mechanism for HashAgg to use
> unbounded memory is disabled. Either enable fallback config
> drill.exec.hashagg.fallback.enabled using Alter session/system command or
> increase memory limit for Drillbit
> {noformat}
> I would recommend that we add a log message with the "alter" command to
> increase the amount of memory allocated, and how much memory to allocate.
> Otherwise, the user may not know what to do.
> I would also not suggest enabling "drill.exec.hashagg.fallback.enabled"
> except as a last resort.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
