[
https://issues.apache.org/jira/browse/IMPALA-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe McDonnell updated IMPALA-15089:
-----------------------------------
Summary: Multiple aggregation row size is overestimated (was: Multiple
aggregation tuple size is overestimated)
> Multiple aggregation row size is overestimated
> ----------------------------------------------
>
> Key: IMPALA-15089
> URL: https://issues.apache.org/jira/browse/IMPALA-15089
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For a query with multiple aggregations (e.g. TPC-DS Q67), the streaming
> aggregation is returning multiple tuples, but at most one will be set.
> However, the row size calculations simply add up all the individual tuple
> sizes and will overestimate the row size. This interacts with the exchange
> sender to limit the number of rows included in each message.
> For example, on TPC-DS Q67, we have this streaming aggregation node:
> {noformat}
> 07:AGGREGATE [STREAMING]
> | Class 0
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
> d_moy, s_store_id
> | Class 1
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
> d_moy, NULL
> | Class 2
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
> NULL, NULL
> | Class 3
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, NULL,
> NULL, NULL
> | Class 4
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, NULL, NULL,
> NULL, NULL
> | Class 5
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, NULL, NULL, NULL, NULL, NULL
> | Class 6
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 7
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 8
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS
> DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | mem-estimate=3.14GB mem-reservation=211.00MB thread-reservation=0
> | tuple-ids=4N,5N,6N,7N,8N,9N,10N,11N,12N row-size=1.07KB cardinality=11.63M
> | in pipelines: 00(GETNEXT){noformat}
> The row size of 1.07KB is overestimate what actually gets sent through the
> subsequent exchange. More accurately estimating the row size for the exchange
> could help it size the messages appropriately.
> However, AggregationNode also uses the row size to estimate the size of its
> hash tables. For that, it does need to know about all the tuples, because it
> will have a hash table for each tuple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
