Joe McDonnell created IMPALA-15089:
--------------------------------------
Summary: Multiple aggregation tuple size is overestimated
Key: IMPALA-15089
URL: https://issues.apache.org/jira/browse/IMPALA-15089
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
For a query with multiple aggregations (e.g. TPC-DS Q67), the streaming
aggregation is returning multiple tuples, but at most one will be set. However,
the row size calculations simply add up all the individual tuple sizes and will
overestimate the row size. This interacts with the exchange sender to limit the
number of rows included in each message.
For example, on TPC-DS Q67, we have this streaming aggregation node:
{noformat}
07:AGGREGATE [STREAMING]
| Class 0
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
d_moy, s_store_id
| Class 1
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
d_moy, NULL
| Class 2
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy,
NULL, NULL
| Class 3
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, i_product_name, d_year, NULL,
NULL, NULL
| Class 4
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, i_product_name, NULL, NULL, NULL,
NULL
| Class 5
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, i_brand, NULL, NULL, NULL, NULL, NULL
| Class 6
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, i_class, NULL, NULL, NULL, NULL, NULL, NULL
| Class 7
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: i_category, NULL, NULL, NULL, NULL, NULL, NULL, NULL
| Class 8
| output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)),
CAST(0 AS DECIMAL(18,2))))
| group by: NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
| mem-estimate=3.14GB mem-reservation=211.00MB thread-reservation=0
| tuple-ids=4N,5N,6N,7N,8N,9N,10N,11N,12N row-size=1.07KB cardinality=11.63M
| in pipelines: 00(GETNEXT){noformat}
The row size of 1.07KB is overestimate what actually gets sent through the
subsequent exchange. More accurately estimating the row size for the exchange
could help it size the messages appropriately.
However, AggregationNode also uses the row size to estimate the size of its
hash tables. For that, it does need to know about all the tuples, because it
will have a hash table for each tuple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
