[
https://issues.apache.org/jira/browse/HIVE-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30118:
----------------------------------
Description:
After HIVE-30107 the keys of a single string key GROUP BY are looked up in an
open-addressing table, but each group still keeps an aggregation buffer object
per aggregate, and every row
updates them through a call per aggregate.
*Fix*
* VectorGroupByAggregationColumns keeps the aggregation state of the table's
entries in one primitive array per aggregate, indexed by entry, plus each
entry's access count. A batch is
aggregated with one loop per aggregate over the entries of its rows.
* Covered aggregates: min, max and sum of LONG and DECIMAL_64, count and
count(*). Their results match the vectorized aggregates they replace: min, max
and sum of a group with no non-NULL
value are NULL, count is 0, and a DECIMAL_64 sum that exceeds its output
precision at any row is NULL.
* The columns are used when they cover every aggregate of the GROUP BY;
otherwise the aggregation buffers stay.
* The table moves the columns' state along when it compacts.
*Measured* (over HIVE-30107)
* Cluster (1TRC part 1, 64 x r5d): 160.3 -> 120.5 s (-25%).
* Local (VectorGroupByBytesKeyBench): 412 keys 13.3 -> 9.6 ns/row; 5M keys
247 -> 146 ns/row.
Depends on HIVE-30107
was:
After HIVE-30107 the keys of a single string key GROUP BY are looked up in an
open-addressing table, but each group still keeps an aggregation buffer object
per aggregate, and every row
updates them through a call per aggregate.
*Fix*
* VectorGroupByAggregationColumns keeps the aggregation state of the table's
entries in one primitive array per aggregate, indexed by entry, plus each
entry's access count. A batch is
aggregated with one loop per aggregate over the entries of its rows.
* Covered aggregates: min, max and sum of LONG and DECIMAL_64, count and
count(*). Their results match the vectorized aggregates they replace: min, max
and sum of a group with no non-NULL
value are NULL, count is 0, and a DECIMAL_64 sum that exceeds its output
precision at any row is NULL.
* The columns are used when they cover every aggregate of the GROUP BY;
otherwise the aggregation buffers stay.
* The table moves the columns' state along when it compacts.
*Measured* (over HIVE-30107)
* Cluster (1TRC part 1, 64 x r5d): 160.3 -> 120.5 s (-25%).
* Local (VectorGroupByBytesKeyBench): 412 keys 13.3 -> 9.6 ns/row; 5M keys
247 -> 146 ns/row.
> Keep aggregation state in primitive arrays for a single string key
> ------------------------------------------------------------------
>
> Key: HIVE-30118
> URL: https://issues.apache.org/jira/browse/HIVE-30118
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> After HIVE-30107 the keys of a single string key GROUP BY are looked up in
> an open-addressing table, but each group still keeps an aggregation buffer
> object per aggregate, and every row
> updates them through a call per aggregate.
>
>
>
>
>
> *Fix*
>
>
> * VectorGroupByAggregationColumns keeps the aggregation state of the
> table's entries in one primitive array per aggregate, indexed by entry, plus
> each entry's access count. A batch is
> aggregated with one loop per aggregate over the entries of its rows.
>
>
> * Covered aggregates: min, max and sum of LONG and DECIMAL_64, count and
> count(*). Their results match the vectorized aggregates they replace: min,
> max and sum of a group with no non-NULL
> value are NULL, count is 0, and a DECIMAL_64 sum that exceeds its output
> precision at any row is NULL.
>
> * The columns are used when they cover every aggregate of the GROUP BY;
> otherwise the aggregation buffers stay.
>
> * The table moves the columns' state along when it compacts.
>
>
>
>
>
> *Measured* (over HIVE-30107)
>
>
> * Cluster (1TRC part 1, 64 x r5d): 160.3 -> 120.5 s (-25%).
>
>
> * Local (VectorGroupByBytesKeyBench): 412 keys 13.3 -> 9.6 ns/row; 5M keys
> 247 -> 146 ns/row.
> Depends on HIVE-30107
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
