[
https://issues.apache.org/jira/browse/HIVE-30224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30224:
----------------------------------
Description:
HIVE-30107 and HIVE-30118 cover a GROUP BY on a single string key. A single
LONG key with more groups than the specialized wrappers handle well, a DECIMAL
key, and keys of several columns
(most TPC-DS GROUP BYs) still take the general path: a VectorHashKeyWrapper
per row and a java.util.HashMap to aggregation buffer objects.
*Fix*
* A single LONG key is looked up by value in VectorGroupByBytesKeyTable, with
no key bytes.
* Two or more LONG and BYTES key columns are serialized a column at a time by
VectorGroupByKeySerializer and looked up as bytes. A LONG value that fits in an
int takes 4 bytes.
* A DECIMAL key of at most 18 digits is serialized as its decimal64 value at
the column's scale. Wider DECIMAL keys keep the general key wrappers.
* These modes also keep their aggregation state in
VectorGroupByAggregationColumns (HIVE-30118). Grouping sets and other key types
keep the wrapper path.
* From 32768 entries on, a slot also holds the hash bits above the slot
index, so a probe skips the records of most other keys. The table switches
between batches; smaller tables keep the
plain lookup. The single bytes key and serialized key modes use the tagged
slots; single LONG key tables keep plain slots.
* ProcessingModeHashAggregateKeyTable holds the table, the aggregation
columns, the flush and the memory estimate; its subclasses find the entries and
write the keys.
*Measured*
Local JMH (VectorGroupByMultiKeyBench; shapes are one letter per key column,
L = BIGINT, S = STRING):
* 1.3-3.9x overall.
* 412 groups: L 16.1 -> 9.2 ns/row, SS 75.8 -> 29.1 ns/row.
* 5M groups: L 252 -> 74 ns/row.
* Tagged slots on two wide BIGINT keys: -10 to -14% at 1M groups, -3 to -9%
at 5M groups.
* TPC-DS q75's reducer-side partial GROUP BY (five int keys and a
decimal(8,2)): ~250 -> 50-64 ns/row.
Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had two
other changes on, the decimal64 LazyBinary conversion (HIVE-30225) and the
parallel loading of a task's map-join hash tables (HIVE-30226), so these
figures are for the three together:
* -135.4 s over 26 queries.
* q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
Depends on HIVE-30118
was:
HIVE-30107 and HIVE-30118 cover a GROUP BY on a single string key. A single
LONG key with more groups than the specialized wrappers handle well, a DECIMAL
key, and keys of several columns
(most TPC-DS GROUP BYs) still take the general path: a VectorHashKeyWrapper
per row and a java.util.HashMap to aggregation buffer objects.
*Fix*
* A single LONG key is looked up by value in VectorGroupByBytesKeyTable, with
no key bytes.
* Two or more LONG and BYTES key columns are serialized a column at a time by
VectorGroupByKeySerializer and looked up as bytes. A LONG value that fits in an
int takes 4 bytes.
* A DECIMAL key of at most 18 digits is serialized as its decimal64 value at
the column's scale. Wider DECIMAL keys keep the general key wrappers.
* These modes also keep their aggregation state in
VectorGroupByAggregationColumns (HIVE-30118). Grouping sets and other key types
keep the wrapper path.
* From 32768 entries on, a slot also holds the hash bits above the slot
index, so a probe skips the records of most other keys. The table switches
between batches; smaller tables keep the
plain lookup. The single bytes key and serialized key modes use the tagged
slots; single LONG key tables keep plain slots.
* ProcessingModeHashAggregateKeyTable holds the table, the aggregation
columns, the flush and the memory estimate; its subclasses find the entries and
write the keys.
*Measured*
Local JMH (VectorGroupByMultiKeyBench; shapes are one letter per key column,
L = BIGINT, S = STRING):
* 1.3-3.9x overall.
* 412 groups: L 16.1 -> 9.2 ns/row, SS 75.8 -> 29.1 ns/row.
* 5M groups: L 252 -> 74 ns/row.
* Tagged slots on two wide BIGINT keys: -10 to -14% at 1M groups, -3 to -9%
at 5M groups.
* TPC-DS q75's reducer-side partial GROUP BY (five int keys and a
decimal(8,2)): ~250 -> 50-64 ns/row.
Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had two
other changes on, the decimal64 LazyBinary conversion and the parallel loading
of a task's map-join hash tables, so these figures are for the three together:
* -135.4 s over 26 queries.
* q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
Depends on HIVE-30118
> Look up LONG, DECIMAL and multi-column GROUP BY keys in the open-addressing
> hash table
> --------------------------------------------------------------------------------------
>
> Key: HIVE-30224
> URL: https://issues.apache.org/jira/browse/HIVE-30224
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
> Labels: performance
>
> HIVE-30107 and HIVE-30118 cover a GROUP BY on a single string key. A single
> LONG key with more groups than the specialized wrappers handle well, a
> DECIMAL key, and keys of several columns
> (most TPC-DS GROUP BYs) still take the general path: a VectorHashKeyWrapper
> per row and a java.util.HashMap to aggregation buffer objects.
>
>
>
>
> *Fix*
>
>
> * A single LONG key is looked up by value in VectorGroupByBytesKeyTable,
> with no key bytes.
>
> * Two or more LONG and BYTES key columns are serialized a column at a time
> by VectorGroupByKeySerializer and looked up as bytes. A LONG value that fits
> in an int takes 4 bytes.
> * A DECIMAL key of at most 18 digits is serialized as its decimal64 value
> at the column's scale. Wider DECIMAL keys keep the general key wrappers.
>
> * These modes also keep their aggregation state in
> VectorGroupByAggregationColumns (HIVE-30118). Grouping sets and other key
> types keep the wrapper path.
> * From 32768 entries on, a slot also holds the hash bits above the slot
> index, so a probe skips the records of most other keys. The table switches
> between batches; smaller tables keep the
> plain lookup. The single bytes key and serialized key modes use the tagged
> slots; single LONG key tables keep plain slots.
>
> * ProcessingModeHashAggregateKeyTable holds the table, the aggregation
> columns, the flush and the memory estimate; its subclasses find the entries
> and write the keys.
>
>
>
> *Measured*
>
>
> Local JMH (VectorGroupByMultiKeyBench; shapes are one letter per key
> column, L = BIGINT, S = STRING):
>
> * 1.3-3.9x overall.
>
>
> * 412 groups: L 16.1 -> 9.2 ns/row, SS 75.8 -> 29.1 ns/row.
>
>
> * 5M groups: L 252 -> 74 ns/row.
>
>
> * Tagged slots on two wide BIGINT keys: -10 to -14% at 1M groups, -3 to -9%
> at 5M groups.
>
> * TPC-DS q75's reducer-side partial GROUP BY (five int keys and a
> decimal(8,2)): ~250 -> 50-64 ns/row.
>
> Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had
> two other changes on, the decimal64 LazyBinary conversion (HIVE-30225) and
> the parallel loading of a task's map-join hash tables (HIVE-30226), so these
> figures are for the three together:
>
> * -135.4 s over 26 queries.
>
>
> * q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
>
> Depends on HIVE-30118
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
