[
https://issues.apache.org/jira/browse/HIVE-30119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30119:
----------------------------------
Description:
A Parquet string column is mostly dictionary-encoded: a column chunk's rows
reference a dictionary of distinct values by id. The reader decodes each id to
its bytes, and a single string key
GROUP BY then hashes and compares those bytes again for every row, although
rows with the same id belong to the same group.
*Fix*
* BytesColumnVector carries the dictionary ids of its rows, the dictionary's
token and its size. reset() clears the token, and shallowCopyTo() copies the
three fields.
* The Parquet reader sets them for a string batch whose rows all came from
dictionary pages: each row's id, and a token minted per column chunk. A batch
with a PLAIN-encoded row carries
none.
* The LLAP decode thread hands the vector over unchanged.
* The single string key hash aggregation looks up an id's entry by value once
per dictionary and flush, and maps the id's later rows through a memo. The memo
is dropped on every flush,
because a flush removes, renumbers or clears entries.
*Measured*
* Cluster (1TRC part 1, 64 x r5d): 99.6 -> 68.0 s (-32%), results identical
on 1T rows.
* Local: 11.01 -> 6.82 ns/row.
Depends on HIVE-30118 and on the hash GROUP BY on LONG, DECIMAL and
multi-column keys through the bytes key table (JIRA to be opened), which gives
the key table mode the structure the memo
hooks into.
> Group dictionary-encoded string keys by dictionary id
> -------------------------------------------------------
>
> Key: HIVE-30119
> URL: https://issues.apache.org/jira/browse/HIVE-30119
> Project: Hive
> Issue Type: Sub-task
> Reporter: Denys Kuzmenko
> Priority: Major
>
> A Parquet string column is mostly dictionary-encoded: a column chunk's rows
> reference a dictionary of distinct values by id. The reader decodes each id
> to its bytes, and a single string key
> GROUP BY then hashes and compares those bytes again for every row, although
> rows with the same id belong to the same group.
>
>
>
>
> *Fix*
>
>
> * BytesColumnVector carries the dictionary ids of its rows, the
> dictionary's token and its size. reset() clears the token, and
> shallowCopyTo() copies the three fields.
> * The Parquet reader sets them for a string batch whose rows all came from
> dictionary pages: each row's id, and a token minted per column chunk. A batch
> with a PLAIN-encoded row carries
> none.
>
>
> * The LLAP decode thread hands the vector over unchanged.
>
>
> * The single string key hash aggregation looks up an id's entry by value
> once per dictionary and flush, and maps the id's later rows through a memo.
> The memo is dropped on every flush,
> because a flush removes, renumbers or clears entries.
>
>
>
>
>
> *Measured*
>
>
> * Cluster (1TRC part 1, 64 x r5d): 99.6 -> 68.0 s (-32%), results identical
> on 1T rows.
>
> * Local: 11.01 -> 6.82 ns/row.
> Depends on HIVE-30118 and on the hash GROUP BY on LONG, DECIMAL and
> multi-column keys through the bytes key table (JIRA to be opened), which
> gives the key table mode the structure the memo
> hooks into.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
