Mihaly Szjatinya created IMPALA-15763:
-----------------------------------------
Summary: Store non-NDV column statistics of Iceberg tables in
Puffin files
Key: IMPALA-15763
URL: https://issues.apache.org/jira/browse/IMPALA-15763
Project: IMPALA
Issue Type: New Feature
Reporter: Mihaly Szjatinya
Assignee: Mihaly Szjatinya
IMPALA-15004 makes {{COMPUTE STATS}} write the column NDVs of Iceberg tables as
standard {{apache-datasketches-theta-v1}} blobs in Puffin statistics files. The
other column statistics are still stored only in the Hive Metastore:
* number of NULLs
* max and average length (STRING-like columns)
* number of TRUE / FALSE values (BOOLEAN)
To keep table statistics without HMS, these need to live with the table
metadata too. The Iceberg spec has no standard blob type for them and no other
engine writes them, so this needs an Impala-specific blob type (e.g.
{{impala-column-stats-v1}}), written next to the theta blobs and ignored by
other engines. Row count and total size need no blob: they come from the
Iceberg snapshot summary.
Open questions:
* Payload format and versioning of the blob.
* Where to write stats when both HMS and Puffin are available (HMS only /
Puffin only / both), e.g. via a query option. In the "both" case the HMS NDV
could be derived from the theta sketch, so the table is scanned only once.
* How the reader picks between HMS and Puffin stats per column (see
IMPALA-14261).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
