Mihaly Szjatinya created IMPALA-15459:
-----------------------------------------
Summary: Add Iceberg-encoded theta sketch flavor for Puffin NDV
stats
Key: IMPALA-15459
URL: https://issues.apache.org/jira/browse/IMPALA-15459
Project: IMPALA
Issue Type: New Feature
Components: Frontend, Backend
Reporter: Mihaly Szjatinya
Assignee: Mihaly Szjatinya
COMPUTE STATS will write per-column NDV statistics as Iceberg Puffin
"apache-datasketches-theta-v1" blobs (IMPALA-15004). For those blobs to
interoperate with other engines (Trino, Spark), each value must be hashed into
the theta sketch using Iceberg's single-value serialization:
https://iceberg.apache.org/spec/#appendix-d-single-value-serialization
(e.g. int = 4 bytes, float = 4 bytes, date = 4 bytes, boolean = 1 byte).
Impala's existing ds_theta_sketch() feeds values through the Apache
DataSketches typed update() overloads, which *widen* small values before
hashing: int -> 8 bytes, float -> double (8 bytes), date -> 8 bytes, boolean ->
8 bytes. This matches Hive's datasketches UDF (so Impala and Hive sketches can
be unioned), but it does NOT match the Iceberg Puffin spec, so sketches written
this way would not interoperate with Trino/Spark for INT, FLOAT, DATE and
BOOLEAN columns. (BIGINT, DOUBLE, STRING, and the TIMESTAMP/DECIMAL
specialisations added in IMPALA-15106 already encode identically to Appendix D.)
Changing ds_theta_sketch() directly is not an option: it would break byte-level
interop with Hive-produced sketches, which is covered by existing tests
(datasketches-theta.test unions/excludes Impala and Hive sketches and expects
identical results).
This issue introduces a second, internal-only aggregate flavor (working name
ds_theta_sketch_iceberg) that encodes values with Iceberg Appendix D
single-value serialization. COMPUTE STATS uses this flavor when writing Puffin
NDV blobs for Iceberg tables; the user-facing ds_theta_sketch() keeps its
current Hive-compatible encoding. Only BOOLEAN, INT, FLOAT and DATE need a
distinct implementation; the remaining Iceberg column types reuse the existing
ds_theta_sketch() update logic. TINYINT/SMALLINT are not registered for the new
flavor because Iceberg has no such column types.
This is a prerequisite for IMPALA-15004.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
