[
https://issues.apache.org/jira/browse/IMPALA-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mihaly Szjatinya resolved IMPALA-15459.
---------------------------------------
Resolution: Fixed
> Add Iceberg-encoded theta sketch flavor for Puffin NDV stats
> ------------------------------------------------------------
>
> Key: IMPALA-15459
> URL: https://issues.apache.org/jira/browse/IMPALA-15459
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Mihaly Szjatinya
> Assignee: Mihaly Szjatinya
> Priority: Major
> Labels: datasketches, impala-iceberg
>
> COMPUTE STATS will write per-column NDV statistics as Iceberg Puffin
> "apache-datasketches-theta-v1" blobs (IMPALA-15004). For those blobs to
> interoperate with other engines (Trino, Spark), each value must be hashed
> into the theta sketch using Iceberg's single-value serialization:
> https://iceberg.apache.org/spec/#appendix-d-single-value-serialization
> Impala's existing {{ds_theta_sketch()}} feeds values through the Apache
> DataSketches typed {{update()}} overloads, which widen INT to 8 bytes and
> FLOAT to a double. This matches Hive's datasketches UDF, but not the Iceberg
> Puffin spec. It also does not support BOOLEAN, DATE, TIMESTAMP and DECIMAL at
> all. Changing {{ds_theta_sketch()}} is not an option, as it would break
> byte-level interop with Hive-produced sketches.
> This issue adds an internal (undocumented) aggregate flavor,
> {{ds_theta_sketch_iceberg()}}, that hashes every Iceberg column type Impala
> computes NDV for with the Appendix D encoding. COMPUTE STATS uses it when
> writing Puffin NDV blobs; the user-facing {{ds_theta_sketch()}} is unchanged.
> ||Type||ds_theta_sketch()||ds_theta_sketch_iceberg()||
> |BOOLEAN|not supported|1 byte|
> |INT|widened to 8 bytes|4-byte LE|
> |BIGINT|8-byte LE|8-byte LE (same)|
> |FLOAT|widened to double|4-byte LE|
> |DOUBLE|8-byte LE|8-byte LE (same)|
> |STRING|raw bytes|raw bytes (same)|
> |DATE|not supported|4-byte LE day count|
> |TIMESTAMP|not supported|8-byte LE micros since epoch|
> |DECIMAL|not supported|unscaled, big-endian two's-complement, minimum bytes|
> TINYINT and SMALLINT are not registered, as Iceberg has no such column types.
> TIMESTAMP is the only lossy conversion (floored to microseconds), which does
> not affect Iceberg, whose timestamps have microsecond precision.
> This is a prerequisite for IMPALA-15004.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
