Hi everyone, I opened a PR to add a standard HyperLogLog (HLL) blob type to the Puffin spec and would like to hear your thoughts before it moves forward.
PR: https://github.com/apache/iceberg/pull/18513 Issue: https://github.com/apache/iceberg/issues/18516 *Motivation* Puffin currently defines only one blob type for NDV statistics, apache-datasketches-theta-v1. In some workloads, Theta sketch computation is slow. In Trino, it causes a 10-20% slowdown on inserts. Trino is considering an HLL option (https://github.com/trinodb/trino/pull/31173). Since the spec has no standard HLL type, that PR has to store the sketch as an engine-specific blob type. That means other engines can’t read it. *Proposal* Add a new blob type, apache-datasketches-hll-v1: - It holds a serialized “compact” Apache DataSketches HLL sketch. - The sketch is built with the default seed. Individual distinct values are fed to it using Iceberg’s single-value serialization, the same as for the Theta type. - The blob metadata may include an ndv property, defined the same way as for the Theta type. Feedback on the PR or in this thread is welcome. Thanks, Yuya
