On Wed, Oct 21, 2026 at 3:12 PM Yuya Ebihara <[email protected]> wrote: > > Hi everyone, > > I opened a PR to add a standard HyperLogLog (HLL) blob type to the Puffin > spec and would like to hear your thoughts before it moves forward. > > PR: https://github.com/apache/iceberg/pull/18513 > Issue: https://github.com/apache/iceberg/issues/18516 > > Motivation > Puffin currently defines only one blob type for NDV statistics, > apache-datasketches-theta-v1. In some workloads, Theta sketch computation is > slow. In Trino, it causes a 10-20% slowdown on inserts. Trino is considering > an HLL option (https://github.com/trinodb/trino/pull/31173). Since the spec > has no standard HLL type, that PR has to store the sketch as an > engine-specific blob type. That means other engines can’t read it.
+1, HLL consumes less memory and faster for simple distinct count. > > Proposal > Add a new blob type, apache-datasketches-hll-v1: > > It holds a serialized “compact” Apache DataSketches HLL sketch. > The sketch is built with the default seed. Individual distinct values are fed > to it using Iceberg’s single-value serialization, the same as for the Theta > type. > The blob metadata may include an ndv property, defined the same way as for > the Theta type. > > Feedback on the PR or in this thread is welcome. > > Thanks, > Yuya -- Regards Junwang Zhao
