tanmayrauth opened a new issue, #1929:
URL: https://github.com/apache/iceberg-go/issues/1929
### Feature Request / Improvement
## Proposal: Table statistics computation (NDV via Theta sketches)
### Motivation
iceberg-go can *read* and *track* `StatisticsFile` metadata, but it cannot
**compute** or **write** it. Tables created or maintained by iceberg-go
therefore carry no NDV (number-of-distinct-values) statistics, so query engines
that use them for cost-based planning (join ordering, cardinality estimation)
get nothing from Go-managed tables. This proposes the missing **producer**,
closing a parity gap with the Java implementation.
### Current state (what already exists)
Modeling and lifecycle are in place — only the producer is missing:
- `StatisticsFile` / `BlobMetadata` types and the
`apache-datasketches-theta-v1` blob-type constant — `table/statistics.go`,
`puffin/puffin.go`
- Metadata registration — `SetStatisticsUpdate` / `RemoveStatisticsUpdate`
(`table/updates.go`), builder support in `table/metadata.go`
- Lifecycle — statistics pruned on `RemoveSnapshots`, deleted on
`ExpireSnapshots`, included in orphan-file cleanup
What is missing: nothing computes a Theta sketch, and the only Puffin
*write* path that emits a blob today is the deletion-vector writer
(`table/dv/dv_writer.go`). There is no "analyze this snapshot → produce a
`StatisticsFile`" entry point.
### Goals
- Compute per-column NDV for a snapshot as Theta sketches.
- Serialize sketches in a form **byte-compatible with Apache
DataSketches**, so blobs written by iceberg-go are readable by Java/Spark and
vice versa.
- Write a valid Puffin `StatisticsFile` (correct `snapshot-id` /
`sequence-number` footer) and register it via the existing `SetStatistics`
update.
- Expose a CLI entry point to compute statistics for a table/snapshot.
### Non-goals
- **Partition statistics computation** (a
`PartitionStatsHandler`-equivalent
producer) — related but separate; tracked as a companion issue.
- Additional sketch types beyond Theta/NDV (e.g. histograms, HLL).
- Engine-side *consumption* of NDV for query planning.
### Design sketch
1. **Theta sketch core** — pure-Go sketch (`Update` / `Union` /
`Estimate`), no Iceberg dependencies.
2. **DataSketches-compatible serialization** — compact serialized form
with Java-produced fixtures as the parity oracle (the correctness-critical
piece).
3. **NDV compute pass** — scan a snapshot's data files, feed values into
per-field-ID sketches, emit theta blob bytes keyed by field ID.
4. **Puffin `StatisticsFile` writer + registration** — assemble blobs into
a Puffin file and commit through `SetStatistics`.
5. **CLI + docs** — a `compute-stats`-style command and reference docs.
### Reference (Apache Iceberg Java)
- Puffin theta blob + `StandardBlobTypes` (`core/.../puffin/`)
- `apache-datasketches` Theta sketch usage for NDV
- Statistics-file assembly / registration in the write path
### Proposed PR breakdown (6 PRs )
1. `feat(table): add Theta sketch core (update/union/estimate)` — pure
algorithm + error-bound tests.
2. `feat(table): DataSketches-compatible Theta sketch serialization` —
compact encode/decode + Java-fixture byte cross-check.
3. `feat(table): compute per-column NDV over a snapshot` — scan →
per-field sketches → blob bytes.
4. `feat(table): write StatisticsFile and register via SetStatistics` —
Puffin write + commit (end-to-end).
5. `feat(cli): add compute-stats command` — CLI wired to (4), text/JSON
output.
6. `docs(website): document table statistics` + read-side surface — docs
and a compute→reload→verify integration test.
### Open questions
**Theta sketch serialization.** The blob must be byte-compatible with
Apache DataSketches so Java/Spark can read it. I don't see a maintained Go port
of the compact Theta format, so my lean is to implement the compact
serialization in-tree (bounded, and spec'd by DataSketches) and validate
byte-for-byte against
Java-produced fixtures — the same approach the DV code uses against Java.
Flagging in case there's a preferred dependency I should use instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
