Hi all, I would like to get feedback on the design direction for exposing Puffin file references through Iceberg metadata tables.
Related issue and PRs: - https://github.com/apache/iceberg/issues/17339 - https://github.com/apache/iceberg/pull/17192 - https://github.com/apache/iceberg/pull/17282 Motivation: Iceberg stores multiple kinds of auxiliary metadata in Puffin files today, including registered table statistics and deletion vectors. These references are discoverable through different metadata paths, which makes it hard to answer questions like: - Which Puffin files are referenced by a snapshot? - Which references come from statistics vs deletion vectors? - Which retained snapshots reference the same physical Puffin file? - Which blob types and fields are associated with these references? The proposed metadata tables are: - `puffin_files`: Puffin file references for one selected snapshot - `all_puffin_files`: Puffin file references for all retained snapshots Each row represents the Puffin blob references for one tuple of snapshot ID, metadata source, and physical Puffin file path. The initial sources are `statistics` and `deletion_vector`. The implementation is intentionally limited: - It does not open Puffin files. - It does not read Puffin footers. - It does not parse blob payloads. - It does not expose blobs that are not referenced by Iceberg metadata. - It does not identify orphan files or files safe to delete. A natural follow-up would be a separate Puffin blob metadata table that opens Puffin files and exposes footer-level blob metadata, such as blob type, fields, snapshot ID, sequence number, properties, and byte ranges. I would like to keep that out of the current PRs so that the first step only establishes snapshot-level Puffin reference discovery. The main design question is naming and scope. Review feedback raised whether this should be modeled as statistics-specific metadata or as a more general Puffin-reference table. My preference is to model the row as a Puffin file reference, with a `source` column such as `statistics` or `deletion_vector`, because this keeps the table useful for current Puffin-backed metadata and future Puffin-backed features such as skipping indexes. I would appreciate feedback on: 1. Whether `puffin_files` / `all_puffin_files` is the right abstraction; 2. Whether a broader name like `metadata_blob_files` would be preferable; 3. Whether the footer-level Puffin blob metadata should remain a separate follow-up, as proposed above. Thanks, Shangqing GitHub: @yangshangqing95
