yangshangqing95 opened a new pull request, #17282:
URL: https://github.com/apache/iceberg/pull/17282
## Summary
This PR adds a new `all_puffin_files` metadata table that exposes Puffin
files associated with all snapshots currently tracked in Iceberg table metadata.
The metadata table reports Puffin files referenced through two metadata
sources:
- Registered statistics files from `TableMetadata#statisticsFiles()`
- Deletion vector files referenced by live delete-manifest entries
The implementation does not open Puffin files, read Puffin footers, or parse
blob payload bytes. All reported information is derived from Iceberg table
metadata, snapshot manifest lists, and delete manifests.
Each row represents a Puffin file associated with one reference snapshot
through one metadata source. A physical Puffin file may appear in multiple rows
when it is referenced by multiple snapshots or through multiple sources.
The table contains the following columns:
| Column | Description |
|---|---|
| `reference_snapshot_id` | ID of the snapshot that references the Puffin
file |
| `file_path` | Fully qualified location of the Puffin file |
| `source` | Metadata source: `statistics` or `deletion_vector` |
| `file_size_in_bytes` | Total size of the Puffin file in bytes |
| `referenced_blob_count` | Number of referenced Puffin blobs represented by
the row |
| `referenced_blob_types` | Distinct types of the referenced blobs |
| `referenced_fields` | Distinct referenced fields, represented as
`field_id` and `current_field_name` pairs |
`current_field_name` is resolved against the current table schema and may be
null when a referenced field has been dropped.
## Example
```sql
SELECT *
FROM catalog.db.table.all_puffin_files;
```
Example output:
| reference_snapshot_id | file_path | source | file_size_in_bytes |
referenced_blob_count | referenced_blob_types | referenced_fields |
|---:|---|---|---:|---:|---|---|
| 1234567890 | `.../1234567890-table.stats` | `statistics` | 2144 | 2 |
`["apache-datasketches-theta-v1"]` |
`[{"field_id":1,"current_field_name":"id"},{"field_id":2,"current_field_name":"data"}]`
|
| 1234567890 | `.../00000-4-deletes.puffin` | `deletion_vector` | 525 | 1 |
`["deletion-vector-v1"]` |
`[{"field_id":<ROW_POSITION_FIELD_ID>,"current_field_name":"_pos"}]` |
| 1234567891 | `.../00000-4-deletes.puffin` | `deletion_vector` | 525 | 1 |
`["deletion-vector-v1"]` |
`[{"field_id":<ROW_POSITION_FIELD_ID>,"current_field_name":"_pos"}]` |
The same deletion-vector Puffin file appears more than once in this example
because it is referenced by multiple snapshots.
## Snapshot history semantics
The `all_puffin_files` metadata table scans all snapshots currently retained
in table metadata.
Deletion vector files are reconstructed independently for each snapshot from
that snapshot's live delete-manifest entries. As a result, the same physical
Puffin file may appear once for every retained snapshot that references one or
more blobs from the file.
Statistics files are obtained from the statistics files currently registered
in table metadata and associated with their registered snapshot IDs.
Statistics registration can be updated independently of data snapshots.
Therefore, the table does not reconstruct historical statistics registration
state. Replaced or removed statistics files are not reported merely because
their associated snapshots are still retained.
The table reports rows by snapshot ID rather than by branch or tag name. If
multiple references point to the same snapshot, the snapshot's Puffin files are
reported once for that snapshot.
## Filter planning
The implementation performs conservative task pruning for predicates over:
- `reference_snapshot_id`
- `source`
Source predicates can avoid preparing or scanning the unrelated metadata
source entirely.
Snapshot predicates can prevent tasks from being created for snapshots that
cannot match the filter.
A three-state evaluator is used during planning:
- `CANNOT_MATCH`
- `MIGHT_MATCH`
- `MUST_MATCH`
Predicates over unsupported fields or values that are not known during the
current planning phase return `MIGHT_MATCH`.
This is especially important for negated predicates, because negating an
unknown predicate must remain unknown and must not incorrectly prune a task.
Deletion-vector tasks are only planned for format version 3 or later.
Snapshot summaries are used conservatively to skip snapshots that explicitly
report zero delete files.
## Motivation
Iceberg stores multiple types of metadata in Puffin files, including table
statistics and deletion vectors.
Users currently need to query individual snapshots or inspect metadata JSON
files, manifest lists, and delete manifests separately to understand how Puffin
file references evolve over time.
Exposing this information through an all-snapshots metadata table makes it
easier to:
- Inspect statistics files registered for retained snapshots
- Identify Puffin files containing deletion vectors
- Determine which snapshots reference a physical Puffin file
- Find Puffin files shared across multiple snapshots
- Determine how many distinct blobs in a Puffin file are referenced by each
snapshot
- Inspect the blob types and fields associated with those references
- Compare statistics and deletion-vector references across snapshot history
- Debug stale, replaced, duplicated, or unexpected Puffin metadata
- Inspect Puffin usage without opening Puffin files or reading blob payloads
## Tests
The added tests cover:
- Metadata table schema and field documentation
- Tables without Puffin references
- Registered statistics files
- Statistics blob count, type, and field aggregation
- Statistics replacement and removal
- Field rename and dropped-field name resolution
- Deletion vector files in format version 3 tables
- Ignoring equality deletes and other non-Puffin delete files
- other behaviors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
