HaoXuAI commented on PR #17219: URL: https://github.com/apache/iceberg/pull/17219#issuecomment-5026982832
You're right — a full `TableMetadata` always materializes the snapshot list and the id→snapshot index, so this doesn't help any path that builds one; `snapshot-loading-mode=refs` is the existing lever there. What this targets is a library/tooling case: reading a table's **full** snapshot history *without* building a `TableMetadata`. The pattern is: - load with `snapshot-loading-mode=refs`, so `TableMetadata` holds only referenced snapshots while `metadataLocation()` still points to the complete `metadata.json`; - then stream that file with `SnapshotParser.fromJson(InputFile)` to walk the full history one snapshot at a time (filtering by id / timestamp / manifest-list), never calling `snapshots()` and never firing the supplier. We use this in a metrics pipeline that scans snapshot histories across many tables — on deep-history tables the eager materialization was the dominant heap cost. Measured on a metadata document with 200k snapshots: **~49 MB retained (eager) vs ~1 MB (streaming)**; under `-Xmx128m` the eager read OOMs where streaming completes under `-Xmx64m`. Full-scan read time is comparable (streaming is slightly faster, skipping the intermediate JSON tree). So it's meant as a companion to REFS mode for out-of-`TableMetadata` history reads, not a change to how `TableMetadata` loads. If you feel it fits better as a `Table`/`SnapshotUtil`-level accessor (using the table's `FileIO`) than a `SnapshotParser` method, I'm glad to reshape it — or close it if it's too niche for core. Wanted your read before going further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
