technicolorbeat opened a new issue, #1986: URL: https://github.com/apache/iceberg-go/issues/1986
## Problem #1956 defers materialization of unreferenced snapshot history when a REST commit response is parsed with `snapshot-loading-mode=refs`. The benefit is lost when the returned table is used to start another transaction. `Table.NewTransaction*` calls `MetadataBuilderFromBase`, which currently requests the complete snapshot collection. This immediately materializes every deferred historical snapshot, even though the normal REST commit path primarily needs the accumulated requirements and updates rather than a fully rebuilt metadata document. This path pays both the deferred indexing cost and the later full-decoding cost. It is therefore more expensive than eager parsing when another transaction is created immediately. Measured on an Apple M1 Max, using medians of five runs: | Metadata | Eager parse + builder | Deferred parse + builder | Time | Allocations | Allocated bytes | |---:|---:|---:|---:|---:|---:| | 1,999,830 B | 80.611 ms | 98.314 ms | +22.0% | +20.7% | +38.9% | | 5,856,882 B | 232.241 ms | 287.704 ms | +23.9% | +20.7% | +38.5% | This is a performance limitation, not a known correctness problem. It is intentionally separate from #1946, whose scope is commit-response parsing. ## Possible directions These are alternatives for future evaluation; this issue does not select a design. ### Short term Delay complete snapshot materialization during `MetadataBuilderFromBase` construction. Materialize only when a builder operation actually requires the complete snapshot collection. ### Mid term Represent builder snapshot changes as mutations over an immutable deferred base, allowing additions, removals, ref validation, and individual ID lookup without decoding unrelated historical snapshots. ### Long term Introduce a shared snapshot-collection abstraction used by metadata and metadata builders, with consistent support for indexed lookup, lazy iteration/materialization, mutation, cloning, validation, and serialization. ## Evaluation criteria - Preserve eager/deferred behavior and parse-time validation semantics. - Preserve complete snapshot history across metadata build, serialization, equality, and v1/v2/v3 conversion. - Preserve defensive-copy and concurrency guarantees. - Avoid full history decoding for REST transactions that do not require it. - Benchmark eager parse, deferred parse, parse plus builder construction, and representative subsequent transaction operations by metadata size. Related: #1946 and #1956. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
