CTTY opened a new pull request, #3733: URL: https://github.com/apache/iceberg-rust/pull/3733
## Which issue does this PR close? - Closes #TODO (issue to be filed before this leaves draft). ## What changes are included in this PR? `ManifestMetadata::parse` looks the manifest's `schema` JSON bytes up in a process-wide cache before deserializing. Manifests with identical schema bytes share one `SchemaRef` instead of each owning a parsed copy. On a 601-manifest table this parse is one of the largest per-manifest costs now that manifests decode concurrently (#3670). Design choices, for review: - **Keyed by the exact JSON bytes, not a digest or `schema-id`.** A hit is identical to a parse by construction. `schema-id` is optional in manifest metadata (pyiceberg omits it), and two writers can serialize the same schema to different bytes, so the bytes are the only key that is both always available and safe. Hashing a few KB of JSON costs microseconds against the parse it replaces. - **Process-wide rather than per `Table`.** The saving is on the first, cold read of a snapshot's manifests, where `ObjectCache` cannot help: it caches whole `Manifest`s and only pays off when the same manifest is read again. - **Bounded.** The map is cleared when it reaches 32 entries. The distinct-key count in one snapshot is roughly the number of schema versions still referenced by live manifests times the number of distinct writers, so a table with frequent schema evolution and no manifest merging can exceed this; see the open item below. - A parse error is returned and not cached. Error messages and kinds are unchanged. Stacked: the Avro writer-schema cache PR builds on this one and uses the same `ParsedCache`. ### Open items while in draft - [ ] Replace clear-at-32 with a byte-bounded LRU (`moka::sync::Cache` with a weigher; `moka` is already a dependency) so a snapshot with many distinct schema versions degrades gracefully instead of thrashing. - [ ] Make the cache injectable (field with a `static` default) so tests own their instance and the `get_or_parse_twice` retry helper goes away. - [ ] Standalone measurement. The numbers I have were taken with the writer-schema cache applied first: planning a 601-manifest pyiceberg-written table with `plan_files`, 4 cpu workers, fresh `Table` per iteration, median of 20, TPC-H SF10 2.8 → 1.8 ms and SF100 25.1 → 15.1 ms. ## Are these changes tested? Unit tests in `spec/manifest/metadata.rs` and `util/parsed_cache.rs`: two manifests with the same schema JSON share one `SchemaRef`; different schema JSON stays apart and parses to the expected metadata; malformed schema JSON still fails with `DataInvalid` and the existing message; the cache does not cache errors and clears at its bound. Existing manifest round-trip tests cover the read path. ## AI Disclosure Written with an AI assistant (Claude via Kiro); I reviewed and tested every change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
