mfellner opened a new pull request, #3731:
URL: https://github.com/apache/iceberg-rust/pull/3731
## Which issue does this PR close?
- Related to the "Iceberg Cache" epic #1226. No dedicated issue yet; happy
to file one if preferred.
## What changes are included in this PR?
`ObjectCache` is crate-private, and `Table::manifest_reader()` always
bypasses it. A caller that must inspect every manifest itself (for example to
validate delete manifests before planning) therefore parses each manifest once,
and `TableScan::plan_files` then fetches and parses it a second time through
the table's object cache.
This adds one public method:
```rust
impl Table {
pub async fn seed_manifest_cache(&self, manifest_file: &ManifestFile,
manifest: Arc<Manifest>) -> bool;
}
```
It stores the already parsed manifest under the key
`ObjectCache::get_manifest` uses (`ManifestKey`: path, snapshot id, sequence
number, `first_row_id`), so a later `plan_files` over the same `ManifestFile`
reuses it. The cache is shared by every clone of the table. It returns `false`
and stores nothing when the cache is disabled, keeps an existing entry, and
changes nothing when it is not called. The contract documented on the method is
that `manifest` is the result of reading `manifest_file` with
`Table::manifest_reader`.
In a Sentinel-style workload, this removes about 2 GETs and an Avro parse
per manifest per query (measured manifest counts up to ~1,100).
## Are these changes tested?
Unit tests in `io/object_cache.rs`: a seeded manifest is returned
(pointer-equal) even after the manifest file is deleted, a second seed keeps
the first entry, and a disabled cache stores nothing. An Iceberg v3 table
written by Spark with Iceberg Java 1.11.0 (merge-on-read, deletion vectors) was
also planned with and without seeding, with the manifest files removed after
seeding, in an Orcrist fork CI job; that harness is not part of this PR. `cargo
fmt`, `cargo clippy --all-targets -D warnings` and the object cache tests pass
locally.
## AI Disclosure
This change was written with Claude Code (Claude Opus 5.5) and tested by the
author.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]