fangchenli opened a new issue, #4300: URL: https://github.com/apache/iceberg-python/issues/4300
### Feature Request / Improvement Planning the same table repeatedly reads the same immutable files from object storage every time, and fails while the store is unreachable, even for a snapshot just planned. Per `load_table` plus `plan_files`, pyiceberg 0.12.0 (and `main`) reads: 1. **`metadata.json`** on every load, on non-REST catalogs (REST returns it inline). 2. **The manifest list** on every plan. `_manifests` re-reads it on purpose, to avoid the O(N²) memory of caching overlapping `ManifestFile` tuples (#1187). 3. **Each manifest** on every plan, via `fetch_manifest_entry`. `_manifest_cache` keeps the `ManifestFile` records, not the manifests' contents. These files are never rewritten, so the bytes at a location never change. Iceberg Java caches these bytes. `ManifestFiles` and `ManifestLists` read through a per-FileIO `ContentCache`, set by these properties ([`CatalogProperties`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/CatalogProperties.java)): - `io.manifest.cache-enabled` (default `false`) - `io.manifest.cache.max-total-bytes` (default 100 MiB) - `io.manifest.cache.max-content-length` (default 8 MiB) - `io.manifest.cache.expiration-interval-ms` (default 60 s) ### Proposal 1. **Cache manifest and manifest-list bytes by location**, off by default and bounded in bytes, as in Java. Parsing stays as it is. Caching bytes avoids the problem in #3978 / #3991. A `ManifestFile` record carries values from the manifest list that names it, so a parsed-object cache keyed by path can serve one reference's values to another. (iceberg-rust adds `added_snapshot_id`, `sequence_number` and `first_row_id` to its key for this reason.) Bytes are the same for every reference. 2. **Let the caller supply the cache**, e.g. `get(location) -> bytes | None` and `put(location, data)`, with the in-memory one as the default. A disk cache behind it would survive restarts and keep planning working through a store outage for snapshots already read. iceberg-rust has a similar hook (`ObjectCacheProvide`) for parsed objects. `metadata.json` could use the same cache. For REST, #4075 / #4094 cover it with `If-None-Match`. Java configures this with FileIO properties; #2952 used `Config` and an env var for `manifest-cache-size`. Either is fine. Related: #4007 and #3662 make each read cheaper; this removes repeated reads. ### What we do today In Strata (a materialization layer that, among other things, caches Iceberg scans) we set `py-io-impl` to a FileIO that wraps the one `load_file_io` would pick. Its `new_input` serves `*.metadata.json` and `*.avro` on remote schemes from a bounded disk cache, keyed by location plus the endpoint properties, and passes everything else through. Drawbacks: - It identifies immutable files by suffix. - It takes over `py-io-impl`, so it must carry a user's own `py-io-impl` under another property. - It builds an inner FileIO per scheme, since a FileIO does not know which locations it will serve. A cache hook at the read points above would remove the need for the wrapper. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
