Anurag-S1ngh opened a new pull request, #4335: URL: https://github.com/apache/iceberg-python/pull/4335
# Rationale for this change Planning a table repeatedly read the same immutable files from object storage every time: the manifest list on every `plan_files`, and every manifest on every plan. `_manifest_cache` only keeps the parsed `ManifestFile` records, not the file contents. This adds a byte cache to `FileIO`, keyed by location. Parsing is unchanged; the bytes are cached, so a `ManifestFile` record parsed from one manifest list cannot be served values from another, which a parsed-object cache keyed by path would allow. It follows Iceberg Java's `io.manifest.cache-*` properties and is off by default: - `io.manifest.cache-enabled` (default `false`) - `io.manifest.cache.max-total-bytes` (default 100 MiB) - `io.manifest.cache.max-content-length` (default 8 MiB) - `io.manifest.cache.expiration-interval-ms` (default 60 s) `ContentCache` is a bounded, least-recently-used map with lazy expiry, matching the `get(location) -> bytes | None` and `put(location, data)` shape from the issue. A FileIO that wants a different backing store, a disk cache for example, can override `FileIO.content_cache`, which is read-only and returns `None` when the properties do not enable caching. Only manifest lists and manifests go through the cache. Data files and `metadata.json` are unchanged, so nothing that can be rewritten is cached. # Are these changes tested? Yes. `tests/io/test_content_cache.py` covers: - caching is off unless `io.manifest.cache-enabled` is truthy, including the values `false`, `0` and the empty string - the limits are read from FileIO properties, including table properties - entries over `max-content-length`, or over `max-total-bytes` on their own, are not cached - least-recently-used eviction and expiry - a manifest list and a manifest are each read once when the cache is enabled, and once per call when it is not Full unit suite: `4586 passed, 5 skipped`. `prek run -a` passes. # Are there any user-facing changes? No behavior change by default. With `io.manifest.cache-enabled=true`, repeated planning against the same `FileIO` reuses the manifest bytes instead of reading them from storage again, which also keeps planning working through a short outage for files already read. Closes #4300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
