123digits opened a new issue, #4075: URL: https://github.com/apache/iceberg-python/issues/4075
### Feature Request / Improvement **Iceberg Java already does this.** Its REST client loads tables "freshness-aware": it keeps a table cache, sends `If-None-Match` with the cached `ETag`, and reuses the cached table when the server answers `304 Not Modified`. The cache is configured by two catalog properties in [`RESTCatalogProperties`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/RESTCatalogProperties.java): - `rest-table-cache.max-entries` (default 100) - `rest-table-cache.expire-after-write-ms` (default 5 minutes) This asks for the same in pyiceberg's `RestCatalog`, whose `load_table` today always performs a full load. **What the spec defines** (`open-api/rest-catalog-open-api.yaml`): - `loadTable` (`GET /v1/{prefix}/namespaces/{namespace}/tables/{table}`) accepts an optional `If-None-Match` header: "allows the server to return 304 (Not Modified) if the metadata is current. The content is the value of the ETag received in a CreateTableResponse, LoadTableResponse or CommitTableResponse." - `LoadTableResponse` carries an `etag` header. - `304`: "Not Modified - Based on the content of the 'If-None-Match' header the table metadata has not changed since." **What pyiceberg does today** (0.12.0 and `main`): `RestCatalog.load_table` sends no `If-None-Match`, keeps no `ETag` from load/create/commit responses, and has no handling for a `304`. **Use case:** clients that load the same tables repeatedly — Kubernetes operators reconciling on an interval, long-running services, notebooks re-reading tables — download and parse the full metadata JSON on every `load_table`, even when nothing changed. That JSON grows with snapshot history and can be large for busy tables. Servers already answer conditionally: Lakekeeper sends an `ETag` and replies `304` from v0.11.0. **Suggested shape** (matching Iceberg Java, so the same catalog properties work across clients): - an optional, bounded, per-catalog cache of identifier -> (`ETag`, loaded metadata and metadata location), filled from the `etag` header of load, create and commit responses; - when an entry exists, `load_table` sends `If-None-Match` and, on `304`, returns the cached table instead of treating the empty body as an error; - configured by Java's property names, `rest-table-cache.max-entries` and `rest-table-cache.expire-after-write-ms`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
