alexanderbianchi commented on PR #3679: URL: https://github.com/apache/iceberg-rust/pull/3679#issuecomment-6929558609
> Just curious, was this motivated by an use case where the eager cache initialization caused noticeable performance overhead? Yes! Our Table instances are request-scoped; we don’t retain them across requests. Each request has its own observability, scheduling priority, and authorization context. In particular, Iceberg-spawned tasks don’t propagate our observability context, so our integration carries it through a request-bound FileIO. The catalog’s storage factory is configured at catalog scope, without a per-load override. With a shared catalog, we currently reconstruct the returned Table to attach that FileIO and the query’s runtime. This means the table-owned manifest cache also has a request lifetime: its construction and teardown costs recur rather than being amortized across requests. It may still help with repeated scans within one request. We also expose a schema-only endpoint for the query service. Those requests construct Tables but never read manifests, so they received no benefit from the eagerly constructed cache. Nevertheless, dropping it performed Moka’s synchronous reclamation work, potentially executing deferred cleanup from other caches. Lazy initialization removes that hidden cost for metadata-only requests. Separately, easier request-context propagation and per-load storage binding would simplify this integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
