alexanderbianchi commented on PR #3679:
URL: https://github.com/apache/iceberg-rust/pull/3679#issuecomment-6929558609

   > Just curious, was this motivated by an use case where the eager cache 
initialization caused noticeable performance overhead?
   Yes! Our Table instances are request-scoped; we don’t retain them across 
requests. Each request has its own observability, scheduling priority, and 
authorization context.
   
   In particular, Iceberg-spawned tasks don’t propagate our observability 
context, so our integration carries it through a request-bound FileIO. The 
catalog’s storage factory is configured at catalog scope, without a per-load 
override. With a shared catalog, we currently reconstruct the returned Table to 
attach that FileIO and the query’s runtime.
   
   This means the table-owned manifest cache also has a request lifetime: its 
construction and teardown costs recur rather than being amortized across 
requests. It may still help with repeated scans within one request.
   
   We also expose a schema-only endpoint for the query service. Those requests 
construct Tables but never read manifests, so they received no benefit from the 
eagerly constructed cache. Nevertheless, dropping it performed Moka’s 
synchronous reclamation work, potentially executing deferred cleanup from other 
caches.
   
   Lazy initialization removes that hidden cost for metadata-only requests. 
Separately, easier request-context propagation and per-load storage binding 
would simplify this integration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to