eric-maynard commented on PR #433: URL: https://github.com/apache/polaris/pull/433#issuecomment-2498898442
> The answer is actually that technically a catalog does not really have to materialize it in full in memory. It could technically stream it directly from the object store - or in chunks from another source. This is true, but we are not talking about what an arbitrary catalog technically has to do but what Polaris actually does do. In the future I think we should optimize this with a structured schema in the metastore (perhaps that more closely matches the schema of the Java TableMetadata object) -- and when we do, we should optimize the cache as well. The question is whether this future optimization is a blocker for adding caching on top of the current behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
