snazy commented on PR #433: URL: https://github.com/apache/polaris/pull/433#issuecomment-2498838202
> Doesn't the catalog always have to do this regardless? We have to load the metadata.json from disk no matter what so the risk from unbounded json's is always present regardless of whether the Catalog also writes and reads a copy for itself. The answer is actually that technically a catalog does not really have to materialize it in full in memory. It could technically stream it directly from the object store - or in chunks from another source. @RussellSpitzer The bigger problem than materializing the big thing once is, that this change RE-serializes it again - and that is the real issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
