snazy commented on PR #433: URL: https://github.com/apache/polaris/pull/433#issuecomment-2500109355
> I added [this commit](https://github.com/apache/polaris/pull/433/commits/59361ed0bec6c3a2f8c609b32bbf4a9ffda12e66) to bound the extra serde. > > I am not in love with this exact implementation, but what do you think like an approach like this? Does this address your concerns around unbounded serde in the event that someone has a metadata.json which exceeds the configured limit? Not sure - but the direction is correct. However, I think there's a much bigger problem introduced by this PR: the _permanent_ heap usage. Even if you bound each metadata to 1MB, that can accumulate to 100,000 * 1MB (~= 95GB) of _permanent_ heap usage. With 10MB that's nearly 1TB of heap. Plus all the other attributes plus heap pressure during runtime. (100,000 entries is the _hard coded_ limit in `o.a.p.core.persistence.cache.EntityCache`.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
