scwhittle commented on pull request #15300: URL: https://github.com/apache/beam/pull/15300#issuecomment-908132864
Actually I see now you are saying that the amount of memory available for processing on a machine with a fixed amount of cpu but increased ram will be greater. That is true, however if the cache itself is the cause of OOMs (not user processing) it doesn't help. With the windmill cache, the objects were cached using the serialized size as the weight but the cached objects were deserialized (as that added a large benefit to the cache). However this meant that the memory used by the cache could be much more than the configured limit depending on the objects cached. It seems here we might be caching the protos so perhaps the size estimate is reliable. However if we change to caching user objects in the future we might need to be more careful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
