scwhittle commented on pull request #15300: URL: https://github.com/apache/beam/pull/15300#issuecomment-907203685
One benefit of the static limit versus percentage is that if OOMs are caused by caching too much (or cache memory tracking not working as well as expected), the static limit allows for the machine size to be increased to avoid the OOMs as the cache will use the same amount of memory. If the max cache size is automatically increased proportional to the worker shape, the OOMs will just continue possibly negating VM right-sizing or manual size tuning. Perhaps instead of a static value or a static percentage we could do a combination: static_value + % of machine size Then we don't scale entirely with the machine size but also will use a larger cache by default for larger machines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
