lukecwik commented on pull request #15300: URL: https://github.com/apache/beam/pull/15300#issuecomment-907350175
> One benefit of the static limit versus percentage is that if OOMs are caused by caching too much (or cache memory tracking not working as well as expected), the static limit allows for the machine size to be increased to avoid the OOMs as the cache will use the same amount of memory. If the max cache size is automatically increased proportional to the worker shape, the OOMs will just continue possibly negating VM right-sizing or manual size tuning. > > Perhaps instead of a static value or a static percentage we could do a combination: > static_value + % of machine size > Then we don't scale entirely with the machine size but also will use a larger cache by default for larger machines. VMR and/or users should specify a machine with _more memory_ and not one with _more cores and memory_ since we want to increase the amount of memory per core. This would still allow a pure percentage based model to work since a machine with _more memory_ will still have a greater amount of memory that is not dedicated to the cache. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
