damccorm opened a new issue, #19645: URL: https://github.com/apache/beam/issues/19645
As a little experiment, we set the WindmillStateCache size to 2 GB. After we let a job run for a few hours, the workers all died from OOMs. Digging into the memory usage, over 10 GB of heap was used by the WindmillStateCache. The total weight the cache had calculated however was only ~800 MB. It looks like the StateCacheEntry calculation is way off. The weight of an entry was calculated as only ~200, while the actual heap usage of it (as reported by YourKit at least) was around 2,400 bytes. Imported from Jira [BEAM-7684](https://issues.apache.org/jira/browse/BEAM-7684). Original Jira may contain additional context. Reported by: SteveNiemitz. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
