damccorm opened a new issue, #21343: URL: https://github.com/apache/beam/issues/21343
The current MemoryMonitor has an option to upload a profile to GCS. However it only uploads the profile to GCS when starting the memory monitor thread. This is currently done when the harness is started only. However MemoryMonitor's tryToDumpHeap is public and therefore can be called clients at any point (e.g. not only on worker restart). In this case we do not upload to GCS until the harness restarts which is not good end user behavior. It's currently called by StreamingDataflowWorker at a point of high memory but before the worker will actually crash, which is how we caught this behavior. Probably we should add a boolean to tryToHeapDump to get an idea for if the worker is about to die or if this is a pre-emptive dump. If it's pre-emptive we can probably immediately upload it to GCS as well if so configured. Imported from Jira [BEAM-13718](https://issues.apache.org/jira/browse/BEAM-13718). Original Jira may contain additional context. Reported by: laraschmidt. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
