wypoon commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-475834874 @squito and @edwinalu - I updated the PR with your feedback. I have extracted the polling logic into a separate class, combining `activeStages` and `stageMetricPeaks` into a single CHM. I also handle task failure and send the task metric peaks in the `TaskFailedReason` (for the same cases where we send accumulator updates). I added new unit tests for sending task metric peaks in the `DirectTaskResult` and the `TaskFailedReason`. I added executor updates from the driver and task end events in the existing test in `EventLoggingListenerSuite`. This verifies that per-stage aggregation of the executor metrics is correct.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
