rezasafi commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-478195761 > I have done testing on an actual cluster. I ran a few applications with and without faster polling (100ms interval). In both cases, I used 1 second for the heartbeat interval. In both cases, I did not enable proc fs metrics. There was no noticeable difference to the task times in the executors or to the stage times. However, the workload is small and the tasks took ~15s to ~1 minute. Thanks for let me now about those tests. Did you also tests the scenarios that this issue care about like a sudden jump in executor memory? (it is mentioned in the jira that "We've seen the memory usage of executors pike over 1 GB in less than a second" ). Was this change able to capture those jumps? I guess that would be a great test to be done.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
