wypoon commented on issue #23767: [SPARK-26329][CORE] Faster polling of executor memory metrics. URL: https://github.com/apache/spark/pull/23767#issuecomment-481370713 @edwinalu @rezasafi I could add event logs for a small application that shows what gets written now with my changes. More frequent polling does not change the fact that stage executor metrics only get logged when the stage finishes (no change in frequency there). The main change is that task executor metrics now get sent in the task end event and that is logged. Currently, in `HistoryServerSuite`, `cases` contain ``` "executor list with executor metrics json" -> "applications/application_1506645932520_24630151/executors", "executor list with executor process tree metrics json" -> "applications/application_1538416563558_0014/executors", "executor list with executor garbage collection metrics json" -> "applications/application_1536831636016_59384/1/executors", ``` I could add another case there, but I was thinking that another option would be to replace the above 3 cases with a new case, that would simply be the new "executor list with executor metrics json". It would show process tree metrics as well as garbage collection metrics, since those are now part of executor metrics. I'd like to know your thoughts. @squito you too, if you have an opinion on this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
