Github user LantaoJin commented on the issue: https://github.com/apache/spark/pull/20532 @jerryshao and @jiangxb1987 , thanks for your advice. In 2.1.x, the two Update events (BlockUpdated & ExecutorMetricsUpdate) are all dumb. And in 2.2.x, only BlockUpdated event has a configuration to be logged. So the fair way is just adding an enable configuration to logging ExecutorMetrics for further using. But actually, we are refactoring the heartbeat to report more Executor information to the Driver. And this information will be logged to event log to be analysed. You know the Update events are so mass and they are not sequential. So we don't need all the events to do analysing. For example, we report Heap Usage after GC of all Executors, 10% events also can help us to get the result we want. So just add a configuration "enabled" for ExecutorMetricsUpdate is poorly simple in a big production cluster. BTW, the work I mentioned about will be contributed back to community as well soon.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org