Github user LantaoJin commented on the issue:
https://github.com/apache/spark/pull/20532
@jerryshao and @jiangxb1987 , thanks for your advice. In 2.1.x, the two
Update events (BlockUpdated & ExecutorMetricsUpdate) are all dumb. And in
2.2.x, only BlockUpdated event has a configuration to be logged. So the fair
way is just adding an enable configuration to logging ExecutorMetrics for
further using. But actually, we are refactoring the heartbeat to report more
Executor information to the Driver. And this information will be logged to
event log to be analysed. You know the Update events are so mass and they are
not sequential. So we don't need all the events to do analysing. For example,
we report Heap Usage after GC of all Executors, 10% events also can help us to
get the result we want. So just add a configuration "enabled" for
ExecutorMetricsUpdate is poorly simple in a big production cluster. BTW, the
work I mentioned about will be contributed back to community as well soon.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
