wypoon commented on issue #25659: [SPARK-28770][CORE][TESTS]Ignore SparkListenerStageExecutorMetrics in testApplicationReplay test URL: https://github.com/apache/spark/pull/25659#issuecomment-527741025 `SparkListenerStageExecutorMetrics` were introduced in SPARK-23429 (https://github.com/apache/spark/pull/21221/). By design, executor metrics update events are not logged by the `EventLoggingListener`. Instead, the listener keeps track of the per stage peaks for any of the executors and the driver for which it has received metrics. On stage completion, the peaks for the stage are logged in `SparkListenerStageExecutorMetrics` events for each of these executors and driver. Since executor metrics update events are not logged in the event log, they do not get replayed. Thus the listener for the replay never sees metrics updates. It is therefore valid to exclude `SparkListenerStageExecutorMetrics` events from both the original and the replay for the purpose of comparison. However, instead of excluding all `SparkListenerStageExecutorMetrics` events from both the original `EventLoggingListener` and the replay listener, we can have a finer-grained fix, which I have proposed in https://github.com/apache/spark/pull/25673/ for comparison. It should be sufficient to exclude any `SparkListenerStageExecutorMetrics` events for the driver. This is because with SPARK-26329 (https://github.com/apache/spark/pull/23767/), executor metrics are also sent in task end events (which do get replayed), so the `EventLoggingListener` always receives metrics for the executors (just not necessarily for the driver), and thus `SparkListenerStageExecutorMetrics` events for the executors always get logged.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
