sumeetgajjar commented on pull request #34536: URL: https://github.com/apache/spark/pull/34536#issuecomment-964631004
> So why would we end up in a situation where we need `SparkListenerExecutorRemoved` to be fired again ? Hi @mridulm, I believe the additional `SparkListenerExecutorRemoved` is to nullify the effects from re-registration of a BlockManager from a dead executor. Please consider the following sequence of events: - Driver receives a Heartbeat from a executor which is removed from schedulers metadata. - Driver asks to re-register the executor's BlockManager - On BlockManager registration a `SparkListenerBlockManagerAdded` is posted: https://github.com/apache/spark/blob/f6a044cf8cd83e6b3b30e515acbac0ec81607463/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L575 - AppStatusListener in `onBlockManagerAdded` tries to get the `LiveExecutor`: https://github.com/apache/spark/blob/f6a044cf8cd83e6b3b30e515acbac0ec81607463/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala#L863 - getOrCreateExecutor returns a new `LiveExecutor` object even when the executor is dead: https://github.com/apache/spark/blob/f6a044cf8cd83e6b3b30e515acbac0ec81607463/core/src/main/scala/org/apache/spark/status/AppStatusListener.scala#L1125 @Ngone51 can you please confirm if my example is valid? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
