Ngone51 commented on a change in pull request #32114:
URL: https://github.com/apache/spark/pull/32114#discussion_r640324064
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -336,13 +345,40 @@ class BlockManagerMasterEndpoint(
private def removeBlockManager(blockManagerId: BlockManagerId): Unit = {
val info = blockManagerInfo(blockManagerId)
+ // Not removing info from the blockManagerInfo map, but only setting the
removal timestamp of
+ // the executor in BlockManagerInfo. This info will be removed from
blockManagerInfo map by the
+ // blockManagerInfoCleaner once now() - info.executorRemovalTs >
executorTimeoutMs.
+ //
+ // We are delaying the removal of BlockManagerInfo to avoid a BlockManager
reregistration
+ // while a executor is shutting. This unwanted reregistration causes
inconsistent bookkeeping
+ // of executors in Spark.
Review comment:
Oh..Just some succinct description should be fine. This's too detailed
for the comment. I think you can remove the detailed description below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
