Ngone51 commented on a change in pull request #34629:
URL: https://github.com/apache/spark/pull/34629#discussion_r767430431
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -620,11 +620,13 @@ private[spark] class BlockManager(
* Note that this method must be called without any BlockInfo locks held.
*/
def reregister(): Unit = {
- // TODO: We might need to rate limit re-registering.
- logInfo(s"BlockManager $blockManagerId re-registering with master")
- master.registerBlockManager(blockManagerId,
diskBlockManager.localDirsString, maxOnHeapMemory,
- maxOffHeapMemory, storageEndpoint)
- reportAllBlocks()
+ if (!SparkEnv.get.isStopped) {
Review comment:
Hmm...so, let's assume such a scenario: driver command a shutdown
request to the executor. Before the executor receives the shutdown request, it
finds that it itself has been removed from the driver when reporting a block to
the driver. Then, the executor starts to reregister itself. Since the executor
hasn't received shutdown request, which means `SparkEnv.get.isStopped=false`,
it successfully reregisters with the driver. Soon, the executor receives the
shutdown request and exits itself.
So, this change doesn't fix the issue thoroughly, right? @wankunde
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
