Ngone51 commented on code in PR #54136:
URL: https://github.com/apache/spark/pull/54136#discussion_r2767451089
##########
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:
##########
@@ -773,6 +802,10 @@ private[spark] class BlockManager(
if (blockId.isShuffle) {
logDebug(s"Putting shuffle block ${blockId}")
+ // Wait for ShuffleManager to be initialized before handling shuffle
migration requests.
+ // This can happen when an executor receives migration requests before
its ShuffleManager
+ // is fully initialized.
+ waitForShuffleManagerInit(blockId)
Review Comment:
Shall we maybe `waitForShuffleManagerInit` before `BlockManager` register
with the driver?
There might be other potential cases or future cases affected by this issue
if we fix in current way. My point is still that we should ensure the
BlockManager is really ready before the driver picks it up as an candidate for
some stuff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
