mridulm commented on code in PR #39459:
URL: https://github.com/apache/spark/pull/39459#discussion_r1108043615
##########
core/src/main/scala/org/apache/spark/storage/BlockManager.scala:
##########
@@ -1424,6 +1457,16 @@ private[spark] class BlockManager(
blockStoreUpdater.save()
}
+ // Check whether a rdd block is visible or not.
+ private[spark] def isRDDBlockVisible(blockId: RDDBlockId): Boolean = {
+ // If the rdd block visibility information not available in the block
manager,
+ // asking master for the information.
+ if (blockInfoManager.isRDDBlockVisible(blockId)) {
+ return true
+ }
+ master.isRDDBlockVisible(blockId)
Review Comment:
Yes, `blockInfoManager.isRDDBlockVisible(blockId)` should check for the
current executor side state of the block to see if it is visible or not.
If visible, immediately return true.
Else, query driver for visibility status - if driver returns true, cache
that.
Also, when a block becomes visible at driver, it will update the block
managers hosting the block about the visibility status - which is then
maintained at executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
