venkata91 commented on a change in pull request #30164:
URL: https://github.com/apache/spark/pull/30164#discussion_r522522341
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -74,6 +74,14 @@ class BlockManagerMasterEndpoint(
// Mapping from block id to the set of block managers that have the block.
private val blockLocations = new JHashMap[BlockId,
mutable.HashSet[BlockManagerId]]
+ // Mapping from host name to shuffle (mergers) services where the current app
+ // registered an executor in the past. Older hosts are removed when the
+ // maxRetainedMergerLocations size is reached in favor of newer locations.
+ private val shuffleMergerLocations = new mutable.LinkedHashMap[String,
BlockManagerId]()
Review comment:
Yes, makes sense. Still I went ahead and made that change. One question
I had is what if its a transient failure due to heavy load or something of that
sorts? Do we still want to remove the host from the list in those cases?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
