Ngone51 commented on a change in pull request #30164:
URL: https://github.com/apache/spark/pull/30164#discussion_r520331124
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -74,6 +74,14 @@ class BlockManagerMasterEndpoint(
// Mapping from block id to the set of block managers that have the block.
private val blockLocations = new JHashMap[BlockId,
mutable.HashSet[BlockManagerId]]
+ // Mapping from host name to shuffle (mergers) services where the current app
+ // registered an executor in the past. Older hosts are removed when the
+ // maxRetainedMergerLocations size is reached in favor of newer locations.
+ private val shuffleMergerLocations = new mutable.LinkedHashMap[String,
BlockManagerId]()
Review comment:
Maybe we could remove the corresponding merger when there's fetch
failure happens on it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
