JulySouthWind commented on a change in pull request #30164:
URL: https://github.com/apache/spark/pull/30164#discussion_r516451905
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -74,6 +74,12 @@ class BlockManagerMasterEndpoint(
// Mapping from block id to the set of block managers that have the block.
private val blockLocations = new JHashMap[BlockId,
mutable.HashSet[BlockManagerId]]
+ // Mapping from host name to shuffle (mergers) services
+ private val mergerLocations = new mutable.LinkedHashMap[String,
BlockManagerId]()
Review comment:
Maybe, we can add a shuffle service master role to manage the shuffle
service worker. App driver requests shuffle service workers from shuffle
service master. The master can customize the politics to assign the workers
for apps. Also, the master manages the blacklist of shuffle service worker,
adding or removing shuffle workers. In addition, the master can notify worker
to delete data of completed apps by heart beat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
