squito commented on a change in pull request #24499: [SPARK-25888][Core] Serve
local disk persisted blocks by the external service after releasing executor by
dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r281833634
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -350,6 +356,10 @@ class BlockManagerMasterEndpoint(
).map(_.flatten.toSeq)
}
+ private def externalShuffleServiceIdOnHost(blockManagerId: BlockManagerId):
BlockManagerId = {
+ BlockManagerId(blockManagerId.executorId, blockManagerId.host,
externalShuffleServicePort)
Review comment:
I had to step through this pretty carefully to figure out why you were
passing in `blockManagerId.executorId` here, though it was actually owned by
the shuffle service. In fact I was originally thinking that you should put in
a "dummy" executor Id in here so it was clear its actually referring to
something owned by the shuffle service. But now I see that you actually need
to know the original executor, to know which dirs to look in. It might be
worth at least a comment here that we need to keep a handle on the original
executor, so the shuffle service knows which dir to look in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
