squito commented on a change in pull request #24499: [SPARK-25888][Core] Serve
local disk persisted blocks by the external service after releasing executor by
dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r281446728
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala
##########
@@ -223,11 +223,11 @@ class BlockManagerMaster(
}
/**
- * Find out if the executor has cached blocks. This method does not consider
broadcast blocks,
- * since they are not reported the master.
+ * Find out if the executor has cached blocks which are only available via
this executor.
+ * This method does not consider broadcast blocks, since they are not
reported the master.
*/
- def hasCachedBlocks(executorId: String): Boolean = {
- driverEndpoint.askSync[Boolean](HasCachedBlocks(executorId))
+ def hasExclusiveCachedBlocks(executorId: String): Boolean = {
Review comment:
"exclusive" is not the greatest name for this. The problem is that you can
have a replicated, MEMORY_ONLY block. So it could be on two executors (so its
not exclusive to any one executor), but not owned by any shuffle service.
that said, I'm not sure of a better name. It should at least be documented
that non-exclusive means owned by the external shuffle service.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
