attilapiros commented on a change in pull request #24499: [SPARK-25888][Core]
Serve local disk persisted blocks by the external service after releasing
executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r279927693
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala
##########
@@ -576,13 +616,28 @@ private[spark] class BlockManagerInfo(
s" (size: ${Utils.bytesToString(diskSize)})")
}
}
- if (!blockId.isBroadcast && blockStatus.isCached) {
- _cachedBlocks += blockId
+
+ if (!blockId.isBroadcast) {
+ if (!externalShuffleServiceEnabled || !storageLevel.useDisk) {
Review comment:
For me reading my version is easier: the block is an exclusive block when
external shuffle service is disabled or it is enabled but not block is in the
disk. Although if it is not exclusive currently but it was known before
(existed) than to be in the safe side (as it could have been exclusive) we
should remove it from the exclusive blocks.
Also it uses the Set operation (add/remove) rarely. So I would like to keep
this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
