attilapiros commented on a change in pull request #24499: [SPARK-25888][Core]
Serve local disk persisted blocks by the external service after releasing
executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r280033069
##########
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##########
@@ -179,6 +179,18 @@ public ManagedBuffer getBlockData(
return getSortBasedShuffleBlockData(executor, shuffleId, mapId, reduceId);
}
+ public ManagedBuffer getBlockData(
Review comment:
I have analysed this problem a little more: external shuffle service might
do the cleanup of executors' local directories when the application is removed.
It is done by calling
org.apache.spark.network.shuffle.ExternalShuffleBlockResolver#applicationRemoved:
https://github.com/apache/spark/blob/df3a80da4270c7b5eddb83383d8149ed64b25a66/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java#L221-L226
My uncertainty comes from this condition which is on the call chain:
https://github.com/apache/spark/blob/df3a80da4270c7b5eddb83383d8149ed64b25a66/core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala#L637-L639
In case of `ApplicationFinished` the first part is definitely true but I
have to check the second one a bit more and even do a test. So I will analyse
this further and I will come back to you with my findings.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
