vanzin commented on issue #22015: [SPARK-20286][SPARK-24786][Core][DynamicAllocation] Release executors on unpersisting RDD URL: https://github.com/apache/spark/pull/22015#issuecomment-468437353 So, I just happen to be playing with this part of the code, and have a different solution for this. It's part of a much bigger change, so if you want to implement the specific solution for the caching problem here, that would be fine. The main change in this part is that I'm getting rid to the calls to `blockManagerMaster.hasCachedBlocks`, which are sort of expensive. It's possible to track this information without that call; the listener just needs to process `onBlockUpdated` and `onUnpersistRDD`. With that info it's also possible to handle `onUnpersistRDD` without the extra changes to the event, since the listener can itself keep track of which executors hold blocks for each RDD. I think that's a better approach.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
