attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283 The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block. There is one change I am thinking about right now where I could been overeager. Before my change the old `RemoveRDD` was sent to every executor: https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178 Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back to be on the safe side.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
