attilapiros edited a comment on issue #24499: [SPARK-27677][Core] Serve local disk persisted blocks by the external service after releasing executor by dynamic allocation URL: https://github.com/apache/spark/pull/24499#issuecomment-493643283 The new message `RemoveBlocks` used only when the fetch RDD via shuffle service feature is enabled and done asynchronously: called within a `Future` block. There is one change I am thinking right now where I could be overeager. Before my change the old `RemoveRDD` was sent to every executor: https://github.com/apache/spark/blob/dfeeda24c0f5d60bf6d2e1868c5290a1f62dc558/core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala#L171-L178 Now I am only sending `RemoveRDD` to those executors where its blocks are registered. And I am wondering whether we miss to delete those blocks which not reported to the master. I can easily change that small logic back to be on the safe side.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
