vanzin commented on a change in pull request #24499: [SPARK-25888][Core] Serve
local disk persisted blocks by the external service after releasing executor by
dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r280619556
##########
File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
##########
@@ -104,6 +104,15 @@ private[deploy] class Worker(
private val CLEANUP_NON_SHUFFLE_FILES_ENABLED =
conf.get(config.STORAGE_CLEANUP_FILES_AFTER_EXECUTOR_EXIT)
+ val EXTERNAL_SHUFFLE_SERVICE_ENABLED =
conf.get(config.SHUFFLE_SERVICE_ENABLED)
+
+ if (CLEANUP_NON_SHUFFLE_FILES_ENABLED && EXTERNAL_SHUFFLE_SERVICE_ENABLED) {
Review comment:
Are you sure that there isn't other stuff you can delete that are neither
shuffle nor cached files?
If really there isn't anything else to clean, it seems that now both of
these config options are basically the same thing.
If there is something, then that should still be done, otherwise you'll be
using disk space unnecessarily.
(There's a whole other discussion about how to even delete orphaned shuffle
and cached files after the executor that created them goes away, but that's for
a separate discussion.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
