HyukjinKwon commented on a change in pull request #31215:
URL: https://github.com/apache/spark/pull/31215#discussion_r559265518
##########
File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala
##########
@@ -119,6 +120,27 @@ object FallbackStorage extends Logging {
}
}
+ /** Clean up the generated fallback location for this app. */
+ def cleanUp(conf: SparkConf, hadoopConf: Configuration): Unit = {
+ if (conf.get(STORAGE_DECOMMISSION_FALLBACK_STORAGE_PATH).isDefined &&
+ conf.get(STORAGE_DECOMMISSION_FALLBACK_STORAGE_CLEANUP) &&
+ conf.contains("spark.app.id")) {
+ val fallbackPath =
+ new Path(conf.get(STORAGE_DECOMMISSION_FALLBACK_STORAGE_PATH).get +
"/" + conf.getAppId)
Review comment:
(BTW, please just don't bother about the co-author stuff manually
@dongjoon-hyun. I think it's fine to follow GitHub Actions' policy for now)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
