dongjoon-hyun commented on code in PR #51199:
URL: https://github.com/apache/spark/pull/51199#discussion_r2456176543
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -604,23 +604,27 @@ package object config {
"cache block replication should be positive.")
.createWithDefaultString("30s")
+ private[spark] val STORAGE_DECOMMISSION_FALLBACK_STORAGE_CLEANUP =
+ ConfigBuilder("spark.storage.decommission.fallbackStorage.cleanUp")
+ .doc("If true, Spark cleans up its fallback storage data once individual
shuffles are " +
+ "freed (interval configured via spark.cleaner.periodicGC.interval),
and during " +
+ "shutting down.")
+ .version("3.2.0")
+ .booleanConf
+ .createWithDefault(false)
+
private[spark] val STORAGE_DECOMMISSION_FALLBACK_STORAGE_PATH =
ConfigBuilder("spark.storage.decommission.fallbackStorage.path")
.doc("The location for fallback storage during block manager
decommissioning. " +
"For example, `s3a://spark-storage/`. In case of empty, fallback
storage is disabled. " +
- "The storage should be managed by TTL because Spark will not clean it
up.")
+ "The storage will not be cleaned up by Spark unless " +
+ s"${STORAGE_DECOMMISSION_FALLBACK_STORAGE_CLEANUP.key} is true. " +
+ "Use an external clean up mechanism when false, for instance a TTL.")
.version("3.1.0")
.stringConf
.checkValue(_.endsWith(java.io.File.separator), "Path should end with
separator.")
.createOptional
- private[spark] val STORAGE_DECOMMISSION_FALLBACK_STORAGE_CLEANUP =
- ConfigBuilder("spark.storage.decommission.fallbackStorage.cleanUp")
- .doc("If true, Spark cleans up its fallback storage data during shutting
down.")
- .version("3.2.0")
- .booleanConf
- .createWithDefault(false)
Review Comment:
Please keep the existing code as much as possible and reduce the git diff,
@EnricoMi . What we need is only `.doc` part change, isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
