Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22504#discussion_r226737882
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -274,11 +282,20 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
pool.scheduleWithFixedDelay(
getRunner(() => checkForLogs()), 0, UPDATE_INTERVAL_S,
TimeUnit.SECONDS)
- if (conf.getBoolean("spark.history.fs.cleaner.enabled", false)) {
+ if (conf.get(CLEANER_ENABLED)) {
// A task that periodically cleans event logs on disk.
pool.scheduleWithFixedDelay(
getRunner(() => cleanLogs()), 0, CLEAN_INTERVAL_S,
TimeUnit.SECONDS)
}
+
+ driverLogFs.foreach { _ =>
--- End diff --
Here it's clearer to use `if (driverLogFs.isDefined && ...)`.
But really this is a bit confusing and I think my previous suggestion is
clearer: don't keep `driverLogFs` in a field, just get it when running the
cleaner task. And do this initialization based on the config entries, not
`driverLogFs`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
