Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19770#discussion_r154787608
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -643,6 +633,44 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
} finally {
iterator.foreach(_.close())
}
+
+ // Clean corrupt or empty files that may have accumulated.
+ if (AGGRESSIVE_CLEANUP) {
+ var untracked: Option[KVStoreIterator[LogInfo]] = None
+ try {
+ untracked = Some(listing.view(classOf[LogInfo])
--- End diff --
This logic seems to be similar to what I have in the pipeline for the new
SHS project at https://github.com/vanzin/spark/pull/40. Except my change takes
care of other things (like also cleaning up any loaded UI data).
Could you take a look at that PR and see whether there's something it's not
covering? I can incorporate any needed changes there.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
