Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47017364
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -678,6 +827,54 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
method.invoke(dfs, action).asInstanceOf[Boolean]
}
+ /**
+ * String description for diagnostics
+ * @return a summary of the component staet
+ */
+ override def toString: String = {
+ val header = s"""
+ | FsHistoryProvider: logdir=$logDir,
+ | last scan time=$lastScanTime
+ | Cached application count =${applications.size}}
+ |
+ """.stripMargin
+ val sb = new StringBuilder(header)
+ applications.foreach(entry => sb.append(entry._2).append("\n"))
+ sb.toString
+ }
+
+ /**
+ * Probe for an update to an (incompleted) application.
+ *
+ * @param appId application ID
+ * @param attemptId optional attempt ID
+ * @param updateTimeMillis time in milliseconds to use as the threshold
for an update.
+ * @param data an [[FsHistoryProviderUpdateState]]
+ * @return true if the attempt has been updated.
+ */
+ override def isUpdated(appId: String,
+ attemptId: Option[String],
+ updateTimeMillis: Long,
+ data: Option[HistoryProviderUpdateState]): Boolean = {
+ val updateState = data.get.asInstanceOf[FsHistoryProviderUpdateState]
+ lookup(appId, attemptId) match {
+ case None =>
+ logDebug(s"Application Attempt $appId/$attemptId not found")
+ false
+ case Some(attempt) =>
+ updateState.version < attempt.version
--- End diff --
I was thinking about that. I think it's cleaner and if some other
implementation does care about time, it can do it itself.
I'll cut it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
