Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/6935#discussion_r47004101
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -678,6 +827,54 @@ private[history] class FsHistoryProvider(conf:
SparkConf, clock: Clock)
method.invoke(dfs, action).asInstanceOf[Boolean]
}
+ /**
+ * String description for diagnostics
+ * @return a summary of the component staet
+ */
+ override def toString: String = {
+ val header = s"""
+ | FsHistoryProvider: logdir=$logDir,
+ | last scan time=$lastScanTime
+ | Cached application count =${applications.size}}
+ |
+ """.stripMargin
+ val sb = new StringBuilder(header)
+ applications.foreach(entry => sb.append(entry._2).append("\n"))
+ sb.toString
+ }
+
+ /**
+ * Probe for an update to an (incompleted) application.
+ *
+ * @param appId application ID
+ * @param attemptId optional attempt ID
+ * @param updateTimeMillis time in milliseconds to use as the threshold
for an update.
+ * @param data an [[FsHistoryProviderUpdateState]]
+ * @return true if the attempt has been updated.
+ */
+ override def isUpdated(appId: String,
+ attemptId: Option[String],
+ updateTimeMillis: Long,
+ data: Option[HistoryProviderUpdateState]): Boolean = {
+ val updateState = data.get.asInstanceOf[FsHistoryProviderUpdateState]
+ lookup(appId, attemptId) match {
+ case None =>
+ logDebug(s"Application Attempt $appId/$attemptId not found")
+ false
+ case Some(attempt) =>
+ updateState.version < attempt.version
--- End diff --
`updateTimeMillis` isn't used here at all. I'm not sure if it actually
needs to be checked, though -- it seems like just relying on versions is
enough? in that case, you could remove it completely from `isUpdated`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
