dongjoon-hyun commented on code in PR #51604:
URL: https://github.com/apache/spark/pull/51604#discussion_r2223846826
##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -364,6 +375,16 @@ private[history] class FsHistoryProvider(conf: SparkConf,
clock: Clock)
Some(loadedUI)
}
+ private def loadFromFallbackLocation(appId: String, attemptId:
Option[String], logPath: String)
+ : ApplicationInfoWrapper = {
+ val date = new Date(0)
+ val info = ApplicationAttemptInfo(attemptId, date, date, date, 0, "spark",
false, "unknown")
+ addListing(new ApplicationInfoWrapper(
+ ApplicationInfo(appId, appId, None, None, None, None, List.empty),
+ List(new AttemptInfoWrapper(info, logPath, 0, Some(1), None, None, None,
None))))
Review Comment:
It sounds like a limitation of a single file event log, @thejdeep . If you
have rolling event logs, SHS have the correct partial information already while
your jobs are running.
> For example, we noticed that a good fraction of our SHS requests are on
demand since users would like to get their reports as soon as their app
finishes and before checkForLogs completes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
