zhouyifan279 commented on code in PR #41105:
URL: https://github.com/apache/spark/pull/41105#discussion_r1189507968
##########
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala:
##########
@@ -48,11 +49,28 @@ private[history] class ApplicationCache(
val retainedApplications: Int,
val clock: Clock) extends Logging {
+ /**
+ * Keep track of SparkUIs in [[ApplicationCache#appCache]] and SparkUIs
removed from
+ * [[ApplicationCache#appCache]] but not detached yet.
+ */
+ private val loadedApps = new ConcurrentHashMap[CacheKey, CountDownLatch]()
+
private val appLoader = new CacheLoader[CacheKey, CacheEntry] {
/** the cache key doesn't match a cached entry, or the entry is
out-of-date, so load it. */
override def load(key: CacheKey): CacheEntry = {
- loadApplicationEntry(key.appId, key.attemptId)
+ // Ensure old SparkUI has been detached before loading new one.
Review Comment:
> I have some questions about this. Shouldn't the `CacheLoader#load` method
only be executed once for accessing the same key? Why is there a scenario where
the same key enters the load method more than once?
>
> Is this a bug of Guava 14.0.1 or did I have an incorrect understanding of
the mechanism of `CacheLoader`? Please correct me.
There is no bug on "`CacheLoader#load` method only be executed once for
accessing the same key".
The CountDownLatch is used to wait the completion of
`RemovalListener#onRemoval` of the old SparkUI with the same key.
`RemovalListener#onRemoval` and `CacheLoader#load` can execute concurrently
because `RemovalListener#onRemoval` is not guarded by LoadingCache's lock for
keys.
> In addition, from the Jira description, does this issue only exist in
Spark 3.1.2?
We found this issue in Spark 3.1.2 . But it should exists in a wide range of
Spark versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
