This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 0c00c54583fe [SPARK-46330] Loading of Spark UI blocks for a long time
when HybridStore enabled
0c00c54583fe is described below
commit 0c00c54583fe3e56f940425aac6e0e4f05c4b9db
Author: zhouyifan279 <[email protected]>
AuthorDate: Wed Dec 20 16:50:38 2023 +0800
[SPARK-46330] Loading of Spark UI blocks for a long time when HybridStore
enabled
### What changes were proposed in this pull request?
Move `LoadedAppUI` invalidate operation out of `FsHistoryProvider`
synchronized block.
### Why are the changes needed?
When closing a HybridStore of a `LoadedAppUI` with a lot of data waiting to
be written to disk, loading of other Spark UIs will be blocked for a long time.
See more details at https://issues.apache.org/jira/browse/SPARK-46330
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Passed existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #44260 from zhouyifan279/SPARK-46330.
Authored-by: zhouyifan279 <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
(cherry picked from commit cf54e8f9a51bf54e8fa3e1011ac370e46134b134)
Signed-off-by: Kent Yao <[email protected]>
---
.../org/apache/spark/deploy/history/FsHistoryProvider.scala | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
index 49b479f3124e..387bc7d9e45b 100644
---
a/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
+++
b/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
@@ -925,11 +925,12 @@ private[history] class FsHistoryProvider(conf: SparkConf,
clock: Clock)
* UI lifecycle.
*/
private def invalidateUI(appId: String, attemptId: Option[String]): Unit = {
- synchronized {
- activeUIs.get((appId, attemptId)).foreach { ui =>
- ui.invalidate()
- ui.ui.store.close()
- }
+ val uiOption = synchronized {
+ activeUIs.get((appId, attemptId))
+ }
+ uiOption.foreach { ui =>
+ ui.invalidate()
+ ui.ui.store.close()
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
