This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 9791e6e08613 [SPARK-57601][CONNECT][UI] Make the Spark Connect tab
available in the History Server
9791e6e08613 is described below
commit 9791e6e08613e29ab6a4cf140c369d183e23d616
Author: Jiwon Park <[email protected]>
AuthorDate: Tue Jun 23 22:48:46 2026 +0800
[SPARK-57601][CONNECT][UI] Make the Spark Connect tab available in the
History Server
### What changes were proposed in this pull request?
This PR makes the Spark Connect UI tab render in the Spark History Server
(SHS). Two changes:
1. Register `SparkConnectServerHistoryServerPlugin` via the
`AppHistoryServerPlugin` SPI (a new
`META-INF/services/org.apache.spark.status.AppHistoryServerPlugin` resource in
the `connect/server` module), so that SHS discovers it through `ServiceLoader`,
as the SQL, Streaming and Hive Thrift Server plugins already do.
2. Make `SparkConnectServerListener` read its configuration from the
`SparkConf` passed to its constructor instead of `SparkEnv.get.conf`.
### Why are the changes needed?
The Spark Connect UI page added in SPARK-44394 works on a live driver but
never appears in the History Server:
- `SparkConnectServerHistoryServerPlugin` is implemented but is not
registered in `META-INF/services`, so SHS never loads it via `ServiceLoader`.
The live UI works only because `SparkConnectService` registers the tab and
listener directly.
- Even once the plugin is registered, the listener's constructor reads
configuration via `SparkEnv.get.conf`. There is no active `SparkEnv` during SHS
replay, so `SparkEnv.get` returns `null` and the listener throws an NPE in
`FsHistoryProvider.rebuildAppStore`, failing the whole application UI with HTTP
500:
```
java.lang.NullPointerException: Cannot invoke
"org.apache.spark.SparkEnv.conf()" because the return value of
"org.apache.spark.SparkEnv$.get()" is null
at
org.apache.spark.sql.connect.ui.SparkConnectServerListener.<init>(SparkConnectServerListener.scala)
at
org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin.createListeners(SparkConnectServerHistoryServerPlugin.scala)
at
org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala)
```
### Does this PR introduce _any_ user-facing change?
Yes. The Spark Connect tab (sessions / executions) now renders in the
History Server when replaying event logs of a Spark Connect application.
Previously the tab was never shown there. This is a change relative to released
versions and master.
### How was this patch tested?
- Added a unit test in `SparkConnectServerListenerSuite` that constructs
the listener with no active `SparkEnv` (simulating History Server replay). It
reproduces the NPE before the fix and passes after.
- Updated the existing test helper to set the UI-retention configs on the
`SparkConf` passed to the listener (previously set on `SparkEnv.get.conf`,
which the listener no longer reads).
- Manually verified on a History Server that the Connect tab renders from
Spark Connect event logs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #56682 from j1wonpark/SPARK-57601.
Authored-by: Jiwon Park <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit dc1a5fe0fed08af17895c5435594f58a7eec96d5)
Signed-off-by: Cheng Pan <[email protected]>
---
.../org.apache.spark.status.AppHistoryServerPlugin | 18 ++++++++++++++++
.../connect/ui/SparkConnectServerListener.scala | 6 ++----
.../ui/SparkConnectServerListenerSuite.scala | 25 +++++++++++++++++++++-
3 files changed, 44 insertions(+), 5 deletions(-)
diff --git
a/sql/connect/server/src/main/resources/META-INF/services/org.apache.spark.status.AppHistoryServerPlugin
b/sql/connect/server/src/main/resources/META-INF/services/org.apache.spark.status.AppHistoryServerPlugin
new file mode 100644
index 000000000000..5091829841fe
--- /dev/null
+++
b/sql/connect/server/src/main/resources/META-INF/services/org.apache.spark.status.AppHistoryServerPlugin
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
index 98dccc6c9a6c..01a775f89744 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.{ConcurrentHashMap, ConcurrentMap}
import scala.collection.mutable
import scala.collection.mutable.ArrayBuffer
-import org.apache.spark.{SparkConf, SparkContext, SparkEnv}
+import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.{OP_ID, SESSION_ID}
import org.apache.spark.internal.config.Status.LIVE_ENTITY_UPDATE_PERIOD
@@ -45,9 +45,7 @@ private[connect] class SparkConnectServerListener(
new ConcurrentHashMap[String, LiveExecutionData]
private val (retainedStatements: Int, retainedSessions: Int) = {
- (
- SparkEnv.get.conf.get(CONNECT_UI_STATEMENT_LIMIT),
- SparkEnv.get.conf.get(CONNECT_UI_SESSION_LIMIT))
+ (sparkConf.get(CONNECT_UI_STATEMENT_LIMIT),
sparkConf.get(CONNECT_UI_SESSION_LIMIT))
}
// How often to update live entities. -1 means "never update" when replaying
applications,
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
index c9c110dd1e62..e9cd7c692373 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
@@ -210,6 +210,30 @@ class SparkConnectServerListenerSuite
listener.onOtherEvent(SparkListenerConnectOperationClosed(unknownJob,
"operationId", 0))
}
+ test("SPARK-57601: listener can be created without an active SparkEnv
(history server)") {
+ // The History Server replays event logs without a SparkContext/SparkEnv,
so
+ // SparkEnv.get returns null there. The listener must read its config from
the
+ // SparkConf passed to its constructor rather than from SparkEnv.get.conf.
+ val previousEnv = SparkEnv.get
+ try {
+ SparkEnv.set(null)
+ val sparkConf = new SparkConf()
+ .set(ASYNC_TRACKING_ENABLED, false)
+ .set(LIVE_ENTITY_UPDATE_PERIOD, 0L)
+ .set(CONNECT_UI_SESSION_LIMIT, 1)
+ .set(CONNECT_UI_STATEMENT_LIMIT, 10)
+ val store = new ElementTrackingStore(new InMemoryStore, sparkConf)
+ try {
+ val listener = new SparkConnectServerListener(store, sparkConf, live =
false)
+ assert(listener.noLiveData())
+ } finally {
+ store.close(false)
+ }
+ } finally {
+ SparkEnv.set(previousEnv)
+ }
+ }
+
private def createProperties: Properties = {
val properties = new Properties()
properties.setProperty(SparkContext.SPARK_JOB_TAGS, jobTag)
@@ -221,7 +245,6 @@ class SparkConnectServerListenerSuite
sparkConf
.set(ASYNC_TRACKING_ENABLED, false)
.set(LIVE_ENTITY_UPDATE_PERIOD, 0L)
- SparkEnv.get.conf
.set(CONNECT_UI_SESSION_LIMIT, 1)
.set(CONNECT_UI_STATEMENT_LIMIT, 10)
kvstore = new ElementTrackingStore(new InMemoryStore, sparkConf)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
