Jiwon Park created SPARK-57601:
----------------------------------
Summary: Spark Connect UI tab is unavailable in the Spark History
Server
Key: SPARK-57601
URL: https://issues.apache.org/jira/browse/SPARK-57601
Project: Spark
Issue Type: Bug
Components: Web UI, Connect
Affects Versions: 4.1.2
Reporter: Jiwon Park
The Spark Connect UI page (SPARK-44394) works on a live driver but never
appears in the Spark History Server (SHS). Two separate problems are involved.
h3. 1. The plugin is not registered for the History Server
{{SparkConnectServerHistoryServerPlugin}} implements
{{AppHistoryServerPlugin}}, but it is not declared in
{{META-INF/services/org.apache.spark.status.AppHistoryServerPlugin}} (the SQL,
Streaming and Hive Thrift Server plugins are). As a result the History Server
never discovers it via {{ServiceLoader}}, even though the class is on the
classpath. The live UI works only because {{SparkConnectService}} registers the
tab and listener directly.
h3. 2. The listener throws NPE during History Server replay
Once the plugin is registered, {{FsHistoryProvider.rebuildAppStore}} calls
{{AppHistoryServerPlugin.createListeners}} -> {{new
SparkConnectServerListener}}, whose constructor reads configuration via
{{SparkEnv.get.conf}}. There is no active {{SparkEnv}} in the History Server
({{SparkEnv.get}} returns null), so it throws an NPE and the whole application
UI rebuild fails with HTTP 500:
{code}
java.lang.NullPointerException: Cannot invoke
"org.apache.spark.SparkEnv.conf()" because the return value of
"org.apache.spark.SparkEnv$.get()" is null
at
org.apache.spark.sql.connect.ui.SparkConnectServerListener.<init>(SparkConnectServerListener.scala:49)
at
org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin.createListeners(SparkConnectServerHistoryServerPlugin.scala:30)
at
org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:1189)
{code}
The listener already receives a {{SparkConf}} through its constructor, so it
should read from that instead of {{SparkEnv.get.conf}}.
h3. How to reproduce
Enable {{spark.eventLog.enabled}}, run a Spark Connect session so Connect
events are written to the event log, then open that application in the History
Server. Every page returns HTTP 500.
h3. Proposed fix
* Register {{SparkConnectServerHistoryServerPlugin}} via the
{{AppHistoryServerPlugin}} SPI.
* Use the constructor's {{sparkConf}} instead of {{SparkEnv.get.conf}} in
{{SparkConnectServerListener}}.
Both are needed for the Spark Connect tab to render from event logs in the
History Server. This affects all 4.x releases (and master).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
