[
https://issues.apache.org/jira/browse/SPARK-57601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57601:
-----------------------------------
Labels: pull-request-available (was: )
> Spark Connect UI tab is unavailable in the Spark History Server
> ---------------------------------------------------------------
>
> Key: SPARK-57601
> URL: https://issues.apache.org/jira/browse/SPARK-57601
> Project: Spark
> Issue Type: Bug
> Components: Connect, Web UI
> Affects Versions: 4.1.2
> Reporter: Jiwon Park
> Priority: Major
> Labels: pull-request-available
>
> The Spark Connect UI page (SPARK-44394) works on a live driver but never
> appears in the Spark History Server (SHS). Two separate problems are involved.
> h3. 1. The plugin is not registered for the History Server
> {{SparkConnectServerHistoryServerPlugin}} implements
> {{AppHistoryServerPlugin}}, but it is not declared in
> {{META-INF/services/org.apache.spark.status.AppHistoryServerPlugin}} (the
> SQL, Streaming and Hive Thrift Server plugins are). As a result the History
> Server never discovers it via {{ServiceLoader}}, even though the class is on
> the classpath. The live UI works only because {{SparkConnectService}}
> registers the tab and listener directly.
> h3. 2. The listener throws NPE during History Server replay
> Once the plugin is registered, {{FsHistoryProvider.rebuildAppStore}} calls
> {{AppHistoryServerPlugin.createListeners}} -> {{new
> SparkConnectServerListener}}, whose constructor reads configuration via
> {{SparkEnv.get.conf}}. There is no active {{SparkEnv}} in the History Server
> ({{SparkEnv.get}} returns null), so it throws an NPE and the whole
> application UI rebuild fails with HTTP 500:
> {code}
> java.lang.NullPointerException: Cannot invoke
> "org.apache.spark.SparkEnv.conf()" because the return value of
> "org.apache.spark.SparkEnv$.get()" is null
> at
> org.apache.spark.sql.connect.ui.SparkConnectServerListener.<init>(SparkConnectServerListener.scala:49)
> at
> org.apache.spark.sql.connect.ui.SparkConnectServerHistoryServerPlugin.createListeners(SparkConnectServerHistoryServerPlugin.scala:30)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:1189)
> {code}
> The listener already receives a {{SparkConf}} through its constructor, so it
> should read from that instead of {{SparkEnv.get.conf}}.
> h3. How to reproduce
> Enable {{spark.eventLog.enabled}}, run a Spark Connect session so Connect
> events are written to the event log, then open that application in the
> History Server. Every page returns HTTP 500.
> h3. Proposed fix
> * Register {{SparkConnectServerHistoryServerPlugin}} via the
> {{AppHistoryServerPlugin}} SPI.
> * Use the constructor's {{sparkConf}} instead of {{SparkEnv.get.conf}} in
> {{SparkConnectServerListener}}.
> Both are needed for the Spark Connect tab to render from event logs in the
> History Server. This affects all 4.x releases (and master).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
