Samra Pusina created SPARK-59613:
------------------------------------
Summary: [SHS] /static/sql handler not registered on all pods →
intermittent 404s on SparkSQL ExecutionPage
Key: SPARK-59613
URL: https://issues.apache.org/jira/browse/SPARK-59613
Project: Spark
Issue Type: Bug
Components: Bug, Spark Core
Affects Versions: 4.0.4
Reporter: Samra Pusina
h2. Summary
The SparkSQL tab on the {{ExecutionPage}} in the Spark History Server (SHS)
intermittently fails to render its execution graphs. {{GET /static/sql}}
returns {*}404 intermittently{*}, while core static assets (e.g.
{{{}/static/jquery-3.5.1.min.js{}}}) consistently return 200. The failure only
appears when *more than one SHS pod* is running.
h2. Problem
* The SparkSQL tab on {{ExecutionPage}} intermittently renders no execution
graphs.
* Browser network logs show {{GET /static/sql}} returning *404* intermittently.
* Other static assets served by the core UI (e.g.
{{{}/static/jquery-3.5.1.min.js{}}}) consistently return {*}200{*}.
* Manually loading the SQL tab once in the browser appeared to "fix" it — but
only temporarily/unreliably.
h2. Root Cause
{{SQLTab}} registers the {{/static/sql}} Jetty static handler in its
constructor:
{{}}
{code:java}
parent.addStaticHandler(SQLTab.STATIC_RESOURCE_DIR, "/static/sql") {code}
{{}}
{{SQLTab}} is only instantiated {*}per-application, lazily{*}, via
{{{}SQLHistoryServerPlugin.setupUI{}}}, when an app with SQL/DataFrame
executions is loaded into the SHS application cache. It is *not* part of the
core UI wired up at SHS startup (unlike the main {{/static}} handler registered
by {{{}HistoryServer{}}}).
Because {{SparkUI}} instances are cached per-application and evicted
({{{}ApplicationCache{}}}), the {{/static/sql}} handler's lifetime is tied to
whichever {{SparkUI}} registered it. When that app is evicted from the cache,
the handler is torn down with it, so {{/static/sql/*}} requests start 404ing —
until another SQL-bearing app is loaded and re-registers it. This produces the
intermittent, cache-eviction-dependent 404 pattern, and matches why manually
loading the SQL tab once "fixed" it (it triggered
{{{}SparkUI{}}}/{{{}SQLTab{}}} creation and handler registration for that
session, working until the next eviction).
h2. Why this only manifests with more than one SHS pod
The {{/static/sql}} handler lives in the *in-memory Jetty state of a single
pod* — the pod whose {{ApplicationCache}} happened to load a SQL-bearing app
and instantiate {{{}SQLTab{}}}. None of this state is shared across pods:
* Each SHS pod has its {*}own independent {{ApplicationCache}}{*}, its own
loaded {{SparkUI}} instances, and therefore its own set of
dynamically-registered handlers ({{{}/static/sql{}}} among them).
* The core {{/static}} handler is wired up at {{HistoryServer}} startup, so it
exists identically on *every* pod — which is why
{{/static/jquery-3.5.1.min.js}} and the other core assets always return 200
regardless of routing.
* {{/static/sql}} only exists on a pod that _currently_ has a SQL app loaded
and un-evicted.
With {*}multiple pods behind the Service/load balancer{*}, one browser page
load fans out into many requests distributed across pods with {*}divergent
handler state{*}:
# The {{ExecutionPage}} HTML is served by {*}pod A{*}, which has the app
loaded and {{/static/sql}} registered.
# The follow-up {{GET /static/sql/...}} for the SparkSQL graph assets is
routed to {*}pod B{*}, which never instantiated {{SQLTab}} (or has since
evicted that {{{}SparkUI{}}}). Pod B has no {{/static/sql}} handler → {*}404{*}.
This is what makes the pattern look random: it depends on which pod each
individual asset request lands on, not just on eviction timing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
