Noufal Rijal created SPARK-58893:
------------------------------------
Summary: Spark UI generates root-relative links in multi-cluster
standalone deployments with path-based reverse proxy
Key: SPARK-58893
URL: https://issues.apache.org/jira/browse/SPARK-58893
Project: Spark
Issue Type: Bug
Components: Spark Core, Web UI
Affects Versions: 4.1.2, 4.2.0, 4.0.0
Reporter: Noufal Rijal
h3. Problem
When multiple Spark standalone clusters are deployed behind a single reverse
proxy using path-based routing (e.g., {{{}/spark-ui/\{cluster-name}/{}}}), the
Spark UI is fully functional only for the first cluster. All other clusters
produce 403 errors on tab navigation and lose CSS styling.
This regression exists even after SPARK-58521 (PR #57549) which added
{{X-Forwarded-Context}} header forwarding.
*Configuration:*
{{spark.ui.reverseProxy=true
spark.ui.reverseProxyUrl=https://proxy.example.com/spark-ui/cluster-a}}
*Reverse proxy routing:*
{{/spark-ui/cluster-a/ → cluster-a master:8080
/spark-ui/cluster-b/ → cluster-b master:8080}}
*Observed behavior:*
# Navigating to
{{https://proxy.example.com/spark-ui/cluster-b/proxy/app-xxx/}} *redirects* to
{{https://proxy.example.com/proxy/app-xxx/jobs/}} — missing the cluster prefix.
The proxy cannot route this URL.
# After the redirect, clicking any tab (SQL, Stages, Executors) generates
links like {{href="/proxy/app-xxx/sql/"}} — root-relative, unroutable through
path-based ingress.
h3. Root Cause (three separate bugs)
*Bug 1 — {{Master.createDriver}} does not propagate {{reverseProxyUrl}} to
cluster-mode driver JVM*
In cluster deploy mode, the worker starts the driver as a subprocess (new JVM).
{{SparkContext}} (line ~640) computes {{{}spark.ui.proxyBase = reverseProxyUrl
+ "/proxy/" + appId{}}}. However, {{reverseProxyUrl}} is read from the driver's
{{{}SparkConf{}}}, which does not contain it — the config only exists in the
master/worker JVM, not in the submitted application's conf. As a result
{{proxyBase}} is computed as {{/proxy/\{appId}}} (root-relative), and all
driver UI links are root-relative.
*Bug 2 — {{JettyUtils.createProxyLocationHeader}} strips the cluster prefix
from redirect {{Location}} headers*
When the browser first opens the driver UI root ({{{}/proxy/\{appId}{}}}), the
driver returns a {{302}} to {{{}/jobs/{}}}. The master's
{{filterServerResponseHeader}} rewrites this via {{createProxyLocationHeader}}
to {{https://proxy.example.com/proxy/\{appId}/jobs/}} — which is correct for a
single-cluster deployment but omits the {{/spark-ui/cluster-b/}} prefix in a
multi-cluster path-routed setup. The browser follows the root-relative redirect
and subsequent navigation is broken.
*Bug 3 — {{JettyUtils.createProxyHandler}} does not include {{reverseProxyUrl}}
in {{X-Forwarded-Context}}*
SPARK-58521 added {{X-Forwarded-Context}} forwarding, but the header value is
{{contextPath + prefix}} (e.g., {{{}/proxy/app-xxx{}}}) — a root-relative path.
For the driver to generate correct absolute links via
{{{}UIUtils.uiRoot(request){}}}, the header must contain the full URL including
the cluster prefix:
{{{}https://proxy.example.com/spark-ui/cluster-b/proxy/app-xxx{}}}. This
matters for client-mode drivers that cannot receive the fix via
{{{}javaOpts{}}}.
h3. Steps to Reproduce
# Start two Spark standalone clusters with {{spark.ui.reverseProxy=true}} and
distinct {{spark.ui.reverseProxyUrl}} values
# Configure path-based routing so {{/spark-ui/cluster-a/}} and
{{/spark-ui/cluster-b/}} route to their respective masters
# Submit a streaming application to cluster-b in cluster deploy mode
# Open the Spark master UI and click the driver application link
# Observe the redirect URL loses the {{/spark-ui/cluster-b/}} prefix
# Click any UI tab — observe root-relative URLs causing 403 errors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
