Gregory J. Suarez created SPARK-60714:
-----------------------------------------
Summary: History Server UI fails with HTTP 500 when
spark.redaction.regex redacts spark.scheduler.mode or spark.task.cpus
Key: SPARK-60714
URL: https://issues.apache.org/jira/browse/SPARK-60714
Project: Spark
Issue Type: Bug
Components: Spark Core, Web UI
Affects Versions: 4.1.4, 4.0.5, 4.2.0, 3.3.4, 3.1.3, 3.5.9, 4.3.0, 5.0.0,
4.2.1
Environment: Spark standalone cluster, Spark 4.2.0 (Scala 2.13.18), 1
master + 4 workers, Ubuntu 26.04, OpenJDK 17.0.20; History Server reading event
logs from a file system directory.
Also each release 3.1.3 (Java 11), 3.3.4, 3.5.9, 4.0.5, 4.1.4, 4.2.0, 4.2.1
(Java 17) in local mode, each release's event logs opened in its own History
Server.
Reporter: Gregory J. Suarez
h3. Problem
When {{spark.redaction.regex}} matches the key or the value of
{{spark.scheduler.mode}} or {{spark.task.cpus}}, the application runs normally,
but its UI in the Spark History Server fails with HTTP 500 as soon as the
application is opened. This happens with a broad regex (for example
{{spark.redaction.regex=.*}}, or one containing words such as {{mode}}), which
is a reasonable choice for users who want to redact almost everything.
Reproduced on 3.1.3, 3.3.4, 3.5.9, 4.0.5, 4.1.4, 4.2.0 and 4.2.1. Cause 1 below
is also present on master (5.0.0-SNAPSHOT, commit e3f4817f1a) and branch-4.3: a
unit test fails there without the fix.
h3. Cause
{{EventLoggingListener.redactEvent}} writes the
{{SparkListenerEnvironmentUpdate}} event with each matching property value
replaced by the redaction text ({{Utils.REDACTION_REPLACEMENT_TEXT}}). The
running application and its live UI use the real values, so nothing fails at
run time. The History Server rebuilds the UI from the event log and parses two
of these values as if they were real.
h4. 1. Jobs page: spark.scheduler.mode
{{AllJobsPage.render}} (the Jobs page, the first page shown for an application)
calls {{SchedulingMode.withName}} on {{spark.scheduler.mode}}.
{{SparkEnv.environmentDetails}} always adds this property, even when the user
did not set it, so any regex that matches the key or the value ({{FIFO}} /
{{FAIR}}) breaks the page:
{noformat}
java.util.NoSuchElementException: No value found for '*********(REDACTED)'
at scala.Enumeration.withName(Enumeration.scala:159)
at
org.apache.spark.ui.jobs.AllJobsPage.$anonfun$render$2(AllJobsPage.scala:318)
at scala.Option.map(Option.scala:242)
at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:318)
at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:91)
at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:86)
{noformat}
SPARK-33991 (3.2.0) fixed the same call for lower-case values only.
h4. 2. Replay: spark.task.cpus
When the job sets {{spark.task.cpus}},
{{AppStatusListener.onEnvironmentUpdate}} calls {{.toInt}} on it. During replay
this throws. {{ReplayListenerBus}} logs the error and continues, but the
{{kvstore.write(new ApplicationEnvironmentInfoWrapper(envInfo))}} on the next
line never runs:
{noformat}
ERROR ReplayListenerBus: Listener AppStatusListener threw an exception
java.lang.NumberFormatException: For input string: "*********(redacted)"
at scala.collection.StringOps$.toInt$extension(StringOps.scala:907)
at
org.apache.spark.status.AppStatusListener.$anonfun$onEnvironmentUpdate$6(AppStatusListener.scala:186)
at
org.apache.spark.status.AppStatusListener.onEnvironmentUpdate(AppStatusListener.scala:186)
at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:89)
at
org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3(FsHistoryProvider.scala:1436)
{noformat}
Because the environment is missing, the Jobs page then fails in
{{AppStatusStore.environmentInfo}} ({{NoSuchElementException}} at
{{AllJobsPage.scala:316}}), {{/api/v1/applications/<app-id>/environment}}
returns 404, and the Environment page returns 500 (3.x to 4.1) or never leaves
"Loading..." (4.2).
On master and branch-4.3 this second cause is already avoided by SPARK-58192
(fractional spark.task.cpus, which parses the value with
{{CpuAmount.parseUntrusted}}). That change is not in any release: branch-3.5,
branch-4.1, branch-4.2 and v4.2.1 still call {{.map(_.toInt)}}.
h3. Steps to reproduce
Local mode is enough, with any release:
{noformat}
bin/spark-sql --master 'local[2]' --name redacted \
--conf spark.eventLog.enabled=true --conf spark.eventLog.dir=file:///tmp/ev \
--conf 'spark.redaction.regex=.*' \
-e 'SELECT count(*) FROM range(1000)'
# The query succeeds. Start a History Server with
spark.history.fs.logDirectory=file:///tmp/ev
# and open http://<host>:18080/history/<app-id>/ -> HTTP 500 (cause 1)
# Cause 2: instead use
# --conf spark.task.cpus=1 --conf 'spark.redaction.regex=(?i)task\.cpus'
# -> NumberFormatException in the History Server log, Jobs page 500, REST
/environment 404
{noformat}
h3. Results
Each release wrote its own event logs and opened them in its own History
Server. The same results came from jobs submitted to a 4-worker standalone
cluster running 4.2.0. The table shows the HTTP status of
{{/history/<app-id>/}}, which redirects to the Jobs page.
Scenarios ({{spark-submit --conf}}):
{noformat}
s0 default spark.redaction.regex
s1 spark.redaction.regex=.*
s2 spark.redaction.regex=(?i)scheduler\.mode
s3 spark.task.cpus=1 spark.redaction.regex=(?i)task\.cpus
s4 spark.task.cpus=1 spark.redaction.regex=.*
s5
spark.redaction.regex=(?i)secret|password|token|access[.]?key|account|user|host|address|url|uri|path|dir|name|id|mode|port
{noformat}
||Scenario||3.1.3||3.3.4||3.5.9||4.0.5||4.1.4||4.2.0||4.2.1||
|s0|200|200|200|200|200|200|200|
|s1|500|500|500|500|500|500|500|
|s2|500|500|500|500|500|500|500|
|s3|500|500|500|500|500|500|500|
|s4|500|500|500|500|500|500|500|
|s5|500|500|500|500|500|500|500|
h3. Proposed fix
Treat environment values read back from an event log as untrusted text:
* {{AllJobsPage}}: read the environment with {{store.asOption(...)}}. Show a
known scheduling mode as its enum name, as before. Show any other value as it
is, so a redacted value appears as the redaction text. Show "Unknown" when the
environment is missing.
* {{AppStatusListener}} (release branches only, since master already has
SPARK-58192): parse {{spark.task.cpus}} with {{toIntOption}}. Keep the previous
default for anything that is not a positive integer, and log a warning, so the
environment is always stored.
I will open a pull request with the fix and a new {{RedactedEnvironmentSuite}}.
The suite replays an application with {{AppStatusListener(live = false)}},
builds a {{SparkUI}} from the store and renders the Jobs page. Without the fix
the new tests fail on master (3 of 4) and on v4.2.0 (4 of 5, including a new
{{AppStatusListenerSuite}} test). With the fix all of them pass. With the
backport applied to a 4.2.0 and a 4.2.1 History Server, all 252 UI and REST
requests in the scenarios above returned 200.
h3. Workaround
Use a redaction regex that matches neither {{spark.scheduler.mode}} /
{{spark.task.cpus}} nor their values, for example the default plus specific
words, instead of {{.*}} or words like {{spark}}, {{mode}} or {{task}}. Event
logs that are already written cannot be opened until the History Server is
fixed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
