[
https://issues.apache.org/jira/browse/SPARK-60714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60714:
-----------------------------------
Labels: pull-request-available (was: )
> History Server UI fails with HTTP 500 when spark.redaction.regex redacts
> spark.scheduler.mode or spark.task.cpus
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60714
> URL: https://issues.apache.org/jira/browse/SPARK-60714
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Affects Versions: 3.1.3, 3.3.4, 4.2.0, 3.5.9, 4.3.0, 5.0.0, 4.2.1, 4.0.5,
> 4.1.4
> Environment: Spark standalone cluster, Spark 4.2.0 (Scala 2.13.18), 1
> master + 4 workers, Ubuntu 26.04, OpenJDK 17.0.20; History Server reading
> event logs from a file system directory.
> Also each release 3.1.3 (Java 11), 3.3.4, 3.5.9, 4.0.5, 4.1.4, 4.2.0, 4.2.1
> (Java 17) in local mode, each release's event logs opened in its own History
> Server.
> Reporter: Gregory J. Suarez
> Priority: Major
> Labels: pull-request-available
> Attachments: after-4.2.0-hotfix-jobs-page.png,
> before-4.2.0-jobs-page-500.png
>
>
> h3. Problem
> When {{spark.redaction.regex}} matches the key or the value of
> {{spark.scheduler.mode}} or {{spark.task.cpus}}, the application runs
> normally, but its UI in the Spark History Server fails with HTTP 500 as soon
> as the application is opened. This happens with a broad regex (for example
> {{spark.redaction.regex=.*}}, or one containing words such as {{mode}}),
> which is a reasonable choice for users who want to redact almost everything.
> Reproduced on 3.1.3, 3.3.4, 3.5.9, 4.0.5, 4.1.4, 4.2.0 and 4.2.1. Cause 1
> below is also present on master (5.0.0-SNAPSHOT, commit e3f4817f1a) and
> branch-4.3: a unit test fails there without the fix.
> h3. Cause
> {{EventLoggingListener.redactEvent}} writes the
> {{SparkListenerEnvironmentUpdate}} event with each matching property value
> replaced by the redaction text ({{Utils.REDACTION_REPLACEMENT_TEXT}}). The
> running application and its live UI use the real values, so nothing fails at
> run time. The History Server rebuilds the UI from the event log and parses
> two of these values as if they were real.
> h4. 1. Jobs page: spark.scheduler.mode
> {{AllJobsPage.render}} (the Jobs page, the first page shown for an
> application) calls {{SchedulingMode.withName}} on {{spark.scheduler.mode}}.
> {{SparkEnv.environmentDetails}} always adds this property, even when the user
> did not set it, so any regex that matches the key or the value ({{FIFO}} /
> {{FAIR}}) breaks the page:
> {noformat}
> java.util.NoSuchElementException: No value found for '*********(REDACTED)'
> at scala.Enumeration.withName(Enumeration.scala:159)
> at
> org.apache.spark.ui.jobs.AllJobsPage.$anonfun$render$2(AllJobsPage.scala:318)
> at scala.Option.map(Option.scala:242)
> at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:318)
> at org.apache.spark.ui.WebUI.$anonfun$attachPage$1(WebUI.scala:91)
> at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:86)
> {noformat}
> SPARK-33991 (3.2.0) fixed the same call for lower-case values only.
> h4. 2. Replay: spark.task.cpus
> When the job sets {{spark.task.cpus}},
> {{AppStatusListener.onEnvironmentUpdate}} calls {{.toInt}} on it. During
> replay this throws. {{ReplayListenerBus}} logs the error and continues, but
> the {{kvstore.write(new ApplicationEnvironmentInfoWrapper(envInfo))}} on the
> next line never runs:
> {noformat}
> ERROR ReplayListenerBus: Listener AppStatusListener threw an exception
> java.lang.NumberFormatException: For input string: "*********(redacted)"
> at scala.collection.StringOps$.toInt$extension(StringOps.scala:907)
> at
> org.apache.spark.status.AppStatusListener.$anonfun$onEnvironmentUpdate$6(AppStatusListener.scala:186)
> at
> org.apache.spark.status.AppStatusListener.onEnvironmentUpdate(AppStatusListener.scala:186)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:89)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3(FsHistoryProvider.scala:1436)
> {noformat}
> Because the environment is missing, the Jobs page then fails in
> {{AppStatusStore.environmentInfo}} ({{NoSuchElementException}} at
> {{AllJobsPage.scala:316}}), {{/api/v1/applications/<app-id>/environment}}
> returns 404, and the Environment page returns 500 (3.x to 4.1) or never
> leaves "Loading..." (4.2).
> On master and branch-4.3 this second cause is already avoided by SPARK-58192
> (fractional spark.task.cpus, which parses the value with
> {{CpuAmount.parseUntrusted}}). That change is not in any release: branch-3.5,
> branch-4.1, branch-4.2 and v4.2.1 still call {{.map(_.toInt)}}.
> h3. Steps to reproduce
> Local mode is enough, with any release:
> {noformat}
> bin/spark-sql --master 'local[2]' --name redacted \
> --conf spark.eventLog.enabled=true --conf spark.eventLog.dir=file:///tmp/ev
> \
> --conf 'spark.redaction.regex=.*' \
> -e 'SELECT count(*) FROM range(1000)'
> # The query succeeds. Start a History Server with
> spark.history.fs.logDirectory=file:///tmp/ev
> # and open http://<host>:18080/history/<app-id>/ -> HTTP 500 (cause 1)
> # Cause 2: instead use
> # --conf spark.task.cpus=1 --conf 'spark.redaction.regex=(?i)task\.cpus'
> # -> NumberFormatException in the History Server log, Jobs page 500, REST
> /environment 404
> {noformat}
> h3. Results
> Each release wrote its own event logs and opened them in its own History
> Server. The same results came from jobs submitted to a 4-worker standalone
> cluster running 4.2.0. The table shows the HTTP status of
> {{/history/<app-id>/}}, which redirects to the Jobs page.
> Scenarios ({{spark-submit --conf}}):
> {noformat}
> s0 default spark.redaction.regex
> s1 spark.redaction.regex=.*
> s2 spark.redaction.regex=(?i)scheduler\.mode
> s3 spark.task.cpus=1 spark.redaction.regex=(?i)task\.cpus
> s4 spark.task.cpus=1 spark.redaction.regex=.*
> s5
> spark.redaction.regex=(?i)secret|password|token|access[.]?key|account|user|host|address|url|uri|path|dir|name|id|mode|port
> {noformat}
> ||Scenario||3.1.3||3.3.4||3.5.9||4.0.5||4.1.4||4.2.0||4.2.1||
> |s0|200|200|200|200|200|200|200|
> |s1|500|500|500|500|500|500|500|
> |s2|500|500|500|500|500|500|500|
> |s3|500|500|500|500|500|500|500|
> |s4|500|500|500|500|500|500|500|
> |s5|500|500|500|500|500|500|500|
> h3. Proposed fix
> Treat environment values read back from an event log as untrusted text:
> * {{AllJobsPage}}: read the environment with {{store.asOption(...)}}. Show a
> known scheduling mode as its enum name, as before. Show any other value as it
> is, so a redacted value appears as the redaction text. Show "Unknown" when
> the environment is missing.
> * {{AppStatusListener}} (release branches only, since master already has
> SPARK-58192): parse {{spark.task.cpus}} with {{toIntOption}}. Keep the
> previous default for anything that is not a positive integer, and log a
> warning, so the environment is always stored.
> I will open a pull request with the fix and a new
> {{RedactedEnvironmentSuite}}. The suite replays an application with
> {{AppStatusListener(live = false)}}, builds a {{SparkUI}} from the store and
> renders the Jobs page. Without the fix the new tests fail on master (3 of 4)
> and on v4.2.0 (4 of 5, including a new {{AppStatusListenerSuite}} test). With
> the fix all of them pass. With the backport applied to a 4.2.0 and a 4.2.1
> History Server, all 252 UI and REST requests in the scenarios above returned
> 200.
> h3. Workaround
> Use a redaction regex that matches neither {{spark.scheduler.mode}} /
> {{spark.task.cpus}} nor their values, for example the default plus specific
> words, instead of {{.*}} or words like {{spark}}, {{mode}} or {{task}}. Event
> logs that are already written cannot be opened until the History Server is
> fixed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
