Suaroman opened a new pull request, #59924:
URL: https://github.com/apache/spark/pull/59924
### What changes were proposed in this pull request?
`AllJobsPage` (the Jobs page, which is the first page shown when an
application is opened in the History Server) now reads `spark.scheduler.mode`
defensively:
- The application environment is read with `store.asOption(...)`, so a
missing environment no longer fails the page. The scheduling mode is then shown
as `Unknown`, as it already was when the property is absent.
- A value that is not a known `SchedulingMode`, such as the redaction text
`*********(redacted)`, is shown as it is instead of throwing. A known value is
shown exactly as before: the upper-case enum name, so `fair` still shows as
`FAIR` (SPARK-33991).
The PR also adds `RedactedEnvironmentSuite`. It replays an application the
way the History Server does (`AppStatusListener` with `live = false`), builds a
`SparkUI` from the store and renders the Jobs page.
SPARK-60714 also covers `spark.task.cpus`: a redacted value made
`AppStatusListener.onEnvironmentUpdate` throw during replay, so the environment
was never stored. On master this is already handled by SPARK-58192, so this PR
only changes `AllJobsPage`. Release branches (3.5, 4.1, 4.2) still call
`.toInt` there. If backports are wanted, I have a branch-4.2 version of this
change that also parses `spark.task.cpus` with `toIntOption`, with a test in
`AppStatusListenerSuite`.
### Why are the changes needed?
`EventLoggingListener` writes every Spark property whose key or value
matches `spark.redaction.regex` as `*********(redacted)`.
`SparkEnv.environmentDetails` always includes `spark.scheduler.mode`, even when
it is not set. A broad regex, such as `.*` or one that contains `mode`,
therefore produces event logs with a redacted scheduling mode. The application
runs normally, but in the History Server `SchedulingMode.withName` throws and
the application UI returns HTTP 500 as soon as it is opened:
```
java.util.NoSuchElementException: No value found for '*********(REDACTED)'
at scala.Enumeration.withName(Enumeration.scala:159)
at
org.apache.spark.ui.jobs.AllJobsPage.$anonfun$render$2(AllJobsPage.scala:318)
at org.apache.spark.ui.jobs.AllJobsPage.render(AllJobsPage.scala:318)
```
This reproduces on 3.1.3, 3.3.4, 3.5.9, 4.0.5, 4.1.4, 4.2.0, 4.2.1 and
master. SPARK-60714 has the reproduction steps and the version matrix.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix in the History Server. An application whose event log has a
redacted (or otherwise unknown) `spark.scheduler.mode`, or no environment at
all, now opens normally. Before, the Jobs page returned HTTP 500. The Jobs page
now shows `Scheduling Mode: *********(redacted)` in that case. Known values
(`FIFO`, `FAIR`) are shown as before, and the live UI is unchanged.
### How was this patch tested?
New unit tests in `RedactedEnvironmentSuite`:
- the Jobs page renders with a redacted `spark.scheduler.mode` and shows the
value as it is
- a known mode is still shown in upper case (`fair` -> `FAIR`)
- the Jobs page renders when the application environment is missing
(`Unknown`)
- a redacted `spark.task.cpus` keeps the application environment
Without the change in `AllJobsPage`, 3 of the 4 tests fail with
`java.util.NoSuchElementException: No value found for '*********(REDACTED)'`
(or with no message, for the missing environment). With it, all pass.
```
build/sbt -Phadoop-3 core/scalastyle core/Test/scalastyle \
"core/testOnly org.apache.spark.ui.jobs.RedactedEnvironmentSuite
org.apache.spark.status.AppStatusListenerWithInMemoryStoreSuite"
```
On master `a28406de`: scalastyle found 0 errors, and 28 of 28 tests passed.
Manual end-to-end check: I wrote event logs for 6 redaction scenarios
(default regex, `.*`, `(?i)scheduler\.mode`, `spark.task.cpus=1` with
`(?i)task\.cpus` and with `.*`, and a broad multi-word regex) on a 4-worker
standalone cluster and in local mode. I then opened every UI page and REST
endpoint of each application in the History Server. Unpatched 4.2.0 and 4.2.1
return HTTP 500 on the Jobs page for every redacted scenario. With the
branch-4.2 version of this change (this fix plus the `spark.task.cpus`
parsing), all 252 requests returned 200.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5, model claude-opus-5-5)
The investigation, fix and tests were produced with Claude Code and reviewed
and verified by me: unit tests in Spark's sbt build, plus the History Server
checks described above.
This contribution is my original work and I license the work to the project
under the project's open source license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
