dongjoon-hyun opened a new pull request, #786:
URL: https://github.com/apache/spark-kubernetes-operator/pull/786
### What changes were proposed in this pull request?
This PR aims to add a new example, `group-by-with-recovery-mode.yaml`, which
demonstrates
`spark.kubernetes.allocation.recoveryMode.enabled` (SPARK-55639, Apache
Spark 4.2.0).
The example runs `GroupByTest` where each mapper task materializes about
450MiB. With
`spark.executor.cores=3` and `spark.executor.memory=1g`, three concurrent
tasks exceed the
executor heap and cause OOM. With the recovery mode enabled, every executor
accepts only a
single task per executor JVM (`SPARK_EXECUTOR_CORES=1` while the pod still
requests 3 CPUs),
so the whole heap serves one task and the job succeeds.
### Why are the changes needed?
To help users try out the recovery-mode executor feature of Apache Spark
4.2.0 with a
self-contained example that actually exercises the OOM boundary:
- As-is (`"true"`): the job succeeds with single-task executors.
- Set to `"false"`: the job fails with executor OOM (exit code 52).
- Removed: the recovery mode is activated automatically when the driver
detects the OOM.
### Does this PR introduce _any_ user-facing change?
No. This is an example addition.
### How was this patch tested?
Manually tested on a local Kubernetes cluster with the operator built from
this repository.
```
$ sed 's/{{SPARK_VERSION}}/4.2.0/' examples/group-by-with-recovery-mode.yaml
| kubectl apply -f -
```
- `enabled: "true"` (as-is): executor pods request 3 CPUs but run with
`SPARK_EXECUTOR_CORES=1`;
the application reached `Succeeded` and then `ResourceReleased`.
- `enabled: "false"`: all executors terminated with exit code 52 (`JVM OOM`)
and the job failed
after task retries, as described in the example comment.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
