jerryshao opened a new pull request, #13725: URL: https://github.com/apache/gravitino/pull/13725
### What changes were proposed in this pull request? > **This PR is stacked on part 1 (#13724)** and stays a draft until it merges. Until then the diff also shows the commit of part 1. To review only this PR, see the [compare view](https://github.com/jerryshao/gravitino/compare/13613-k8s-executor-module...13613-spark-application). This is part 2 of 3 of #13613, which adds a job executor named `k8s` that runs Spark jobs on Kubernetes. This part adds how the executor talks to [apache/spark-kubernetes-operator](https://github.com/apache/spark-kubernetes-operator) 0.9.x. Both classes are plain functions that don't call Kubernetes: - **`SparkApplicationUtils`** builds the `SparkApplication` of a Spark job template. - The executable, main class, arguments, jars, files, archives and Spark configs go into the SparkApplication. - `spark.master` and `spark.submit.deployMode` are removed, as the operator owns them. - The image, driver service account and namespace come from the executor config as defaults, and the template can override them. - Resources must be URIs that the cluster can reach: paths on the Gravitino server are rejected. - Templates with environment variables are rejected until #13617 passes them through Secrets. - **`SparkApplicationStatusUtils`** maps the state of a SparkApplication to the job status. - The final states `ResourceReleased` and `TerminatedWithoutReleaseResources` walk back through the state history to the last outcome. - A SparkApplication without any status after a timeout fails. - An unknown state, of a newer operator, keeps the job unfinished. - The started and finished times come from the driver container, because the operator records a state some time after it happens. ### Why are the changes needed? See #13554. The executor in part 3 (#13700) uses these to submit jobs and to report their status. Part of: #13613 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests. The state mapping was also checked against operator 0.9.0 on a local Kubernetes cluster, as part of the manual tests of #13700. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
