jerryshao opened a new pull request, #13700: URL: https://github.com/apache/gravitino/pull/13700
### What changes were proposed in this pull request? > **This PR is stacked on #13563** and stays a draft until it merges. Until then the diff also shows the commits of #13563. To review only this PR, see the [compare view](https://github.com/jerryshao/gravitino/compare/13555-job-resource-utils...13554-k8s-job-executor). Adds a job executor named `k8s` in a new module `plugins/k8s-job-executor`. It runs SPARK job templates as `SparkApplication` resources of [apache/spark-kubernetes-operator](https://github.com/apache/spark-kubernetes-operator) 0.9.x through the fabric8 Kubernetes client. It rejects SHELL templates for now. - **Module**: a shaded jar with fabric8 kubernetes-client 7.2.0 and its JDK HTTP client. Jackson and the YAML libraries are relocated to `org.apache.gravitino.k8s.shaded`. - **Connection** (`K8sClientUtils`): a kubeconfig and context, or an API server URL with a CA certificate and a token file. The token file is read again when it changes. Without either, fabric8's default lookup applies, which covers running inside the job cluster. - **Job identity** (`K8sJobExecutionId`, `K8sJobResourceUtils`): - The SparkApplication is named `<namePrefix>-<jobId>` and labeled with the Gravitino job id. - The job execution id is `<cluster>/sparkapp/<namespace>/<name>`. - On a conflict, an existing SparkApplication is adopted only if it belongs to the same job. - **Submission** (`SparkApplicationUtils`): - The executable, main class, arguments, jars, files, archives and Spark configs go into the SparkApplication. - `spark.master` and `spark.submit.deployMode` are removed, as the operator owns them. - The image, driver service account and namespace come from the executor config as defaults, and the template can override them. - Resources must be URIs that the cluster can reach: paths on the Gravitino server are rejected. - Templates with environment variables are rejected until #13617 passes them through Secrets. - **Status** (`SparkApplicationStatusUtils`): - Operator states map to Gravitino statuses. The final states `ResourceReleased` and `TerminatedWithoutReleaseResources` walk back through the state history to the last outcome. - The started and finished times come from the driver container, because the operator records a state some time after it happens. - A SparkApplication without any status after `noStatusTimeoutMs` is deleted and the job fails. - A failed job whose driver pod hasn't terminated gets that pod deleted, so it can't run later. - **Status cache** (`NamespacedResourceCache`): the SparkApplications of a namespace are listed once and cached for `statusCacheTtlMs`. A miss falls back to a GET. - **Cancel**: the SparkApplication gets a cancel-requested annotation, then is deleted with `Foreground` propagation. The job is `CANCELLED` once it is being deleted or has stopped. - **Output** (`PodLogUtils`): stdout is the tail of the driver pod log. Kubernetes merges stderr into it, so stderr is empty. - `LICENSE.bin` lists the new bundled dependencies. The executor isn't registered in `JobExecutorFactory` or shipped in the distribution yet. That is #13616. ### Why are the changes needed? The only built-in job executor, `local`, runs jobs as child processes of the Gravitino server, which isn't suitable for production. This is the first step of #13554: an executor that runs Spark jobs on Kubernetes and keeps all the job state there, so any Gravitino server can submit, query and cancel any job. Fix: #13613 ### Does this PR introduce _any_ user-facing change? Yes, new configurations under `gravitino.jobExecutor.k8s.`. They only take effect once the executor is registered in #13616. - Connection: `cluster`, `kubeconfig`, `context`, `masterUrl`, `caCertFile`, `tokenFile` - Jobs: `namespace`, `namePrefix`, `statusCacheTtlMs`, `noStatusTimeoutMs` - Spark: `spark.image` and `spark.sparkVersion` (both required), `spark.serviceAccount`, `spark.resourceRetainDurationMs`, `spark.ttlAfterStopMs`, `spark.driverStartTimeoutMs`, `spark.driverReadyTimeoutMs`, `spark.conf.*` ### How was this patch tested? - Unit tests in `plugins/k8s-job-executor`. The executor tests run against the fabric8 mock server. - Manually on a local Kubernetes cluster (OrbStack) with operator 0.9.0, driving the executor directly: a successful job, a failing job, a cancelled job, a job whose driver can't be scheduled, and reading the driver output. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
