Copilot commented on code in PR #13950: URL: https://github.com/apache/gravitino/pull/13950#discussion_r4321364983
########## docs/k8s-job-executor.md: ########## @@ -0,0 +1,701 @@ +--- +title: "Kubernetes Job Executor" +slug: "/k8s-job-executor" +keyword: "job executor, Kubernetes, k8s, Spark operator, SparkApplication, YuniKorn, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +The `k8s` job executor runs Spark jobs on Kubernetes. Each job becomes a `SparkApplication` resource +of the [Apache Spark Kubernetes Operator](https://github.com/apache/spark-kubernetes-operator), which +starts the driver and executor pods of the job. + +All the job state lives in Kubernetes, not in the Gravitino server. Jobs keep running while Gravitino +restarts, and every Gravitino server of a deployment can query and cancel any job. + +The `k8s` job executor only runs Spark templates. A run of a shell template is rejected. + +For job templates and the job API, see [Manage Jobs](./manage-jobs-in-gravitino.md). To run jobs as +processes on the Gravitino server instead, see [Local Job Executor](./local-job-executor.md). + +## Requirements + +- Apache Spark Kubernetes Operator 0.9.x in the Kubernetes cluster that runs the jobs, watching + every namespace the jobs run in. Operator 1.0 and later require Spark 4.0 or later, while the + built-in jobs of Gravitino are built for Spark 3.5. +- A Spark 3.5 image, see [Spark Image](#spark-image). +- Access from the Gravitino server to the API server of the cluster, with the permissions listed in + [Kubernetes Permissions](#kubernetes-permissions). + +The job executor is the `gravitino-k8s-job-executor-<version>.jar` file in the `libs` directory of +the Gravitino distribution. Nothing else needs to be installed on the Gravitino server. + +## Job Execution + +When a job runs, the job executor creates a `SparkApplication` from the job template and the job +configuration: + +| Job template | SparkApplication | +|-----------------------------|----------------------------------------------------------------------------------------| +| `executable` | `spec.jars`, or `spec.pyFiles` for a file that ends with `.py` | +| `className` | `spec.mainClass` | +| `arguments` | `spec.driverArgs` | +| `configs` | `spec.sparkConf` | +| `environments` | `spark.kubernetes.driverEnv.<name>` and `spark.executorEnv.<name>` in `spec.sparkConf` | +| `jars`, `files`, `archives` | Appended to `spark.jars`, `spark.files`, and `spark.archives` in `spec.sparkConf` | Review Comment: The SparkApplication field names in this mapping table look inconsistent with the Apache Spark Kubernetes Operator CRD (e.g., `spec.driverArgs` / `spec.jars` / `spec.pyFiles` may not be the correct fields for main application + args). This is likely to mislead users trying to correlate templates with the created SparkApplication. Please verify against the operator 0.9.x CRD and update the table to use the exact field names actually produced/used (and, if resources are primarily passed via Spark conf rather than SparkApplication `deps`, explicitly state that). ########## docs/k8s-job-executor.md: ########## @@ -0,0 +1,701 @@ +--- +title: "Kubernetes Job Executor" +slug: "/k8s-job-executor" +keyword: "job executor, Kubernetes, k8s, Spark operator, SparkApplication, YuniKorn, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +The `k8s` job executor runs Spark jobs on Kubernetes. Each job becomes a `SparkApplication` resource +of the [Apache Spark Kubernetes Operator](https://github.com/apache/spark-kubernetes-operator), which +starts the driver and executor pods of the job. + +All the job state lives in Kubernetes, not in the Gravitino server. Jobs keep running while Gravitino +restarts, and every Gravitino server of a deployment can query and cancel any job. + +The `k8s` job executor only runs Spark templates. A run of a shell template is rejected. + +For job templates and the job API, see [Manage Jobs](./manage-jobs-in-gravitino.md). To run jobs as +processes on the Gravitino server instead, see [Local Job Executor](./local-job-executor.md). + +## Requirements + +- Apache Spark Kubernetes Operator 0.9.x in the Kubernetes cluster that runs the jobs, watching + every namespace the jobs run in. Operator 1.0 and later require Spark 4.0 or later, while the + built-in jobs of Gravitino are built for Spark 3.5. +- A Spark 3.5 image, see [Spark Image](#spark-image). +- Access from the Gravitino server to the API server of the cluster, with the permissions listed in + [Kubernetes Permissions](#kubernetes-permissions). + +The job executor is the `gravitino-k8s-job-executor-<version>.jar` file in the `libs` directory of +the Gravitino distribution. Nothing else needs to be installed on the Gravitino server. + +## Job Execution + +When a job runs, the job executor creates a `SparkApplication` from the job template and the job +configuration: + +| Job template | SparkApplication | +|-----------------------------|----------------------------------------------------------------------------------------| +| `executable` | `spec.jars`, or `spec.pyFiles` for a file that ends with `.py` | +| `className` | `spec.mainClass` | +| `arguments` | `spec.driverArgs` | +| `configs` | `spec.sparkConf` | +| `environments` | `spark.kubernetes.driverEnv.<name>` and `spark.executorEnv.<name>` in `spec.sparkConf` | +| `jars`, `files`, `archives` | Appended to `spark.jars`, `spark.files`, and `spark.archives` in `spec.sparkConf` | + +The Spark configurations of a job come from three places, each overriding the ones before it: + +1. The server-level defaults, set with the `gravitino.jobExecutor.k8s.spark.conf.` prefix. +2. The `environments` of the job template. +3. The `configs` of the job template. + +The job executor then adjusts the result: + +- `spark.master` and `spark.submit.deployMode` are removed, as the operator decides them. The + operator runs Spark in cluster mode only, so a job whose `spark.submit.deployMode` is another + mode, such as `client`, is rejected. +- `spark.kubernetes.container.image` and `spark.kubernetes.authenticate.driver.serviceAccountName` + take the values of `spark.image` and `spark.serviceAccount` of the job executor if neither the + job template nor a server-level default sets them. A job template can therefore run with its + own image and service account. +- `spark.kubernetes.namespace` is the namespace that the `configs` of the job template set, or else + `namespace` of the job executor. A server-level default of it with the `spark.conf.` prefix is + ignored: set `namespace` instead. + +The `SparkApplication` is named `<namePrefix>-<job ID>`, for example +`gravitino-job-688822913391439376`, and carries the labels `app.kubernetes.io/managed-by=gravitino` +and `gravitino.apache.org/job-id=<job ID>`. List the jobs of a namespace with: + +```shell +kubectl get sparkapplications -l app.kubernetes.io/managed-by=gravitino +``` + +### Job Resources + +Gravitino doesn't fetch the resources of a job. It passes their URIs on to Spark, which fetches them +in the cluster. The `executable`, `jars`, `files`, and `archives` of a template must therefore be +URIs that the driver and executor pods can reach, such as `local://` for a file in the Spark image, +`https://`, `s3a://`, or `hdfs://`. So must every entry of `spark.jars`, `spark.files`, and +`spark.archives`, whether a job template or a server-level default sets them. A run with a path on +the Gravitino server, that is a path without a scheme or with the `file` scheme, is rejected. + +The built-in job templates are the exception. Their `executable` is the path of the built-in jobs +jar on the Gravitino server, and the job executor replaces it with `spark.builtinJobsJar`, by default +`local:///opt/gravitino/jobs/gravitino-jobs.jar`, a file in the [Spark image](#spark-image). + +### Built-In Jobs + +The built-in job templates set the resources of the driver and the executors in their own +`configs`, from placeholders. Size a built-in job with its job configuration, for example +`spark_executor_instances`, `spark_executor_cores`, `spark_executor_memory`, and +`spark_driver_memory`, not with server-level defaults: the `configs` of a template override them. +The `spark_master` parameter has no effect, as `spark.master` is removed. + +The `spark_conf` parameter of a built-in job is applied by the driver after it starts. Setting +`spark.jars` there has no effect, and setting `spark.master` there runs the job in local mode inside +the driver pod. Set such configurations as server-level defaults instead, which apply to every +Spark configuration that the built-in templates don't set, such as `spark.jars`, the scheduler +configurations, or volumes. + +### Rejected Runs + +A run is rejected, and no job is created, when: + +- The job template is a shell template. +- A resource of the job is a path on the Gravitino server, see [Job Resources](#job-resources). +- `spark.submit.deployMode` is another mode than `cluster`. +- `spark.kubernetes.namespace` isn't a valid namespace name, or an environment variable name is + invalid, see [Environment Variables](#environment-variables). +- Kubernetes rejects the `SparkApplication`, for example because the namespace doesn't exist, the + job executor lacks a permission, or the operator isn't installed. The error carries the reason + that Kubernetes gives. + +### Job Status + +The job status follows the state of the `SparkApplication`: + +| Job status | SparkApplication | +|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `QUEUED` | No state yet, `Submitted`, `DriverRequested`, or `ScheduledToRestart`. The driver hasn't started, for example because its pod waits for resources. | +| `STARTED` | `DriverStarted`, `DriverReady`, `RunningHealthy`, `RunningWithPartialCapacity`, `InitializedBelowThresholdExecutors`, or `RunningWithBelowThresholdExecutors` | +| `SUCCEEDED` | `Succeeded` | +| `FAILED` | `Failed`, `SchedulingFailure`, `DriverEvicted`, `DriverStartTimedOut`, `DriverReadyTimedOut`, or `ExecutorsStartTimedOut` | +| `CANCELLED` | Cancelled through Gravitino | + +A job also fails in these cases: + +- The operator writes no status for the `SparkApplication` within `noStatusTimeoutInMs`, for + example because the operator isn't running or doesn't watch the namespace. The job executor + deletes the `SparkApplication` before failing the job. +- The `SparkApplication` is deleted by someone else before the job finishes. + +After its outcome, a `SparkApplication` moves to `ResourceReleased` or +`TerminatedWithoutReleaseResources`. A job in these states keeps the outcome that the operator +recorded before them, or fails if there is none. + +A state that the job executor doesn't know, for example of a newer operator, keeps the job `STARTED` +if it has started and `QUEUED` otherwise, until the operator records an outcome. A status that the +job executor can't read keeps the job `QUEUED` until the operator deletes the `SparkApplication`. +Use a supported operator version. + +A job is one run of the application: the operator never restarts it. + +## Setup + +### Kubernetes Permissions + +The job executor needs these permissions in every namespace that jobs run in. It needs no +permission on Secrets or ConfigMaps, and no cluster-wide permission. + +| Resource | Verb | Used to | +|------------------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------| +| `sparkapplications` (`spark.apache.org`) | `create` | Submit a job | +| `sparkapplications` (`spark.apache.org`) | `get` | Query, submit, and cancel a job, and read its output | +| `sparkapplications` (`spark.apache.org`) | `list` | Query all the jobs of a namespace with one call. Optional: without it, the job executor gets the jobs one by one and logs a warning | +| `sparkapplications` (`spark.apache.org`) | `update` | Mark a job as cancelled by Gravitino | +| `sparkapplications` (`spark.apache.org`) | `delete` | Cancel a job, and delete the `SparkApplication` of a job that got no status or whose submission failed | +| `pods` | `list` | Find the driver pod, for the output and the times of a job | +| `pods` | `delete` | Delete the driver pod of a failed job that hasn't terminated, for example after the driver start timeout | +| `pods/log` | `get` | Read the output of a job | + +For example, in the namespace `default`: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gravitino-job-executor + namespace: default +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: gravitino-job-executor + namespace: default +rules: + - apiGroups: ["spark.apache.org"] + resources: ["sparkapplications"] + verbs: ["create", "get", "list", "update", "delete"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "delete"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: gravitino-job-executor + namespace: default +subjects: + - kind: ServiceAccount + name: gravitino-job-executor + namespace: default +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: gravitino-job-executor +``` + +The driver pod of a job runs with another service account, `spark.serviceAccount`, which Spark uses +to create the executor pods. The Helm chart of the operator creates the service account `spark` and +binds it to the cluster role `spark-workload-clusterrole` in the namespaces it is installed for, set +with `workloadResources.namespaces.data` of the chart. + +### Spark Image + +The image of the driver and the executors, `spark.image`, needs: + +- Spark 3.5, whose version is set as `spark.sparkVersion`. +- The built-in jobs jar of Gravitino at `/opt/gravitino/jobs/gravitino-jobs.jar`, to run the built-in + job templates. It is the `auxlib/gravitino-jobs-<version>.jar` file of the Gravitino distribution, + and must come from the same version as the Gravitino server. +- The Iceberg Spark runtime for Spark 3.5 and Scala 2.12, to run the built-in Iceberg jobs. Iceberg + 1.11 needs Java 17, so build the image from a Java 17 Spark image. + +For example: + +```dockerfile +FROM apache/spark:3.5.9-java17 +USER root +COPY gravitino-jobs-*.jar /opt/gravitino/jobs/gravitino-jobs.jar +ADD https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar /opt/spark/jars/ +RUN chmod 644 /opt/gravitino/jobs/gravitino-jobs.jar /opt/spark/jars/iceberg-spark-runtime-3.5_2.12-1.11.0.jar +USER spark +``` + +An image without the Iceberg Spark runtime also works if every job gets the runtime through a +server-level default. The driver of every job then downloads the runtime when it starts. For +example: + +```text +gravitino.jobExecutor.k8s.spark.conf.spark.jars = https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar +``` + +### Cluster Connection + +The cluster that runs the jobs can be the one Gravitino runs in or another one. The job executor +connects to it in one of these ways, in this order: + +| Way | Configurations | Notes | +|-----------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------| +| API server URL | `masterUrl`, `caCertFile`, `tokenFile` | The token file is read again whenever it changes, so the token can be rotated without restarting Gravitino. | +| Kubeconfig file | `kubeconfig`, `context` | Uses the current context of the file if `context` isn't set. | +| Default lookup | `context` | The `KUBECONFIG` environment variable or `~/.kube/config`, then the service account of the pod that Gravitino runs in. | + +The Gravitino server doesn't start if the connection is misconfigured: + +- `masterUrl` is set together with `kubeconfig` or `context`, or `caCertFile` or `tokenFile` is set + without `masterUrl`. +- `masterUrl` doesn't start with `https://` or `http://`. +- `masterUrl` starts with `http://` and `tokenFile` is set, which would send the token in clear + text, or `caCertFile` is set, which has no effect without TLS. +- A configured file doesn't exist, or the token file has no token. +- The kubeconfig is empty or can't be parsed, has no `context` of that name, or has no current + context while `context` isn't set, or the context selects no cluster with a server. +- The default lookup finds no kubeconfig, and Gravitino doesn't run in a pod. + +In every case, the jobs run in `namespace` unless their job template sets its own, not in the +namespace of the kubeconfig context or of the pod that Gravitino runs in. + +For example, to connect to another cluster with the service account created in +[Kubernetes Permissions](#kubernetes-permissions), issue a token for it in that cluster: + +```shell +kubectl create token gravitino-job-executor --duration=24h > /etc/gravitino/k8s/token +``` + +A token issued this way expires, and Gravitino doesn't refresh it. Have whatever provides the file +replace the token before it expires, for example by updating the Secret that the file is mounted +from. Then point the job executor to the API server, its CA certificate, and the token: + +```text +gravitino.jobExecutor.k8s.masterUrl = https://k8s.example.com:6443 +gravitino.jobExecutor.k8s.caCertFile = /etc/gravitino/k8s/ca.crt +gravitino.jobExecutor.k8s.tokenFile = /etc/gravitino/k8s/token +``` + +`cluster` names the cluster in the job execution ID that Gravitino stores for each job, +`<cluster>/sparkapp/<namespace>/<name>`. Set the same name on every Gravitino server, and don't +change it while jobs run: the jobs submitted under another name can't be queried or cancelled, and +stay unfinished until the name is changed back. + +If the API server can't be reached, each request is tried `1 + requestRetries` times, each waiting +up to `connectionTimeoutInMs`, or `requestTimeoutInMs` for an API server that accepts the request +but doesn't answer. With the defaults, a job submission or cancellation fails after about 15 +seconds if the API server can't be reached, or 30 seconds if it doesn't answer. After a status +query fails this way, the status queries of the other jobs fail right away for 10 seconds, and +Gravitino tries them again on its next status pull. + +### Enable the Job Executor + +Set the job executor and its three required configurations in the `gravitino.conf` file, then +restart the Gravitino server: + +```text +gravitino.job.executor = k8s +gravitino.jobExecutor.k8s.namespace = <the namespace of the jobs> +gravitino.jobExecutor.k8s.spark.image = <the Spark image> +gravitino.jobExecutor.k8s.spark.sparkVersion = 3.5.9 +``` + +## Configuration + +Set the following configurations in the `gravitino.conf` file. An invalid value stops the Gravitino +server from starting. + +| Property name | Description | Default value | Required | +|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------|----------| +| `gravitino.jobExecutor.k8s.cluster` | The name of the cluster that runs the jobs, a part of the job execution ID. It can't contain `/`, and must be the same on every Gravitino server | `default` | No | +| `gravitino.jobExecutor.k8s.kubeconfig` | The kubeconfig file of the cluster | (none) | No | +| `gravitino.jobExecutor.k8s.context` | The kubeconfig context to use | The current context | No | +| `gravitino.jobExecutor.k8s.masterUrl` | The URL of the API server of the cluster, an alternative to the kubeconfig. It starts with `https://`, or with `http://` without `caCertFile` and `tokenFile` | (none) | No | +| `gravitino.jobExecutor.k8s.caCertFile` | The CA certificate file of the API server, used with `masterUrl` | (none) | No | +| `gravitino.jobExecutor.k8s.tokenFile` | The file of the bearer token to access the API server, used with `masterUrl`. It is read again whenever it changes | (none) | No | +| `gravitino.jobExecutor.k8s.namespace` | The namespace of the jobs whose job template doesn't set `spark.kubernetes.namespace` | (none) | Yes | +| `gravitino.jobExecutor.k8s.namePrefix` | The prefix of the `SparkApplication` names: at most 24 lowercase letters, digits, and `-`, starting with a letter and ending with a letter or a digit | `gravitino-job` | No | +| `gravitino.jobExecutor.k8s.connectionTimeoutInMs` | How long in milliseconds to wait to connect to the API server | `5000` (5 seconds) | No | +| `gravitino.jobExecutor.k8s.requestTimeoutInMs` | How long in milliseconds to wait for the response of a request to the API server | `10000` (10 seconds) | No | +| `gravitino.jobExecutor.k8s.requestRetries` | How many times a request to the API server is tried again when the server can't be reached, doesn't answer in time, or answers with a server error | `2` | No | +| `gravitino.jobExecutor.k8s.statusCacheTtlInMs` | How long in milliseconds the `SparkApplication` resources listed in a namespace are cached for status queries. Zero or less disables the cache | `5000` (5 seconds) | No | +| `gravitino.jobExecutor.k8s.noStatusTimeoutInMs` | How long in milliseconds a job waits for the operator to report the first state of its `SparkApplication` before it fails | `600000` (10 minutes) | No | +| `gravitino.jobExecutor.k8s.spark.image` | The image of the Spark driver and executors, unless a job sets `spark.kubernetes.container.image` | (none) | Yes | +| `gravitino.jobExecutor.k8s.spark.sparkVersion` | The Spark version of the image | (none) | Yes | +| `gravitino.jobExecutor.k8s.spark.serviceAccount` | The service account of the Spark driver, unless a job sets `spark.kubernetes.authenticate.driver.serviceAccountName` | `spark` | No | +| `gravitino.jobExecutor.k8s.spark.resourceRetainDurationInMs` | How long in milliseconds the pods of a finished job are retained. The output of a job can be read during this time | `86400000` (1 day) | No | +| `gravitino.jobExecutor.k8s.spark.ttlAfterStopInMs` | How long in milliseconds the `SparkApplication` of a job is kept after the job stops | `604800000` (7 days) | No | +| `gravitino.jobExecutor.k8s.spark.driverStartTimeoutInMs` | How long in milliseconds the operator waits for the driver pod to start before the job fails. It includes the time the driver pod waits for resources or in a scheduler queue | `3600000` (1 hour) | No | +| `gravitino.jobExecutor.k8s.spark.driverReadyTimeoutInMs` | How long in milliseconds the operator waits for the started driver to become ready before the job fails | `3600000` (1 hour) | No | +| `gravitino.jobExecutor.k8s.spark.builtinJobsJar` | The URI of the built-in jobs jar that the Spark pods can reach, see [Job Resources](#job-resources) | `local:///opt/gravitino/jobs/gravitino-jobs.jar` | No | +| `gravitino.jobExecutor.k8s.spark.conf.<key>` | A default Spark configuration `<key>` of every job, see [Job Execution](#job-execution) | (none) | No | + +## Queueing + +The job executor creates the `SparkApplication` of a job as soon as the job is submitted. It doesn't +limit the number of unfinished jobs and has no queue of its own, so a burst of jobs can exhaust the +cluster: + +- A driver pod that doesn't fit stays pending, and its job stays `QUEUED` until + `spark.driverStartTimeoutInMs`, then fails. +- A job whose driver runs but whose executors can't be scheduled stays `STARTED` and makes no + progress. + +To queue jobs, run them with a scheduler that has queues, such as +[Apache YuniKorn](https://yunikorn.apache.org/). + +### YuniKorn + +Send every job to a YuniKorn queue with three server-level Spark configurations. No annotation with +the application ID is needed: YuniKorn groups the driver and the executors of a job into one +application. + +```text +gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.scheduler.name = yunikorn +gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.driver.label.queue = root.gravitino +gravitino.jobExecutor.k8s.spark.conf.spark.kubernetes.executor.label.queue = root.gravitino +``` + +A job template can also set these Spark configurations in its `configs`, to run in a queue of its +own. + +Limit the queue in the YuniKorn configuration, the `queues.yaml` entry of the ConfigMap +`yunikorn-configs`: + +```yaml +partitions: + - name: default + queues: + - name: root + submitacl: '*' + queues: + - name: gravitino + maxapplications: 10 + resources: + max: + vcore: 40 + memory: 160Gi +``` + +- `maxapplications` limits the jobs that run at the same time. The driver pods of the other jobs + stay pending and their jobs are `QUEUED`. The next job starts a few seconds after the pods of a + running job end. +- `resources.max` limits the resources of the pods in the queue, not the number of jobs. A driver + pod is admitted as soon as it fits, even if its executors don't: such a job is `STARTED` and + waits for the executors of other jobs to end. Combine it with `maxapplications`, so that the + admitted jobs can all get their executors. +- Cancelling a queued job removes it from the queue. +- A job that waits in the queue for longer than `spark.driverStartTimeoutInMs` fails. Set this + timeout longer than the longest time a job may wait. + +The job executor doesn't depend on a scheduler: queueing works through the Spark configurations +above. The behaviors above were observed with YuniKorn 1.10.0 installed without its admission +controller (`embedAdmissionController=false`). + +## Environment Variables + +The `environments` of a job template are set for the driver and the executors as plain Spark +configurations: a variable `REGION` becomes `spark.kubernetes.driverEnv.REGION` and +`spark.executorEnv.REGION`. A name must consist of letters, digits, `_`, `-`, and `.`, can't start +with a digit or `..`, and can't be `.`. Otherwise, the run is rejected. Review Comment: The documented allowed character set for environment variable names (`-` and `.` allowed) conflicts with Kubernetes' EnvVar name constraints (typically `[A-Za-z_][A-Za-z0-9_]*`, i.e., no `-` or `.`). Since these names ultimately become container environment variables, documenting a broader set is likely to cause failed pods/SparkApplications. Recommend aligning this section with Kubernetes' actual validation rules (and matching whatever validation the executor performs). ########## docs/k8s-job-executor.md: ########## @@ -0,0 +1,701 @@ +--- +title: "Kubernetes Job Executor" +slug: "/k8s-job-executor" +keyword: "job executor, Kubernetes, k8s, Spark operator, SparkApplication, YuniKorn, Gravitino" +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +The `k8s` job executor runs Spark jobs on Kubernetes. Each job becomes a `SparkApplication` resource +of the [Apache Spark Kubernetes Operator](https://github.com/apache/spark-kubernetes-operator), which +starts the driver and executor pods of the job. + +All the job state lives in Kubernetes, not in the Gravitino server. Jobs keep running while Gravitino +restarts, and every Gravitino server of a deployment can query and cancel any job. + +The `k8s` job executor only runs Spark templates. A run of a shell template is rejected. + +For job templates and the job API, see [Manage Jobs](./manage-jobs-in-gravitino.md). To run jobs as +processes on the Gravitino server instead, see [Local Job Executor](./local-job-executor.md). + +## Requirements + +- Apache Spark Kubernetes Operator 0.9.x in the Kubernetes cluster that runs the jobs, watching + every namespace the jobs run in. Operator 1.0 and later require Spark 4.0 or later, while the + built-in jobs of Gravitino are built for Spark 3.5. +- A Spark 3.5 image, see [Spark Image](#spark-image). +- Access from the Gravitino server to the API server of the cluster, with the permissions listed in + [Kubernetes Permissions](#kubernetes-permissions). + +The job executor is the `gravitino-k8s-job-executor-<version>.jar` file in the `libs` directory of +the Gravitino distribution. Nothing else needs to be installed on the Gravitino server. + +## Job Execution + +When a job runs, the job executor creates a `SparkApplication` from the job template and the job +configuration: + +| Job template | SparkApplication | +|-----------------------------|----------------------------------------------------------------------------------------| +| `executable` | `spec.jars`, or `spec.pyFiles` for a file that ends with `.py` | +| `className` | `spec.mainClass` | +| `arguments` | `spec.driverArgs` | +| `configs` | `spec.sparkConf` | +| `environments` | `spark.kubernetes.driverEnv.<name>` and `spark.executorEnv.<name>` in `spec.sparkConf` | +| `jars`, `files`, `archives` | Appended to `spark.jars`, `spark.files`, and `spark.archives` in `spec.sparkConf` | + +The Spark configurations of a job come from three places, each overriding the ones before it: + +1. The server-level defaults, set with the `gravitino.jobExecutor.k8s.spark.conf.` prefix. +2. The `environments` of the job template. +3. The `configs` of the job template. + +The job executor then adjusts the result: + +- `spark.master` and `spark.submit.deployMode` are removed, as the operator decides them. The + operator runs Spark in cluster mode only, so a job whose `spark.submit.deployMode` is another + mode, such as `client`, is rejected. +- `spark.kubernetes.container.image` and `spark.kubernetes.authenticate.driver.serviceAccountName` + take the values of `spark.image` and `spark.serviceAccount` of the job executor if neither the + job template nor a server-level default sets them. A job template can therefore run with its + own image and service account. +- `spark.kubernetes.namespace` is the namespace that the `configs` of the job template set, or else + `namespace` of the job executor. A server-level default of it with the `spark.conf.` prefix is + ignored: set `namespace` instead. + +The `SparkApplication` is named `<namePrefix>-<job ID>`, for example +`gravitino-job-688822913391439376`, and carries the labels `app.kubernetes.io/managed-by=gravitino` +and `gravitino.apache.org/job-id=<job ID>`. List the jobs of a namespace with: + +```shell +kubectl get sparkapplications -l app.kubernetes.io/managed-by=gravitino +``` + +### Job Resources + +Gravitino doesn't fetch the resources of a job. It passes their URIs on to Spark, which fetches them +in the cluster. The `executable`, `jars`, `files`, and `archives` of a template must therefore be +URIs that the driver and executor pods can reach, such as `local://` for a file in the Spark image, +`https://`, `s3a://`, or `hdfs://`. So must every entry of `spark.jars`, `spark.files`, and +`spark.archives`, whether a job template or a server-level default sets them. A run with a path on +the Gravitino server, that is a path without a scheme or with the `file` scheme, is rejected. + +The built-in job templates are the exception. Their `executable` is the path of the built-in jobs +jar on the Gravitino server, and the job executor replaces it with `spark.builtinJobsJar`, by default +`local:///opt/gravitino/jobs/gravitino-jobs.jar`, a file in the [Spark image](#spark-image). + +### Built-In Jobs + +The built-in job templates set the resources of the driver and the executors in their own +`configs`, from placeholders. Size a built-in job with its job configuration, for example +`spark_executor_instances`, `spark_executor_cores`, `spark_executor_memory`, and +`spark_driver_memory`, not with server-level defaults: the `configs` of a template override them. +The `spark_master` parameter has no effect, as `spark.master` is removed. + +The `spark_conf` parameter of a built-in job is applied by the driver after it starts. Setting +`spark.jars` there has no effect, and setting `spark.master` there runs the job in local mode inside +the driver pod. Set such configurations as server-level defaults instead, which apply to every +Spark configuration that the built-in templates don't set, such as `spark.jars`, the scheduler +configurations, or volumes. + +### Rejected Runs + +A run is rejected, and no job is created, when: + +- The job template is a shell template. +- A resource of the job is a path on the Gravitino server, see [Job Resources](#job-resources). +- `spark.submit.deployMode` is another mode than `cluster`. +- `spark.kubernetes.namespace` isn't a valid namespace name, or an environment variable name is + invalid, see [Environment Variables](#environment-variables). +- Kubernetes rejects the `SparkApplication`, for example because the namespace doesn't exist, the + job executor lacks a permission, or the operator isn't installed. The error carries the reason + that Kubernetes gives. + +### Job Status + +The job status follows the state of the `SparkApplication`: + +| Job status | SparkApplication | +|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `QUEUED` | No state yet, `Submitted`, `DriverRequested`, or `ScheduledToRestart`. The driver hasn't started, for example because its pod waits for resources. | +| `STARTED` | `DriverStarted`, `DriverReady`, `RunningHealthy`, `RunningWithPartialCapacity`, `InitializedBelowThresholdExecutors`, or `RunningWithBelowThresholdExecutors` | +| `SUCCEEDED` | `Succeeded` | +| `FAILED` | `Failed`, `SchedulingFailure`, `DriverEvicted`, `DriverStartTimedOut`, `DriverReadyTimedOut`, or `ExecutorsStartTimedOut` | +| `CANCELLED` | Cancelled through Gravitino | + +A job also fails in these cases: + +- The operator writes no status for the `SparkApplication` within `noStatusTimeoutInMs`, for + example because the operator isn't running or doesn't watch the namespace. The job executor + deletes the `SparkApplication` before failing the job. +- The `SparkApplication` is deleted by someone else before the job finishes. + +After its outcome, a `SparkApplication` moves to `ResourceReleased` or +`TerminatedWithoutReleaseResources`. A job in these states keeps the outcome that the operator +recorded before them, or fails if there is none. + +A state that the job executor doesn't know, for example of a newer operator, keeps the job `STARTED` +if it has started and `QUEUED` otherwise, until the operator records an outcome. A status that the +job executor can't read keeps the job `QUEUED` until the operator deletes the `SparkApplication`. +Use a supported operator version. + +A job is one run of the application: the operator never restarts it. + +## Setup + +### Kubernetes Permissions + +The job executor needs these permissions in every namespace that jobs run in. It needs no +permission on Secrets or ConfigMaps, and no cluster-wide permission. + +| Resource | Verb | Used to | +|------------------------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------| +| `sparkapplications` (`spark.apache.org`) | `create` | Submit a job | +| `sparkapplications` (`spark.apache.org`) | `get` | Query, submit, and cancel a job, and read its output | +| `sparkapplications` (`spark.apache.org`) | `list` | Query all the jobs of a namespace with one call. Optional: without it, the job executor gets the jobs one by one and logs a warning | +| `sparkapplications` (`spark.apache.org`) | `update` | Mark a job as cancelled by Gravitino | +| `sparkapplications` (`spark.apache.org`) | `delete` | Cancel a job, and delete the `SparkApplication` of a job that got no status or whose submission failed | +| `pods` | `list` | Find the driver pod, for the output and the times of a job | +| `pods` | `delete` | Delete the driver pod of a failed job that hasn't terminated, for example after the driver start timeout | +| `pods/log` | `get` | Read the output of a job | + +For example, in the namespace `default`: + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gravitino-job-executor + namespace: default +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: gravitino-job-executor + namespace: default +rules: + - apiGroups: ["spark.apache.org"] + resources: ["sparkapplications"] + verbs: ["create", "get", "list", "update", "delete"] + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "delete"] + - apiGroups: [""] + resources: ["pods/log"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: gravitino-job-executor + namespace: default +subjects: + - kind: ServiceAccount + name: gravitino-job-executor + namespace: default +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: gravitino-job-executor +``` + +The driver pod of a job runs with another service account, `spark.serviceAccount`, which Spark uses +to create the executor pods. The Helm chart of the operator creates the service account `spark` and +binds it to the cluster role `spark-workload-clusterrole` in the namespaces it is installed for, set +with `workloadResources.namespaces.data` of the chart. + +### Spark Image + +The image of the driver and the executors, `spark.image`, needs: + +- Spark 3.5, whose version is set as `spark.sparkVersion`. +- The built-in jobs jar of Gravitino at `/opt/gravitino/jobs/gravitino-jobs.jar`, to run the built-in + job templates. It is the `auxlib/gravitino-jobs-<version>.jar` file of the Gravitino distribution, + and must come from the same version as the Gravitino server. +- The Iceberg Spark runtime for Spark 3.5 and Scala 2.12, to run the built-in Iceberg jobs. Iceberg + 1.11 needs Java 17, so build the image from a Java 17 Spark image. + +For example: + +```dockerfile +FROM apache/spark:3.5.9-java17 +USER root +COPY gravitino-jobs-*.jar /opt/gravitino/jobs/gravitino-jobs.jar +ADD https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.11.0/iceberg-spark-runtime-3.5_2.12-1.11.0.jar /opt/spark/jars/ +RUN chmod 644 /opt/gravitino/jobs/gravitino-jobs.jar /opt/spark/jars/iceberg-spark-runtime-3.5_2.12-1.11.0.jar +USER spark Review Comment: The example Dockerfile uses `ADD` to download a Maven artifact at build time without integrity verification. This makes builds non-reproducible and introduces supply-chain risk if the fetched content is tampered with or if the download is intercepted. Prefer documenting an approach that pins and verifies the artifact (e.g., download with a tool that supports checksum verification and validate SHA-256), or instruct users to vendor the jar into the build context and copy it in. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
