jerryshao opened a new pull request, #13892:
URL: https://github.com/apache/gravitino/pull/13892
### What changes were proposed in this pull request?
The `k8s` job executor added in #13613 can now be used by Gravitino, and
runs the built-in Spark jobs.
- **Registration**: `JobExecutorFactory` registers `k8s` by class name, as
the executor is a plugin that depends on `core`. The name and class name are
public constants of `JobExecutorFactory`, which the plugin uses, and a plugin
test checks the class name against `K8sJobExecutor`. If the class can't be
loaded, the error names the missing `gravitino-k8s-job-executor-<version>.jar`
and the `libs` directory.
- **Distribution**: the shaded jar of the executor is copied into `libs` of
the distribution package. `LICENSE.bin` already lists everything it bundles
(fabric8, zjsonpatch, Jackson, SnakeYAML, SnakeYAML Engine).
- **Smoke test of the shaded jar**: a separate `shadedJarTest` source set
and task, part of `check`, puts the shaded jar on the runtime classpath of
`:server` (with its Jackson 2.15.2), without the module's classes or unshaded
dependencies. It checks that Jackson and SnakeYAML are relocated, that the jar
bundles no class that is also on the server classpath, and creates the executor
through `JobExecutorFactory` to submit a built-in job to a stub API server. It
only needs `shadowJar`, not a distribution build.
- **Built-in jobs**: the executable of a built-in template is the path of
the jobs jar on the Gravitino server, which a Spark pod can't reach. The new
`spark.builtinJobsJar` is the URI of the jobs jar that the driver and executors
can reach, `local:///opt/gravitino/jobs/gravitino-jobs.jar` by default: a file
in the Spark image or in a mounted volume, or `https://`, `s3a://` and so on. A
template whose name starts with `builtin-` and whose executable is a path on
the server runs that jar instead. An executable that the cluster can already
reach is kept. The stored templates don't change, and the `local` executor
keeps using the jar on the server.
- The check of whether a resource is reachable from the cluster moves to
`K8sJobResourceUtils.isClusterReachable`, so that the resource check, the
`spark.builtinJobsJar` validation and the substitution share it.
### Why are the changes needed?
Without this, `gravitino.job.executor = k8s` fails to load the executor, and
the built-in jobs can't run on it.
Fix: #13616
### Does this PR introduce _any_ user-facing change?
Yes.
- `gravitino.job.executor = k8s` selects the `k8s` job executor, configured
with the `gravitino.jobExecutor.k8s.` keys added in #13613.
- New key `gravitino.jobExecutor.k8s.spark.builtinJobsJar`, default
`local:///opt/gravitino/jobs/gravitino-jobs.jar`.
- The distribution package ships
`libs/gravitino-k8s-job-executor-<version>.jar`.
The `gravitino-spark` image that holds the jobs jar at the default path, and
the docs, come with #13618.
### How was this patch tested?
- Unit tests for the factory, the new configuration and the substitution,
and the `shadedJarTest` smoke test.
- Manually, with a Gravitino server started from the distribution package of
this branch against a local Kubernetes cluster (OrbStack) with
spark-kubernetes-operator 0.9.0, through the REST API:
- the built-in Spark Pi job: success with its times and driver output,
cancel, a failing job, and a user template whose executable is a path on the
server, which is rejected;
- the jobs jar given as an `http://` link through `spark.builtinJobsJar`;
- built-in Iceberg jobs on an image without the Iceberg runtime, with the
runtime given through `gravitino.jobExecutor.k8s.spark.conf.spark.jars` or
`spark.conf.spark.jars.packages` (with `spark.jars.ivy`). Passing `spark.jars`
in the job's `spark_conf` argument doesn't work, which #13703 addresses.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
