jerryshao opened a new issue, #13703:
URL: https://github.com/apache/gravitino/issues/13703
### Describe the subtask
The Spark configurations of a built-in job template are fixed. A job run can
only add Spark configurations through the `spark_conf` argument of the built-in
Iceberg jobs, which the job applies inside the driver when it creates the
`SparkSession`. That is too late for the configurations that must be set when
the job is submitted.
Verified with the `k8s` job executor on apache/spark-kubernetes-operator
0.9.0 and Spark 3.5.9:
- `spark.jars` in `spark_conf` doesn't put the jar on the driver classpath.
With an image that doesn't contain the Iceberg runtime, the job fails with
`Missing Iceberg Spark session extensions`, although the docs and that error
message suggest this usage.
- `spark.master` in `spark_conf`, as in the examples of the docs, overrides
the master set by the job executor. The job runs in local mode inside the
driver pod, never requests an executor, and still reports success.
- `spark.jars.packages` is resolved by `spark-submit` only, so it can't work
in `spark_conf` either.
- `builtin-sparkpi` has no `spark_conf` argument at all.
So a job run can't choose the Iceberg runtime jar. It has to be in the Spark
image, or set for all jobs with
`gravitino.jobExecutor.k8s.spark.conf.spark.jars` or
`gravitino.jobExecutor.k8s.spark.conf.spark.jars.packages`.
Proposed changes, in `maintenance/jobs` and `maintenance/optimizer-api`:
- Add placeholders with empty defaults to the Spark configurations of the
built-in templates, such as `spark.jars` = `{{spark_jars:-}}` and
`spark.jars.packages` = `{{spark_jars_packages:-}}`, and bump the template
versions.
- Make sure an empty value neither breaks the `local` job executor nor
overrides the default set in the `k8s` job executor configuration.
- Ignore `spark.master` and `spark.submit.deployMode` in `spark_conf` when
the job is submitted to a cluster.
- Fix the docs and the error message that recommend `spark.jars` in
`spark_conf`.
### Parent issue
#13554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
