jerryshao commented on issue #13703:
URL: https://github.com/apache/gravitino/issues/13703#issuecomment-6285415650
The implementation in #13982 takes a different route from the one proposed
above: instead of adding a placeholder per submit-time configuration
(`{{spark_jars:-}}`, ...), the k8s job executor applies the `spark_conf` of a
built-in job when it submits it, so every submit-time configuration works
without new parameters, and existing `spark_conf` usages keep working.
- `spark.master` / `spark.submit.deployMode` and `spark.kubernetes.*` in
`spark_conf` are ignored, since whoever runs a job sets it; only the executor
configurations and job templates choose the namespace, image, service account,
Secrets, and volumes.
- The local job executor is unchanged: the driver still applies
`spark_conf`, so put jars such as the Iceberg runtime into `SPARK_HOME/jars`
there.
- `builtin-sparkpi` doesn't get a `spark_conf` parameter.
- Also covers the epic follow-up: a non-Kubernetes `spark.master` is now
rejected by the k8s job executor instead of being dropped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
