jerryshao opened a new pull request, #13982: URL: https://github.com/apache/gravitino/pull/13982
### What changes were proposed in this pull request? - k8s job executor: for a `builtin-` template, parse its `--spark-conf` argument (`spark_conf`) and merge it into the SparkApplication `spec.sparkConf` (server `spark.conf.*` defaults < environments < template `configs` < `spark_conf`). In `spark_conf`, `spark.master` / `spark.submit.deployMode` and all `spark.kubernetes.*` keys are ignored with a warning; an invalid `spark_conf` rejects the run. - k8s job executor: reject a `spark.master` that isn't a `k8s://` master (from the template or the executor defaults), like a client deploy mode, instead of silently dropping it. - Blank Spark configuration values mean "unset": the local job executor doesn't pass them to spark-submit, and the k8s job executor doesn't let them override its defaults. - Built-in templates: `spark_master` defaults to empty instead of `local[*]` (spark-submit's own default), template versions bumped. - Built-in jobs (driver): ignore `spark.master` / `spark.submit.deployMode` from `spark_conf` once spark-submit has set them, drop `spark.kubernetes.*` and blank values. - Optimizer `submit-update-stats-job`: no longer defaults `spark_master` to `local[*]` or takes it from `spark.master` in `spark_conf`. - Docs and the "Missing Iceberg Spark session extensions" error message no longer recommend `spark.jars` in `spark_conf` for the local job executor. ### Why are the changes needed? `spark_conf` of a built-in job is applied by the driver after it starts, too late for submit-time configurations: `spark.jars` / `spark.jars.packages` don't reach the driver classpath, and `spark.master` in `spark_conf` made a job run in local mode inside the driver pod while reporting success. Built-in templates also defaulted to `local[*]`, which only worked on Kubernetes because the executor silently dropped the master. Fix: #13703 ### Does this PR introduce _any_ user-facing change? - With the k8s job executor, `spark_conf` of a built-in job takes effect at submit time (e.g. `spark.jars`, `spark.jars.packages`, `spark.driver.*`); its `spark.kubernetes.*` keys are ignored. - A job whose `spark.master` isn't a Kubernetes master is rejected by the k8s job executor; an invalid `spark_conf` of a built-in job is rejected. - Built-in template versions: `v3` (`builtin-iceberg-update-manifest-stats`: `v2`); `spark_master` defaults to empty. - No new configuration keys. ### How was this patch tested? - New unit tests in `TestSparkApplicationUtils`, `TestSparkProcessBuilder`, `TestIcebergJobUtils`, `TestOptimizerCmd` and the template tests. - Manually on OrbStack (spark-kubernetes-operator 0.9.0, Spark 3.5.9, image without the Iceberg runtime): built-in Iceberg jobs with the runtime from `spark.jars` (https) and `spark.jars.packages` in `spark_conf` succeed; `spark.master` and `spark.kubernetes.*` in `spark_conf` are ignored; a local master and an invalid `spark_conf` are rejected; an empty `spark_conf` keeps the executor's default `spark.jars`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
