jerryshao opened a new issue, #13385:
URL: https://github.com/apache/gravitino/issues/13385
### Describe the subtask
Today every placeholder in a job template must be supplied in `jobConf`. If
a key is missing, the literal `{{name}}` is passed through and the job fails at
runtime: for example, `--strategy {{strategy}}` fails validation and
`spark.master={{spark_master}}` breaks spark-submit. As a result, callers such
as the optimizer have to pass every key, use `""` for optional ones, and keep
the defaults in their own code.
Proposal:
- Support inline defaults: `{{name:-default}}`, or `{{name:-}}` for an empty
default. A value is resolved in this order: the `jobConf` value (an explicit
`""` counts as provided), then the template default. If neither exists, the
parameter is missing.
- Reject a submission with missing parameters with HTTP 400 and list all of
them. Do this before the staging directory is created or any file is fetched.
- Support `\{{` so a template can contain a literal `{{`.
- At registration time, reject templates that give the same parameter
conflicting defaults, or that have a default containing `}}`.
- Log a warning for `jobConf` keys that the template does not reference.
- Add defaults to the built-in templates and bump them to v2:
- `slices:-2` and `strategy:-binpack`.
- Empty defaults for the optional Iceberg arguments.
- Spark resource defaults such as `spark_master:-local[*]`.
- `where_clause` and the catalog `type`/`uri`/`warehouse` stay required.
- Change the expire snapshots `{{stream_results}}` argument to
`--stream-results {{stream_results:-false}}`.
Behavior change: in 2.0, templates that rely on unresolved placeholders
being passed through literally will be rejected with 400.
### Parent issue
https://github.com/apache/gravitino/issues/12667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
