jerryshao commented on issue #13385:
URL: https://github.com/apache/gravitino/issues/13385#issuecomment-5772124643
## What jobs need to support, and how we check it
Placeholder defaults only need the job's cooperation in two cases, and only
when the template author chooses them:
- An empty default (`{{x:-}}`) means the job must treat an empty value as
"not set".
- A switch written as `--flag {{flag:-false}}` means the job must accept
`true`/`false` as the value.
If a job can't do either, the template author leaves the placeholder without
a default, which makes it required. The job then needs no change; callers just
pass the value. So writing an empty default is the template author's promise
that the job handles empty values.
Gravitino can't verify arbitrary jobs, since they are black boxes (any
language, any binary). So the rendering rules never depend on how a job
behaves: rendering is deterministic, and a job that doesn't follow the
convention only breaks itself, not Gravitino or other jobs. How much we can
check depends on who owns the job:
| Job | How compliance is ensured |
|---|---|
| Built-in jobs | **Single source of truth.** A `JobOptions` definition
generates both the template arguments and the job's argument parser, so the two
can't drift. A contract test in `TestBuiltInJobTemplateProvider` renders every
built-in template twice, once with only the required keys and once with all
keys. It feeds the result to the job's parser and checks the parsed values and
that no literal `{{...}}` leaks through. |
| Java jobs built on our helpers | Publish `JobOptions` / `JobArgs` plus a
contract-test helper (e.g. `JobTemplateContract.verify(provider)`), so
third-party jobs can get the same guarantee in their own CI. Optional. |
| Any other job | **Registration-time lint** that only warns, e.g. an empty
default on a standalone positional argument, or a switch-like placeholder
(`enable_*`) that doesn't follow a `--flag`. **A dry-run API** that returns the
rendered arguments, configs and environments for a given `jobConf` without
running the job. **A job author guide** that uses the built-in jobs as
reference implementations. **Output log retrieval** (#12716) to diagnose
argument-parsing failures. |
Proposed follow-ups, each as a separate subtask:
1. `JobOptions` / `JobArgs` and contract tests for the built-in jobs. This
also merges the two nearly identical argument parsers
(`IcebergJobUtils.parseArguments` and the copy in
`IcebergUpdateStatsAndMetricsJob`), which currently treat a bare `--flag`
differently.
2. A dry-run API that renders a job template for a given `jobConf` without
running it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
