laserninja opened a new pull request, #12942:
URL: https://github.com/apache/gravitino/pull/12942
### What changes were proposed in this pull request?
Adds `design-docs/iceberg-rewrite-manifests-job.md`, the design for the
`builtin-iceberg-rewrite-manifests` maintenance job. Requested by @lasdf1234
during review of #12937.
The document covers:
- **Background**: how Iceberg scan planning walks the metadata tree, why
manifest count grows with commit frequency rather than data volume, and why
neither `rewrite_data_files` nor `expire_snapshots` consolidates the manifests
of the current snapshot. Engine-side delegation (#10500) covers the manual
Trino path and leaves the server-side gap.
- **Solution Investigations**: four approaches, including why the Iceberg
Java `RewriteManifests` action was rejected in favour of the Spark SQL
procedure - the `compileOnly` boundary, per-backend catalog loading, and
consistency with the three sibling jobs.
- **Proposal**: template registration, the parameter surface (`table`,
`use_caching`, `spec_id`, `spark_conf`) and its validation, generated SQL with
the two escaping rules, procedure output handling, the end-to-end user process,
and an implementation flow diagram.
- **Placeholder resolution**: `JobManager.replacePlaceholder` keeps a
placeholder verbatim when its `jobConf` key is absent, so an omitted
`use_caching` reaches the job as the literal `{{use_caching}}` and
`Boolean.parseBoolean` turns it into `false` - silently disabling caching the
caller never configured. This is what shapes the job's input handling, and the
same hazard exists in the sibling jobs.
- **Follow-up**: the manifest metrics needed before a policy can fire on
them, and a policy sketch that reuses the recommender path. Per the review
discussion on #11700, the handler declares `DataRequirement.TABLE_STATISTICS`
and any last-run signal lives in `statistic_meta` as a `custom-` statistic,
aligning with table compaction rather than introducing bespoke state.
- **Task Breakdown**: four phases - the job itself (#12937), manifest
metrics, policy-driven triggering, and placeholder hardening across the
existing jobs.
Docs only. No code changes.
Related: #11196 (tracking), #12937 (implementation), #8864 (umbrella),
#11216 (original PR by @ibrahimErbilen). Sibling designs under the same
umbrella: #11727 (expire-snapshots), #11700 (remove-orphan-files).
### Why are the changes needed?
The implementation in #12937 makes several non-obvious choices - the Spark
SQL procedure over the Java action, dropping unresolved placeholders, reading
procedure output as `Number` - whose reasons live in the diff rather than
anywhere a reviewer or future maintainer would look. The doc also records what
the job deliberately does not do: there is no built-in policy that triggers it,
because `system_iceberg_compaction` evaluates data-file metrics and no
manifest-level metrics exist yet. Writing down that sequencing keeps the next
person from assuming automatic triggering already works.
### Does this PR introduce _any_ user-facing change?
No. Design document only. The user-facing documentation for the job is in
#12937.
### How was this patch tested?
- `./gradlew rat` - passing (Apache license header present).
- `./gradlew :docs:build` - passing.
- Every API, class, config key, statistic name, and file path referenced was
checked against `main`, and the Iceberg procedure signature and output types
against `iceberg-spark-runtime-3.5_2.12:1.11.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
