laserninja opened a new pull request, #11206:
URL: https://github.com/apache/gravitino/pull/11206
### What changes were proposed in this pull request?
Add a new built-in Iceberg maintenance job
`builtin-iceberg-expire-snapshots` that expires old snapshots from Iceberg
tables via Spark's `expire_snapshots` procedure.
**Changes:**
- New `IcebergExpireSnapshotsJob` class in `maintenance/jobs` following the
same pattern as `IcebergRewriteDataFilesJob`
- Supports configurable parameters: `older_than` (timestamp), `retain_last`
(number of snapshots to keep), `stream_results` (boolean)
- SQL injection protection via `escapeSqlString()` and
`escapeSqlIdentifier()`
- Input validation for `retain_last` (must be positive integer) and
`stream_results` (must be true/false)
- Registered in `BuiltInJobTemplateProvider`
### Why are the changes needed?
Without periodic snapshot expiration, Iceberg table metadata grows
indefinitely, accumulating snapshot JSON files and manifest lists that slow
down table operations and waste storage. The existing built-in jobs
(`builtin-iceberg-rewrite-data-files` and `builtin-iceberg-update-stats`) cover
data compaction and metrics but do not address metadata cleanup.
This is one of the most critical Iceberg housekeeping operations. PR #10500
added Trino-side delegation for `expire_snapshots` as a procedure, but there is
no server-side built-in job that can be triggered automatically via the
Optimizer.
Fix: #11194
**Note:** This PR covers the job layer. The end-to-end policy/strategy
integration (e.g. `IcebergSnapshotExpirationContent`,
`SnapshotExpirationStrategyHandler`) can be added as a follow-up, as discussed
in the issue.
### Does this PR introduce _any_ user-facing change?
No user-facing API changes. Adds a new built-in job template
`builtin-iceberg-expire-snapshots` that will be available for maintenance job
scheduling.
### How was this patch tested?
- Added `TestIcebergExpireSnapshotsJob` with 40 unit tests covering:
- Job template metadata (name, comment, executable, mainClass, arguments,
sparkConfigs, version)
- Argument parsing (required, optional, empty values, missing values, all
options, order independence)
- Procedure call building (minimal, with older-than, retain-last,
stream-results, all params, empty params)
- SQL escaping and injection prevention
- Input validation for retain-last and stream-results
- Custom Spark config parsing (valid JSON, numeric values, empty, null,
invalid JSON)
- All 142 tests in `maintenance:jobs` module pass with 0 failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
