laserninja opened a new issue, #11194: URL: https://github.com/apache/gravitino/issues/11194
### Describe the feature Add a new built-in Iceberg maintenance job `builtin-iceberg-expire-snapshots` that expires old snapshots from Iceberg tables. This would use Iceberg's `ExpireSnapshots` action (via Spark's `expire_snapshots` procedure) to remove snapshot metadata and associated data files that are no longer needed, preventing unbounded metadata growth. ### Motivation Without periodic snapshot expiration, Iceberg table metadata grows indefinitely — accumulating snapshot JSON files and manifest lists that slow down table operations and waste storage. The existing built-in jobs (`builtin-iceberg-rewrite-data-files` and `builtin-iceberg-update-stats`) cover data compaction and metrics but don't address metadata cleanup. This is one of the most critical Iceberg housekeeping operations. PR #10500 (merged) already added Trino-side delegation for `expire_snapshots` as a procedure, but there is no server-side built-in job that can be triggered automatically via the Table Maintenance Service (Optimizer) policies. ### Describe the solution - Create `IcebergExpireSnapshotsJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob` - Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-expire-snapshots` - Support configurable parameters: `older_than` (timestamp), `retain_last` (number of snapshots to keep), `stream_results` - Optionally add a corresponding policy type (e.g., `system_iceberg_snapshot_expiration`) similar to `system_iceberg_compaction` ### Additional context Related: #10280 (closed — Trino-side procedure delegation), #8864 (umbrella — add built-in jobs for table maintenance) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
