laserninja opened a new issue, #11195: URL: https://github.com/apache/gravitino/issues/11195
### Describe the feature Add a new built-in Iceberg maintenance job `builtin-iceberg-remove-orphan-files` that identifies and removes orphaned data and metadata files from Iceberg table storage locations. This would use Iceberg's `RemoveOrphanFiles` action (via Spark's `remove_orphan_files` procedure) to reclaim storage from files that are no longer referenced by any table snapshot. ### Motivation Orphan files accumulate from failed writes, incomplete transactions, schema evolution, or concurrent operations. Without periodic cleanup, these files waste significant storage — especially in high-write-volume environments. The existing maintenance jobs don't address orphan file cleanup. PR #10500 (merged) added Trino-side delegation for `remove_orphan_files` as a procedure, but there is no server-side built-in job for automatic policy-driven cleanup. ### Describe the solution - Create `IcebergRemoveOrphanFilesJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob` - Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-remove-orphan-files` - Support configurable parameters: `older_than` (timestamp — default 3 days), `location` (custom location override), `dry_run` (preview-only mode) - Optionally add a corresponding policy type for scheduled orphan cleanup ### Additional context Related: #10280 (closed — Trino-side procedure delegation), #8864 (umbrella — add built-in jobs for table maintenance) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
