laserninja commented on issue #11194: URL: https://github.com/apache/gravitino/issues/11194#issuecomment-4514978010
Thanks @jerryshao, good point. Agreed. This needs more than just the job class. Looking at how `builtin-iceberg-rewrite-data-files` was implemented end to end, I think the full scope includes: 1. **Job**: `IcebergExpireSnapshotsJob` in `maintenance/jobs` — calls Iceberg's `expire_snapshots` procedure via Spark, with parameters like `older_than`, `retain_last`, and `stream_results` 2. **Policy type**: A new `system_iceberg_snapshot_expiration` policy (similar to `system_iceberg_compaction`) so users can attach expiration policies to tables 3. **Strategy/Recommender**: Logic in `maintenance/optimizer` to determine when expiration should be triggered (e.g., based on snapshot count thresholds or age) 4. **Command**: The corresponding command in the optimizer to submit expire-snapshot jobs 5. **Updater**: Metrics collection in `maintenance/updaters` to feed the recommender (e.g., current snapshot count, oldest snapshot timestamp) I'll update the issue description to reflect the full scope. Happy to break this into sub-tasks if that's preferred. Is there a design doc or reference PR from the compaction implementation that covers the end-to-end pattern I should follow? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
