roryqi opened a new pull request, #11299: URL: https://github.com/apache/gravitino/pull/11299
### What changes were proposed in this pull request? - Add `IcebergCleanupManager`: a server-wide worker pool that polls the `iceberg_cleanup_job` store for `PENDING` jobs, claims them via a heartbeat lease, deletes the dropped table's reachable files in bulk through a shared `deleteExecutor` (with `CallerRunsPolicy` back-pressure), renews heartbeats on a separate scheduler thread, drives the retry state machine (transient failure → back to `PENDING`, give up at `max-attempts` → `FAILED`), and prunes finished rows past the retention window. - Add `TestIcebergCleanupManager` unit tests, running against the H2/MySQL/PostgreSQL backend matrix via `TestJDBCBackend`. - Sync the async hard-deletion design doc with the merged persistence layer (the `dropTable` enqueue snippet and the testing section). ### Why are the changes needed? Second of three stacked PRs for async hard deletion in the Iceberg REST catalog. The persistence layer (#11266) added the durable job store; this PR adds the worker engine that drains jobs and deletes files. REST integration follows in PR 3. Fix: #11298 ### Does this PR introduce _any_ user-facing change? No. The manager is not yet wired into the REST drop path (that lands in PR 3), and the config keys (`async-cleanup.*`) were introduced with the persistence layer (#11266). ### How was this patch tested? `./gradlew :iceberg:iceberg-rest-server:test --tests "org.apache.gravitino.iceberg.service.cleanup.TestIcebergCleanupManager" -PskipITs` — 7 tests pass (lifecycle to `SUCCEEDED`, transient-retry to `FAILED`, bulk-delete batching, already-deleted-file tolerance, reachable-file cleanup, and tombstone delegation). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
