laserninja opened a new pull request, #13153: URL: https://github.com/apache/gravitino/pull/13153
### What changes were proposed in this pull request? Add and register `builtin-iceberg-remove-orphan-files`, with `older_than`, `location`, `dry_run`, and custom Spark configuration. Validate the scan location against the target table before calling Iceberg, reject symbolic links, and log candidate paths during dry runs. This is the job-layer implementation from #11700. Policy, strategy, and adapter integration will follow separately. ### Why are the changes needed? Failed or incomplete writes leave unreferenced files in table storage. This job provides direct orphan cleanup through Gravitino's existing job submission API. Related to #11195. ### Does this PR introduce _any_ user-facing change? Adds a v1 Spark job template and documents direct submission. The cutoff defaults to three days ago and dry-run defaults to false. Custom locations must remain within the target table's storage location. Iceberg 1.11's SQL procedure rejects cutoffs less than 24 hours old. This implementation preserves that safeguard; the design's zero-day example is not supported by this procedure. ### How was this patch tested? With JDK 17: ```bash ./gradlew :maintenance:jobs:spotlessApply :maintenance:jobs:build :maintenance:jobs:javadoc -PskipWeb=true ``` All 153 tests passed, including 15 new tests. Real local Spark tests cover CLI invocation, dry-run preservation, deletion of old orphans, retention of referenced and recent files, explicit cutoffs, quoted identifiers, and rejection of unsafe inputs and locations. Remote filesystem validation is unit-tested with Hadoop filesystem mocks. Docker-dependent tests were not enabled. Javadoc generation succeeded with seven pre-existing warnings in unchanged files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
