laserninja opened a new issue, #11196: URL: https://github.com/apache/gravitino/issues/11196
### Describe the feature Add a new built-in Iceberg maintenance job `builtin-iceberg-rewrite-manifests` that rewrites Iceberg manifest files to optimize scan planning performance. This would use Iceberg's `RewriteManifests` action (via Spark's `rewrite_manifests` procedure) to consolidate small manifests and rewrite manifests with improved partition specs. ### Motivation Over time, Iceberg tables accumulate many small manifest files from frequent commits. This slows down scan planning because the engine must open and read each manifest to determine which data files match a query filter. Rewriting manifests into fewer, larger files with better-aligned partition specs improves query planning performance significantly. The existing `builtin-iceberg-rewrite-data-files` job addresses data file layout but not manifest file bloat. PR #10500 (merged) added Trino-side delegation for `rewrite_manifests` as a procedure, but there is no server-side built-in job for automatic policy-driven manifest optimization. ### Describe the solution - Create `IcebergRewriteManifestsJob` in `maintenance/jobs/src/main/java/org/apache/gravitino/maintenance/jobs/iceberg/` following the same pattern as `IcebergRewriteDataFilesJob` - Register it in `BuiltInJobTemplateProvider` as `builtin-iceberg-rewrite-manifests` - Support configurable parameters: `use_caching` (boolean) - Consider adding manifest-related metrics (e.g., manifest count, avg manifest size) to `IcebergUpdateStatsAndMetricsJob` for policy-driven triggering ### Additional context Related: #10280 (closed — Trino-side procedure delegation), #8864 (umbrella — add built-in jobs for table maintenance) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
