lasdf1234 commented on code in PR #12942: URL: https://github.com/apache/gravitino/pull/12942#discussion_r4005274849
########## design-docs/iceberg-rewrite-manifests-job.md: ########## @@ -0,0 +1,387 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of the Iceberg Rewrite Manifests Job in Gravitino + +Tracking issue: [#11196](https://github.com/apache/gravitino/issues/11196). Umbrella: [#8864](https://github.com/apache/gravitino/issues/8864). Implementation: [#12937](https://github.com/apache/gravitino/pull/12937), continuing [#11216](https://github.com/apache/gravitino/pull/11216). + +--- + +## Background + +### How Iceberg scan planning reads metadata + +An Iceberg table stores its file inventory in a three-level tree. Planning a scan walks the tree from the top: + +``` +table metadata (vN.metadata.json) + | + +-- snapshot --> manifest list (snap-<id>.avro) + | + +-- manifest A (.avro) --> data file, data file, ... + +-- manifest B (.avro) --> data file, data file, ... + +-- manifest C (.avro) --> data file, data file, ... +``` + +The manifest list holds per-manifest partition summaries, so a filter can skip whole manifests. Every manifest that survives that skip must then be opened and read to find matching data files. Planning cost therefore tracks the **number of manifests the filter cannot exclude**, not the size of the table. + +### Why manifest count grows + +Each commit writes at least one new manifest for the files it adds. Manifest count grows with **commit frequency**, independent of how much data the table holds: + +| Workload | Commits/day | Manifests after 30 days | +|----------|-------------|-------------------------| +| Hourly batch load | 24 | ~720 | +| Streaming, 1-minute checkpoints | 1,440 | ~43,200 | + Review Comment: Could u align your table? ########## design-docs/iceberg-rewrite-manifests-job.md: ########## @@ -0,0 +1,387 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of the Iceberg Rewrite Manifests Job in Gravitino + +Tracking issue: [#11196](https://github.com/apache/gravitino/issues/11196). Umbrella: [#8864](https://github.com/apache/gravitino/issues/8864). Implementation: [#12937](https://github.com/apache/gravitino/pull/12937), continuing [#11216](https://github.com/apache/gravitino/pull/11216). + +--- + +## Background + +### How Iceberg scan planning reads metadata + +An Iceberg table stores its file inventory in a three-level tree. Planning a scan walks the tree from the top: + +``` +table metadata (vN.metadata.json) + | + +-- snapshot --> manifest list (snap-<id>.avro) + | + +-- manifest A (.avro) --> data file, data file, ... + +-- manifest B (.avro) --> data file, data file, ... + +-- manifest C (.avro) --> data file, data file, ... +``` + +The manifest list holds per-manifest partition summaries, so a filter can skip whole manifests. Every manifest that survives that skip must then be opened and read to find matching data files. Planning cost therefore tracks the **number of manifests the filter cannot exclude**, not the size of the table. + +### Why manifest count grows + +Each commit writes at least one new manifest for the files it adds. Manifest count grows with **commit frequency**, independent of how much data the table holds: + +| Workload | Commits/day | Manifests after 30 days | +|----------|-------------|-------------------------| +| Hourly batch load | 24 | ~720 | +| Streaming, 1-minute checkpoints | 1,440 | ~43,200 | + +A streaming table of modest size can accumulate tens of thousands of manifests, each covering a handful of data files. Filters exclude few of them because a small manifest written per commit spans whatever partitions that commit touched, so partition summaries end up wide and non-selective. + +Compaction makes this worse rather than better. `rewrite_data_files` commits its results, and that commit writes new manifests of its own. + +### What Gravitino ships today + +The table maintenance service separates **deciding** that work is needed from **executing** it. Built-in job templates are the execution half; policies, strategies, and the recommender are the decision half. + +| Built-in job template | Purpose | Triggered by a built-in policy? | +|-----------------------|---------|---------------------------------| +| `builtin-iceberg-update-stats` | Writes the statistics and metrics policies read | No, submitted directly or by the CLI | +| `builtin-iceberg-rewrite-data-files` | Merges small **data** files | Yes, `system_iceberg_compaction` | +| `builtin-iceberg-expire-snapshots` | Drops old snapshots and their metadata | No, submitted directly | + Review Comment: Could u align your table? ########## design-docs/iceberg-rewrite-manifests-job.md: ########## @@ -0,0 +1,387 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design of the Iceberg Rewrite Manifests Job in Gravitino + +Tracking issue: [#11196](https://github.com/apache/gravitino/issues/11196). Umbrella: [#8864](https://github.com/apache/gravitino/issues/8864). Implementation: [#12937](https://github.com/apache/gravitino/pull/12937), continuing [#11216](https://github.com/apache/gravitino/pull/11216). + +--- + +## Background + +### How Iceberg scan planning reads metadata + +An Iceberg table stores its file inventory in a three-level tree. Planning a scan walks the tree from the top: + +``` +table metadata (vN.metadata.json) + | + +-- snapshot --> manifest list (snap-<id>.avro) + | + +-- manifest A (.avro) --> data file, data file, ... + +-- manifest B (.avro) --> data file, data file, ... + +-- manifest C (.avro) --> data file, data file, ... +``` + +The manifest list holds per-manifest partition summaries, so a filter can skip whole manifests. Every manifest that survives that skip must then be opened and read to find matching data files. Planning cost therefore tracks the **number of manifests the filter cannot exclude**, not the size of the table. + +### Why manifest count grows + +Each commit writes at least one new manifest for the files it adds. Manifest count grows with **commit frequency**, independent of how much data the table holds: + +| Workload | Commits/day | Manifests after 30 days | +|----------|-------------|-------------------------| +| Hourly batch load | 24 | ~720 | +| Streaming, 1-minute checkpoints | 1,440 | ~43,200 | + +A streaming table of modest size can accumulate tens of thousands of manifests, each covering a handful of data files. Filters exclude few of them because a small manifest written per commit spans whatever partitions that commit touched, so partition summaries end up wide and non-selective. + +Compaction makes this worse rather than better. `rewrite_data_files` commits its results, and that commit writes new manifests of its own. + +### What Gravitino ships today + +The table maintenance service separates **deciding** that work is needed from **executing** it. Built-in job templates are the execution half; policies, strategies, and the recommender are the decision half. + +| Built-in job template | Purpose | Triggered by a built-in policy? | +|-----------------------|---------|---------------------------------| +| `builtin-iceberg-update-stats` | Writes the statistics and metrics policies read | No, submitted directly or by the CLI | +| `builtin-iceberg-rewrite-data-files` | Merges small **data** files | Yes, `system_iceberg_compaction` | +| `builtin-iceberg-expire-snapshots` | Drops old snapshots and their metadata | No, submitted directly | + +### The problem + +Nothing in that list consolidates the manifests of the **current** snapshot. + +- `rewrite_data_files` rewrites data files. The manifests it writes are a side effect of its commit, not a consolidation of what was there before. +- `expire_snapshots` deletes manifests reachable only from expired snapshots. Manifests belonging to the live snapshot are exactly the ones scan planning reads, and expiration never touches them. It is not a substitute. + +Iceberg has an action for this, exposed as the `rewrite_manifests` Spark procedure, and Gravitino already delegates it on the **engine** side: [#10500](https://github.com/apache/gravitino/pull/10500) lets a Trino user issue `CALL ... rewrite_manifests`. That closes the manual path and leaves two gaps: + +1. It requires a human at a Trino session. There is no server-side job the maintenance service can submit, and therefore no path to policy-driven manifest optimization. +2. It only helps Trino users. Gravitino deployments driving Spark or Flink have no equivalent. + +--- + +## Goals + +1. **Built-in template**: A template named `builtin-iceberg-rewrite-manifests` appears in `GET /api/metalakes/{metalake}/jobs/templates` once `gravitino-jobs` is on the server classpath, and is submittable through the existing jobs REST API with no new endpoint. +2. **Full procedure surface**: Every parameter Iceberg's `rewrite_manifests` procedure accepts - `table`, `use_caching`, `spec_id` - is reachable through `jobConf`. +3. **Absent means default**: Omitting an optional parameter leaves Iceberg's own default in force. The generated SQL names only the parameters the caller actually supplied. +4. **Invalid input fails before Spark starts**: A malformed `use_caching` or `spec_id` exits non-zero with a message naming the offending value, rather than being coerced into a valid-looking SQL literal. +5. **Runtime version tolerance**: The job runs against any Iceberg version exposing the procedure, without assuming the integer width of its output columns. Iceberg is a `compileOnly` dependency supplied by the cluster, so the compile-time and runtime versions differ in practice. +6. **Injection safety**: A catalog name or table identifier containing quotes, backticks, or statement separators cannot change the shape of the generated statement. +7. **Consistency with siblings**: The job's class layout, argument convention, template registration, and staging log output match `IcebergExpireSnapshotsJob` closely enough that an operator familiar with one can read the other. + +--- + +## Non-Goals + +1. **A built-in policy that triggers this job automatically**: `system_iceberg_compaction` is the only built-in policy type, and it evaluates `custom-data-file-mse` and `custom-delete-file-number` - both data-file metrics. Automatic triggering needs manifest-level metrics to exist first, so it is sequenced as follow-up work below rather than bundled here. +2. **Replacing `expire_snapshots` or `rewrite_data_files`**: The three address different layers - snapshot history, data files, and the manifests indexing them. Running this job does not reduce the need for the other two. +3. **Partition-scoped or predicate-scoped rewrites**: Iceberg's procedure takes no `where` clause; it rewrites all manifests of the current snapshot. Offering a filter would mean bypassing the procedure for the Java action API, which Solution Investigations rejects below. +4. **Engine-side delegation**: Issuing `rewrite_manifests` from a query engine session is already covered by [#10500](https://github.com/apache/gravitino/pull/10500). This job is the server-side counterpart, not a replacement. +5. **Scheduling**: The maintenance service is driven by explicit submission or by the CLI workflow, with no internal scheduler. This job inherits that and does not introduce one. + +--- + +## Solution Investigations + +| Approach | Pros | Cons | Decision | +|----------|------|------|----------| +| A. Engine-side delegation only (status quo, #10500) | Already merged; zero new server code | Needs a human in a Trino session; not policy-drivable; useless to non-Trino deployments | Rejected - leaves the server-side gap that #11196 is about | +| B. Iceberg Java `RewriteManifests` action, called directly | Type-safe; exposes `rewriteIf`, `stagingLocation`, `specId`; no SQL text to escape | Requires catalog-loading plumbing no other job has; binds us to `iceberg-core` internals across the `compileOnly` boundary; diverges from all three sibling jobs | Rejected - see below | Review Comment: Could u align your table? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
