vbabenkoru opened a new pull request, #9349: URL: https://github.com/apache/paimon/pull/9349
### Purpose This PR is stacked on #9348 (the first two commits). Review only the top commit. The REST metadata committer expects the catalog to be exactly one snapshot behind. It treats any other catalog state as an invalid base. To recover, it drops and recreates the catalog table. Since #9245, it may drop and then re-register the table instead. On AWS Glue, this requires `glue:DeleteTable`. It also gives the catalog table a new identity for downstream users such as Glue and Snowflake. This turns a temporary publishing failure into a destructive operation. The local metadata file is written before the catalog commit. An unclear failure (`CommitStateUnknownException`) can therefore leave the catalog behind. Today, recovery requires dropping the table (#8875). This PR adds `metadata.iceberg.rest-auto-recreate`. Its default value is `true`, which keeps the current behavior. When this option is `false`, the committer fixes the catalog in place and never drops the catalog table: * After an unclear REST commit (`CommitStateUnknownException`), the committer reloads the catalog. If the commit succeeded, it reports success. Otherwise, it retries the same updates once using the unchanged base. If the result is still unclear, it reports the ambiguity. * If the catalog is behind, the committer updates it by replaying every missing retained snapshot in order. Each Iceberg snapshot contains everything it needs through its manifest list. This produces exactly the same result as consecutive successful commits. If the catalog already contains a snapshot, the committer re-activates it with a ref move instead of adding it again. Iceberg rejects adding an existing snapshot id. * The committer detects regenerated local history by comparing manifest lists, not only snapshot ids. This covers cases with the same snapshot ids but different manifest lists, such as a full-history rebuild from #9348. It removes stale catalog entries. It skips entries whose sequence numbers were already used and logs a warning, because Iceberg sequence numbers can only be used once. * In every mode, including the default, a commit is a no-op when the catalog already contains the same snapshot with identical content. Previously, the committer recreated the table only to produce the same content. * States that cannot be fixed without destructive changes fail with `IcebergRestCatalogOutOfSyncException`. These states include a catalog that is ahead, a different snapshot stored under the id being published, or a v3 row-id space that the catalog cannot accept. The error names both states and explains the manual fix. Initial table creation, empty-table recovery, the v3 nonzero-watermark registration path from #9245, field-id mapping, schema deduplication, and next-row-id handling are unchanged. The v3 nonzero-watermark registration path remains available only in the default mode. ### Tests Five new tests were added to `IcebergRestMetadataCommitterTest`: a behind catalog catches up by replaying multiple snapshots; an already-published snapshot is a no-op in both modes; an ahead catalog fails with the out-of-sync error without dropping the table; regenerated history is found by comparing manifest lists and stale entries are removed; and a retry publishes existing metadata again. The full `paimon-iceberg` suite passes with both configurations: JDK 11 / Iceberg 1.8.1 and JDK 17 / `-Piceberg-ga` (Iceberg 1.11). ### API and Format New optional table option: `metadata.iceberg.rest-auto-recreate` (default `true`). The default behavior is unchanged, except that the table is no longer recreated when the catalog already contains the commit's snapshot. ### Documentation The option is documented in its description, which is used to generate the configuration docs. --- *AI notice: The code is generated using Fable 5 (with reviews from Codex) but has been verified to run on a real cluster with Flink, Paimon, Iceberg, StarRocks and Snowflake.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
