markhoerth opened a new pull request, #14458: URL: https://github.com/apache/gravitino/pull/14458
### What changes were proposed in this pull request? Depends on #14439 (it extends `IcebergCommitValidation`); until that is merged, this PR also shows its commit. `IcebergCommitValidation` (an extra `UpdateRequirement`, evaluated by Iceberg against the metadata each commit attempt is built on) now also checks: - `add-schema`: a field in the current schema keeps its kind and changes type only by an allowed promotion (`TypeUtil.isPromotionAllowed`). A field id not in the current schema must be above `last-column-id`, or name the same field it named before. A new required field needs an initial default when the table has data. A replace (which removes `main`) skips the data checks. - `add-spec` on v2+ tables: a field id another spec uses must name the same field, and a new field id must be above `last-partition-id`. Specs are checked after binding, so the ids are exactly those Iceberg will apply. - `remove-partition-specs`: refused while a manifest of any snapshot uses the spec. Manifest lists are read with the table's own FileIO from the catalog (fetched only when this update is present), so any catalog-specific FileIO configuration applies. - A partition spec or sort order with a transform the table spec does not define cannot be made the default. `IcebergObjectMapper` accepts a partition spec without `spec-id` in create-table and `add-spec`; the server assigns the spec id. ### Why are the changes needed? These commits left tables whose existing data engines could no longer read, partition metadata engines rejected, or tables Gravitino could not load. Fixes #14457 ### Does this PR introduce _any_ user-facing change? Yes. These commits are refused with 400. A partition spec without `spec-id` is accepted. ### How was this patch tested? `TestIcebergCommitValidation`, through Iceberg's `CatalogHandlers.updateTable` on an in-memory catalog: - incompatible type changes and a required field without a default (refused), and an optional new field (accepted); - reuse of a dropped field id (refused), and a replace redefining a column (accepted); - partition field id collisions and ids below `last-partition-id` (refused), and a valid evolution (accepted); - removing a spec with live manifests (refused) and an unused spec (accepted); - an unknown transform as the default sort order (refused) and added but not default (accepted); - create and `add-spec` without `spec-id` (accepted, with the server-assigned id). `./gradlew :iceberg:iceberg-rest-server:test -PskipITs` 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Eu6fb9pLfmrSJzGQDVLi1g -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
