markhoerth opened a new issue, #14457:
URL: https://github.com/apache/gravitino/issues/14457
### Version
main branch
### Describe what's wrong
Table commits through the Iceberg REST server apply schema and partition
updates without checking them against the table, and the result cannot be read
by engines:
1. `add-schema` changes a field to an incompatible type (long to int, int to
string, decimal scale change, primitive to struct), reuses the id of a dropped
field for a different field, or adds a required field with no default. Data
written earlier can no longer be read (pyiceberg `ResolveError`, Spark `Not a
primitive type`), or old values come back as null. pyiceberg's `update_column`
to a different decimal scale reaches this path.
2. `add-spec` (v2+) uses a partition field id that another spec uses for a
different field, or a new field id below `last-partition-id`. Spark's
`.partitions` metadata table then fails with "Conflicting partition fields",
and pyiceberg loses partition values.
3. `remove-partition-specs` removes a spec that manifests of the current
snapshot still use. Every reader then fails (pyiceberg `KeyError: 0`, Spark
NPE, the server's own scan planning 500).
4. A partition spec or sort order with a transform the table spec does not
define (for example `zorder`) can be made the default. Gravitino can then no
longer load the table (501).
5. A partition spec without `spec-id` is refused with 400 "Cannot parse
missing int: spec-id", although the REST spec makes `spec-id` read-only.
The server passes these updates to Iceberg's `TableMetadata.Builder`, which
applies them without these checks.
### Error message and/or stacktrace
```
POST .../tables/t {"updates":[{"action":"add-schema","schema":{...
"id":1,"type":"int" ...}}]} 200
pyiceberg scan: ResolveError: Cannot promote long to int
POST .../tables/t
{"updates":[{"action":"remove-partition-specs","spec-ids":[0]}]}
200
pyiceberg scan: KeyError: 0
```
### How to reproduce
1. Create a table with a `long` column `id`, append data, then commit an
`add-schema` with `id` as `int` and `set-current-schema`. Read the table.
2. Partition a table by `identity(s)`, append data, add and set a new
default spec, then commit `remove-partition-specs` with spec 0. Read the table.
### Additional context
The Iceberg table spec defines the allowed type promotions, partition field
id rules, and that every manifest's spec must remain in table metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
