lasdf1234 opened a new issue, #14536:
URL: https://github.com/apache/gravitino/issues/14536
### Version
main branch / 1.3.x (Iceberg 1.11.0), Iceberg REST catalog with server-side
scan planning enabled.
### Describe what's wrong
Two related defects in `POST
/v1/{prefix}/namespaces/{ns}/tables/{table}/plan` (`CatalogWrapperForREST`):
1. **`use-snapshot-schema` is ignored.** When `snapshot-id` is set, the
server always binds `filter` / `select` against the snapshot schema
(`table.newScan().useSnapshot(...)`), even with `use-snapshot-schema: false`.
The REST spec says time travel should use the snapshot schema (`true`), while
scanning a **branch** should use the **current table schema** (`false`, the
default). After a column rename, a branch scan that filters on the current name
returns 500 `Cannot find field`, and a filter on the old name is accepted even
though the current schema no longer has it. The Iceberg 1.11 Java client sends
`use-snapshot-schema: false` for branch scans.
2. **Older partition spec + dropped source column.** After partition
evolution drops the source column of a non-default spec (partition by `s`,
evolve to `n`, drop `s`), every plan request returns `400` with `Content-Type:
text/plain` and body `Type: unknown is not supported` (not an `ErrorModel`).
The table is valid and readable with client-side planning. Iceberg's
`PartitionSpec.partitionType()` marks the orphaned field as `unknown`, and
`ContentFileParser` / `SingleValueParser` throw while serializing partition
values.
### Error message and/or stacktrace
```
POST .../plan {"snapshot-id": S1, "use-snapshot-schema": false, "filter":
{"type":"eq","term":"l2","value":10}}
500 RuntimeException: Cannot find field 'l2' in struct: struct<1: id:
optional long, 2: l: optional long>
POST .../plan {} # after dropping a retired partition source column
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Type: unknown is not supported
```
### How to reproduce
**use-snapshot-schema**
1. Create a table, append several snapshots, rename a column, note an older
snapshot id S1.
2. Plan with `snapshot-id=S1`, `use-snapshot-schema=false`, filter on the
**new** column name: 500.
3. Same request with `use-snapshot-schema=true` and the **old** name: 200.
**Dropped partition source**
1. Create a table partitioned by `s`, append data, add a new spec on `n` and
set it as default, then drop `s`.
2. `POST .../plan` with `{}`: 400 text/plain `Type: unknown is not
supported`.
### Additional context
Expected: with `use-snapshot-schema: false`, bind filter/select to the
current schema while reading files from the given snapshot; with `true`, use
the snapshot schema. For tables whose historical specs reference dropped
columns, plan should return 200 with file-scan-tasks (or a proper ErrorModel if
planning is impossible), never a bare text/plain 400 for a valid request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
