dhruvarya-db opened a new pull request, #17543:
URL: https://github.com/apache/iceberg/pull/17543
Opening this to drive discussion on what the REST catalog spec needs for
table format version 4. It is a proposal rather than a merge candidate — parts
of it depend on table spec work that is still in review, and one change is
deliberately breaking. Both are called out under Open questions.
## Motivation
A REST catalog serves tables at mixed format versions to clients at mixed
capability levels, and v4's changes are concentrated in metadata structure,
which is exactly what the REST contract carries. Three constraints in the
current spec prevent a catalog from serving a v4 table at all:
| Constraint | V4 requirement |
| --- | --- |
| `TableMetadata.format-version` has `maximum: 3` | must accept `4` |
| `Snapshot` requires `manifest-list` | v4 snapshots reference a root
manifest instead |
| `CommitTableResponse` requires `metadata-location` | may not exist when
the catalog owns table state |
## Changes
**`TableMetadata`** — raise the `format-version` maximum to 4. Document that
`location` is optional in v4, where it may be supplied by the catalog rather
than carried in metadata.
**`Snapshot`** — add `root-manifest`, make `manifest-list` optional, and add
a `oneOf` requiring exactly one of the two:
```yaml
oneOf:
- required: [manifest-list]
- required: [root-manifest]
```
A snapshot with neither is invalid and must not be read as a snapshot with
no files; one with both is also invalid. Which of the two applies is determined
by `format-version`, which cannot be expressed here because that field lives in
the enclosing `TableMetadata`, so it is stated normatively in the description.
**`LoadTableResult` / `CommitTableResponse`** — add `table-location`,
carrying the table's base location for resolving relative paths. V4 makes
`TableMetadata.location` optional while allowing relative paths, so a table can
legitimately have metadata that is unreadable without a catalog-supplied base.
`CommitTableResponse` no longer requires `metadata-location`, and both
responses document that it may be absent when the catalog is the source of
truth for table state and no client-visible metadata pointer exists. Clients
obtain a location for such a table through `unregisterTable`, whose existing
contract guarantees the catalog stops accepting commits, so the returned
pointer cannot go stale.
**`UnregisterTableResult`** — add `table-location`, so the endpoint that
hands back a metadata location cannot return metadata with no base to resolve
against.
**Endpoint paths** — move the table endpoints to `/v2`. Every path whose
response can carry table metadata moves together: `tables`, `tables/{table}`,
`register`, and `tables/{table}/unregister`.
## Open questions
**`root-manifest` is not in the ratified spec.** It comes from the adaptive
metadata tree proposal (#16025), which is still open, so the field name and
shape may change. Note also that `SnapshotParser` currently writes
`manifest-list` for v4 tables, so the `oneOf` as written would reject snapshots
this repository produces today. Whether to keep the field, defer it until
#16025 lands, or make it purely additive is the main thing worth discussing
here.
**Moving the table endpoints to `/v2` is breaking**, and assumes v1 is no
longer supported. Adding `/v2` alongside `/v1` is the obvious alternative if
that assumption does not hold — that requires v1 to refuse format versions
above 3, ideally with a typed error so clients can retry on v2 rather than
guessing at a generic 400.
**Is path versioning the right mechanism** at all, versus a header or a
content-type parameter? The path prefix composes with the existing
`CatalogConfig.endpoints` advertisement without new machinery, which is why it
is used here.
**Snapshot offloading** is not yet in the spec and would interact with the
`snapshots` query parameter on `loadTable` if it lands later. Worth knowing
whether it should be folded in before any endpoint versioning is finalised,
since retrofitting it would mean another version.
## Testing
`make lint` and `make generate` both pass; the generated model is committed
and regeneration is idempotent.
One note on the generated Python: `datamodel-codegen` expands the `Snapshot`
`oneOf` into `Snapshot1`, `Snapshot2`, and `Snapshot(RootModel[Snapshot1 |
Snapshot2])`. That union is a lossy rendering — it matches the first satisfying
member, so a snapshot carrying both pointers coerces into `Snapshot1` rather
than being rejected. The YAML schema does reject it, verified with
`jsonschema`, so this is a codegen artifact rather than a gap in the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
