Yufei, The compatibility issue is related to the conflict of relaxing the required table location field for relative patha and table version field.
We cannot guarantee which order a client processes the fields. Therefore, if the missing location field is encountered, an older client would reasonably fail due the missing required field and never have an opportunity to perform the version check. There will be more fields potentially as we add/modify fields for the other features introduced in v4, but this one alone highlights the compatibility issue. -Dan On Thu, Aug 20, 2026, 3:10 PM Yufei Gu <[email protected]> wrote: > I largely agree with Dan and Amogh. I just want to clarify one point: > > > clients will fail to parse the JSON before they can determine it's a v4 > table > > TableMetadata is required in LoadTableResult, and format-version is > required within TableMetadata.[1] So, absent an incompatible change to the > metadata’s JSON structure or a strict client-side parser, a client should > be able to identify the table format version before interpreting the rest > of the metadata. If the payload is malformed, failure is expected for v1 > clients as well. That case alone does not seem like a compatibility concern. > > 1. > https://github.com/apache/polaris/blob/4a3fba7d4b26ef9c985086d5aba53b26b0b63935/spec/iceberg-rest-catalog-open-api.yaml#L3541 > > Yufei > > > On Tue, Aug 18, 2026 at 9:04 AM Daniel Weeks <[email protected]> wrote: > >> Hey Dhruv, >> >> I think the second approach is a much cleaner solution, though it relies >> on some nuance for backward compatibility with clients. >> >> A few points to note: >> 1) The REST spec's required/optional fields are not intended to match >> exactly what is in the spec. TableMetadata is a great example of this, >> where most fields are required in the table spec, but are optional in the >> REST spec. >> 2) Backward compatibility comes down to whether a client will break when >> receiving a payload that it doesn't understand. >> 3) Unfortunately, some of the gates you might expect to be possible (like >> table version) aren't feasible if you break client-side parsers (clients >> will fail to parse the JSON before they can determine it's a v4 table). >> >> All that said, the only thing I believe we need to do is add a >> `v2::loadTable` endpoint and require all v4 tables to be loaded through >> that mechanism. It's important to note that only load table requires a >> version bump since committing a table still use the normal snapshot update >> and clients will know from the supported endpoints whether v4 is even >> supported (via the v2::loadTable endpoint). >> >> I love to hear what others think, but I'm strongly in favor of both >> starting the process of introducing new versioned endpoints and scoping >> what we need for V4. >> >> There are a few open questions, like what other metadata changes will be >> necessary (e.g. check constraints/default expressions/generated >> columns/etc)? How do we want to introduce the new endpoint (e.g. allow it >> to evolve with breaking changes until we vote on v2 endpoint / v4 spec? >> >> I think this is a great start though and would love to hear what others >> think, >> -Dan >> >> On Thu, Aug 13, 2026 at 3:58 PM Dhruv Arya <[email protected]> wrote: >> >>> Hi all, >>> >>> Kicking off this thread to discuss the initial REST spec changes needed >>> for Iceberg V4: >>> 1. root-manifest in Snapshot: We need the returned Snapshot to have a >>> root-manifest . (oneOf of root-manifest or manifest-list). For V4 tables, >>> manifest-list will not be set. >>> 2. Add table-location: LoadTableResult, CommitTableResponse, and >>> UnregisterTableResult all need to return table-location. When defined, this >>> will be the absolute path of the table root. (Allows the paths inside the >>> metadata to be relative making table migration easier). >>> 3. Make metadata-location in CommitTableResponse optional: This is for >>> catalog managed tables. >>> >>> Some of these changes are not backwards compatible: >>> `manifest-list` in Snapshot is a required field right now. Some clients >>> might rely on this field being set in the response. If we allow this to be >>> empty when `root-manifest` is set, these clients could break. The same >>> problem applies to `metadata-location`. Furthermore, many endpoints >>> (unregister, table, etc) transitively depend on Snapshot. If we create a >>> new SnapshotV2, many endpoints will need a version bump to V2. >>> >>> I have a PR that shows what this would roughly look like: (approach 1) >>> https://github.com/apache/iceberg/pull/17543 . The diff is pretty big >>> and we end up duplicating many classes and endpoints. >>> >>> Another way of solving this would be to make a required response field >>> optional (manifest-list and metadata-location) without bumping the >>> model/api versions. This might be a more practical approach assuming that >>> most clients don't break this way. This PR shows how minimal these changes >>> are: (approach 2) https://github.com/apache/iceberg/pull/17642 >>> >>> I lean toward the in-place approach, assuming few clients actually >>> require these fields to be present. Does that seem reasonable, or are there >>> compatibility concerns I'm missing? >>> >>> Regards, >>> Dhruv Arya >>> >>
