I think what you're saying is largely in line with what Dhruv is proposing. The one exception is the `oneOf` case (or I might have misunderstood you). The 'oneOf' just allows for the v2::loadtable to return the correctly structured response for newer clients. The v1::loadTable would only ever return the original structure for v1-3. (We're probably saying the same thing here; just want to clarify). There are many semantic requirements of the responses that are not enforced by OpenAPI, so it's more about the requirements on the catalog implementation and what it can return.
For #2: yes, some sort of error response from the catalog to not allow the client to load the table would be the correct response. This clearly indicates the issue to the client and doesn't rely on client-side handling of the response payload. I do think we want to combine changes necessary for v4, but I'm uncertain about taking on something like partial metadata loading as part of the v2 endpoint. We've built a lot of consensus about V4 features, but there are still strong opinions about the viability/necessity of a partial metadata load. On Mon, Aug 24, 2026 at 10:58 AM Yufei Gu <[email protected]> wrote: > Thanks Dan. I agree that clients cannot be required to process > `format-version` before other fields. I don’t think the REST spec needs to > ensure that old clients can load v4 tables. Old clients are not expected to > support future table format versions. However, one of the incompatiblity is > that the server should NOT return `200` with a response that violates its > documented schema(e.g., missing the required manifest-list path in a > snapshot) via v1 endpoint. That mighty be a reason for a v2 endpoint. > > Based on that distinction, the following approach makes sense to me: > > 1. The v1 `loadTable` endpoint MUST NOT return v4 table metadata. Adding a > `oneOf` alternative to the existing version of LoadTableResult would not > solve the problem for already released clients. For the new response model, > we could use `format-version` as a discriminator and guide clients to > dispatch on it before validating version-specific fields. The v2 > `loadTable` endpoint could then return metadata for v1–v4 tables. > 2. If v1 `loadTable` is used for a v4 table, should the server return a > documented `IcebergErrorResponse`, potentially a `406` with an error type > such as `UnsupportedTableFormatVersionException`? > > As a side note: if we introduce a v2 endpoint, perhaps we should collect > other requirements now to reduce the likelihood of another version bump in > the near future. One example is partially loading table metadata so that > catalogs do not need to return large snapshot histories, as discussed in > the earlier proposal [1]. > > [1] https://lists.apache.org/thread/qocy0w2y1wk1775x2n68mt9vompcbhmc > > Yufei > > > On Fri, Aug 21, 2026 at 10:30 PM Daniel Weeks <[email protected]> wrote: > >> Yufei, >> >> The compatibility issue is related to the conflict of relaxing the >> required table location field for relative patha and table version field. >> >> We cannot guarantee which order a client processes the fields. >> Therefore, if the missing location field is encountered, an older client >> would reasonably fail due the missing required field and never have an >> opportunity to perform the version check. >> >> There will be more fields potentially as we add/modify fields for the >> other features introduced in v4, but this one alone highlights the >> compatibility issue. >> >> -Dan >> >> On Thu, Aug 20, 2026, 3:10 PM Yufei Gu <[email protected]> wrote: >> >>> I largely agree with Dan and Amogh. I just want to clarify one point: >>> >>> > clients will fail to parse the JSON before they can determine it's a >>> v4 table >>> >>> TableMetadata is required in LoadTableResult, and format-version is >>> required within TableMetadata.[1] So, absent an incompatible change to the >>> metadata’s JSON structure or a strict client-side parser, a client should >>> be able to identify the table format version before interpreting the rest >>> of the metadata. If the payload is malformed, failure is expected for v1 >>> clients as well. That case alone does not seem like a compatibility concern. >>> >>> 1. >>> https://github.com/apache/polaris/blob/4a3fba7d4b26ef9c985086d5aba53b26b0b63935/spec/iceberg-rest-catalog-open-api.yaml#L3541 >>> >>> Yufei >>> >>> >>> On Tue, Aug 18, 2026 at 9:04 AM Daniel Weeks <[email protected]> wrote: >>> >>>> Hey Dhruv, >>>> >>>> I think the second approach is a much cleaner solution, though it >>>> relies on some nuance for backward compatibility with clients. >>>> >>>> A few points to note: >>>> 1) The REST spec's required/optional fields are not intended to match >>>> exactly what is in the spec. TableMetadata is a great example of this, >>>> where most fields are required in the table spec, but are optional in the >>>> REST spec. >>>> 2) Backward compatibility comes down to whether a client will break >>>> when receiving a payload that it doesn't understand. >>>> 3) Unfortunately, some of the gates you might expect to be possible >>>> (like table version) aren't feasible if you break client-side parsers >>>> (clients will fail to parse the JSON before they can determine it's a v4 >>>> table). >>>> >>>> All that said, the only thing I believe we need to do is add a >>>> `v2::loadTable` endpoint and require all v4 tables to be loaded through >>>> that mechanism. It's important to note that only load table requires a >>>> version bump since committing a table still use the normal snapshot update >>>> and clients will know from the supported endpoints whether v4 is even >>>> supported (via the v2::loadTable endpoint). >>>> >>>> I love to hear what others think, but I'm strongly in favor of both >>>> starting the process of introducing new versioned endpoints and scoping >>>> what we need for V4. >>>> >>>> There are a few open questions, like what other metadata changes will >>>> be necessary (e.g. check constraints/default expressions/generated >>>> columns/etc)? How do we want to introduce the new endpoint (e.g. allow it >>>> to evolve with breaking changes until we vote on v2 endpoint / v4 spec? >>>> >>>> I think this is a great start though and would love to hear what others >>>> think, >>>> -Dan >>>> >>>> On Thu, Aug 13, 2026 at 3:58 PM Dhruv Arya <[email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Kicking off this thread to discuss the initial REST spec changes >>>>> needed for Iceberg V4: >>>>> 1. root-manifest in Snapshot: We need the returned Snapshot to have a >>>>> root-manifest . (oneOf of root-manifest or manifest-list). For V4 tables, >>>>> manifest-list will not be set. >>>>> 2. Add table-location: LoadTableResult, CommitTableResponse, and >>>>> UnregisterTableResult all need to return table-location. When defined, >>>>> this >>>>> will be the absolute path of the table root. (Allows the paths inside the >>>>> metadata to be relative making table migration easier). >>>>> 3. Make metadata-location in CommitTableResponse optional: This is for >>>>> catalog managed tables. >>>>> >>>>> Some of these changes are not backwards compatible: >>>>> `manifest-list` in Snapshot is a required field right now. Some >>>>> clients might rely on this field being set in the response. If we allow >>>>> this to be empty when `root-manifest` is set, these clients could break. >>>>> The >>>>> same problem applies to `metadata-location`. Furthermore, many endpoints >>>>> (unregister, table, etc) transitively depend on Snapshot. If we create a >>>>> new SnapshotV2, many endpoints will need a version bump to V2. >>>>> >>>>> I have a PR that shows what this would roughly look like: (approach 1) >>>>> https://github.com/apache/iceberg/pull/17543 . The diff is pretty big >>>>> and we end up duplicating many classes and endpoints. >>>>> >>>>> Another way of solving this would be to make a required response field >>>>> optional (manifest-list and metadata-location) without bumping the >>>>> model/api versions. This might be a more practical approach assuming that >>>>> most clients don't break this way. This PR shows how minimal these changes >>>>> are: (approach 2) https://github.com/apache/iceberg/pull/17642 >>>>> >>>>> I lean toward the in-place approach, assuming few clients actually >>>>> require these fields to be present. Does that seem reasonable, or are >>>>> there >>>>> compatibility concerns I'm missing? >>>>> >>>>> Regards, >>>>> Dhruv Arya >>>>> >>>>
