Hi Dan Thanks for the update, I will take a look.
Regards JB On Wed, Nov 11, 2026 at 11:01 PM Daniel Weeks via dev < [email protected]> wrote: > Hey everyone, > > Following up from the last REST catalog sync discussion, I created a > branch named `rest-v4-changes` to collect spec/openapi changes related to > introducing v4 tables. > > The branch should be isolated to spec changes, which keeps the rebase > surface area very small (effectively just the openapi spec yaml file). > > Implementations for requests/repsonse or metadata parsers are expected to > be reviewed and baselined as we normally do. This will provide more > flexibility to update the spec as we work on the implementation. > > Please take a look a the PRs (17642 > <https://github.com/apache/iceberg/pull/17642>) and provide feedback, > Dan > > On Mon, Aug 24, 2026 at 12:53 PM Daniel Weeks <[email protected]> wrote: > >> I think what you're saying is largely in line with what Dhruv is >> proposing. The one exception is the `oneOf` case (or I might have >> misunderstood you). The 'oneOf' just allows for the v2::loadtable to >> return the correctly structured response for newer clients. The >> v1::loadTable would only ever return the original structure for v1-3. >> (We're probably saying the same thing here; just want to clarify). There >> are many semantic requirements of the responses that are not enforced by >> OpenAPI, so it's more about the requirements on the catalog >> implementation and what it can return. >> >> For #2: yes, some sort of error response from the catalog to not allow >> the client to load the table would be the correct response. This clearly >> indicates the issue to the client and doesn't rely on client-side handling >> of the response payload. >> >> I do think we want to combine changes necessary for v4, but I'm >> uncertain about taking on something like partial metadata loading as part >> of the v2 endpoint. We've built a lot of consensus about V4 features, but >> there are still strong opinions about the viability/necessity of a partial >> metadata load. >> >> On Mon, Aug 24, 2026 at 10:58 AM Yufei Gu <[email protected]> wrote: >> >>> Thanks Dan. I agree that clients cannot be required to process >>> `format-version` before other fields. I don’t think the REST spec needs to >>> ensure that old clients can load v4 tables. Old clients are not expected to >>> support future table format versions. However, one of the incompatiblity is >>> that the server should NOT return `200` with a response that violates its >>> documented schema(e.g., missing the required manifest-list path in a >>> snapshot) via v1 endpoint. That mighty be a reason for a v2 endpoint. >>> >>> Based on that distinction, the following approach makes sense to me: >>> >>> 1. The v1 `loadTable` endpoint MUST NOT return v4 table metadata. Adding >>> a `oneOf` alternative to the existing version of LoadTableResult would not >>> solve the problem for already released clients. For the new response model, >>> we could use `format-version` as a discriminator and guide clients to >>> dispatch on it before validating version-specific fields. The v2 >>> `loadTable` endpoint could then return metadata for v1–v4 tables. >>> 2. If v1 `loadTable` is used for a v4 table, should the server return a >>> documented `IcebergErrorResponse`, potentially a `406` with an error type >>> such as `UnsupportedTableFormatVersionException`? >>> >>> As a side note: if we introduce a v2 endpoint, perhaps we should collect >>> other requirements now to reduce the likelihood of another version bump in >>> the near future. One example is partially loading table metadata so that >>> catalogs do not need to return large snapshot histories, as discussed in >>> the earlier proposal [1]. >>> >>> [1] https://lists.apache.org/thread/qocy0w2y1wk1775x2n68mt9vompcbhmc >>> >>> Yufei >>> >>> >>> On Fri, Aug 21, 2026 at 10:30 PM Daniel Weeks <[email protected]> wrote: >>> >>>> Yufei, >>>> >>>> The compatibility issue is related to the conflict of relaxing the >>>> required table location field for relative patha and table version field. >>>> >>>> We cannot guarantee which order a client processes the fields. >>>> Therefore, if the missing location field is encountered, an older client >>>> would reasonably fail due the missing required field and never have an >>>> opportunity to perform the version check. >>>> >>>> There will be more fields potentially as we add/modify fields for the >>>> other features introduced in v4, but this one alone highlights the >>>> compatibility issue. >>>> >>>> -Dan >>>> >>>> On Thu, Aug 20, 2026, 3:10 PM Yufei Gu <[email protected]> wrote: >>>> >>>>> I largely agree with Dan and Amogh. I just want to clarify one point: >>>>> >>>>> > clients will fail to parse the JSON before they can determine it's >>>>> a v4 table >>>>> >>>>> TableMetadata is required in LoadTableResult, and format-version is >>>>> required within TableMetadata.[1] So, absent an incompatible change to the >>>>> metadata’s JSON structure or a strict client-side parser, a client should >>>>> be able to identify the table format version before interpreting the rest >>>>> of the metadata. If the payload is malformed, failure is expected for v1 >>>>> clients as well. That case alone does not seem like a compatibility >>>>> concern. >>>>> >>>>> 1. >>>>> https://github.com/apache/polaris/blob/4a3fba7d4b26ef9c985086d5aba53b26b0b63935/spec/iceberg-rest-catalog-open-api.yaml#L3541 >>>>> >>>>> Yufei >>>>> >>>>> >>>>> On Tue, Aug 18, 2026 at 9:04 AM Daniel Weeks <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey Dhruv, >>>>>> >>>>>> I think the second approach is a much cleaner solution, though it >>>>>> relies on some nuance for backward compatibility with clients. >>>>>> >>>>>> A few points to note: >>>>>> 1) The REST spec's required/optional fields are not intended to match >>>>>> exactly what is in the spec. TableMetadata is a great example of this, >>>>>> where most fields are required in the table spec, but are optional in the >>>>>> REST spec. >>>>>> 2) Backward compatibility comes down to whether a client will break >>>>>> when receiving a payload that it doesn't understand. >>>>>> 3) Unfortunately, some of the gates you might expect to be possible >>>>>> (like table version) aren't feasible if you break client-side parsers >>>>>> (clients will fail to parse the JSON before they can determine it's a v4 >>>>>> table). >>>>>> >>>>>> All that said, the only thing I believe we need to do is add a >>>>>> `v2::loadTable` endpoint and require all v4 tables to be loaded through >>>>>> that mechanism. It's important to note that only load table requires a >>>>>> version bump since committing a table still use the normal snapshot >>>>>> update >>>>>> and clients will know from the supported endpoints whether v4 is even >>>>>> supported (via the v2::loadTable endpoint). >>>>>> >>>>>> I love to hear what others think, but I'm strongly in favor of both >>>>>> starting the process of introducing new versioned endpoints and scoping >>>>>> what we need for V4. >>>>>> >>>>>> There are a few open questions, like what other metadata changes will >>>>>> be necessary (e.g. check constraints/default expressions/generated >>>>>> columns/etc)? How do we want to introduce the new endpoint (e.g. allow >>>>>> it >>>>>> to evolve with breaking changes until we vote on v2 endpoint / v4 spec? >>>>>> >>>>>> I think this is a great start though and would love to hear what >>>>>> others think, >>>>>> -Dan >>>>>> >>>>>> On Thu, Aug 13, 2026 at 3:58 PM Dhruv Arya <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Kicking off this thread to discuss the initial REST spec changes >>>>>>> needed for Iceberg V4: >>>>>>> 1. root-manifest in Snapshot: We need the returned Snapshot to have >>>>>>> a root-manifest . (oneOf of root-manifest or manifest-list). For V4 >>>>>>> tables, >>>>>>> manifest-list will not be set. >>>>>>> 2. Add table-location: LoadTableResult, CommitTableResponse, and >>>>>>> UnregisterTableResult all need to return table-location. When defined, >>>>>>> this >>>>>>> will be the absolute path of the table root. (Allows the paths inside >>>>>>> the >>>>>>> metadata to be relative making table migration easier). >>>>>>> 3. Make metadata-location in CommitTableResponse optional: This is >>>>>>> for catalog managed tables. >>>>>>> >>>>>>> Some of these changes are not backwards compatible: >>>>>>> `manifest-list` in Snapshot is a required field right now. Some >>>>>>> clients might rely on this field being set in the response. If we allow >>>>>>> this to be empty when `root-manifest` is set, these clients could >>>>>>> break. The >>>>>>> same problem applies to `metadata-location`. Furthermore, many endpoints >>>>>>> (unregister, table, etc) transitively depend on Snapshot. If we create a >>>>>>> new SnapshotV2, many endpoints will need a version bump to V2. >>>>>>> >>>>>>> I have a PR that shows what this would roughly look like: (approach >>>>>>> 1) https://github.com/apache/iceberg/pull/17543 . The diff is >>>>>>> pretty big and we end up duplicating many classes and endpoints. >>>>>>> >>>>>>> Another way of solving this would be to make a required response >>>>>>> field optional (manifest-list and metadata-location) without bumping the >>>>>>> model/api versions. This might be a more practical approach assuming >>>>>>> that >>>>>>> most clients don't break this way. This PR shows how minimal these >>>>>>> changes >>>>>>> are: (approach 2) https://github.com/apache/iceberg/pull/17642 >>>>>>> >>>>>>> I lean toward the in-place approach, assuming few clients actually >>>>>>> require these fields to be present. Does that seem reasonable, or are >>>>>>> there >>>>>>> compatibility concerns I'm missing? >>>>>>> >>>>>>> Regards, >>>>>>> Dhruv Arya >>>>>>> >>>>>>
