That sounds like a good plan to me. Even though v4 introduces a different metadata structure, we have been trying to minimize the downstream impact through the use of an adapter layer. <https://github.com/apache/iceberg/pull/16100> The new metadata structs used in the AMT are abstracted by the lower-level v4 reader/writers in iceberg-core.
On Wed, Aug 12, 2026 at 8:26 AM Russell Spitzer <[email protected]> wrote: > I agree with the previous comments. A V4 reader must be able to read > V1,V2, and V3 manifest-lists and manifests. I do think in the future we may > want to have a break just to give implementations a rest, but that's > probably not good to do in the same version we introduce brand new metadata > layouts. Updating metadata and similar items to V4 can be an implementation > decision or process; no spec work is required. > > I would definitely be open to considering V5 a breaking point and say you > can only go from Pure V4 to V5 or something like that. > > On Wed, Aug 12, 2026 at 9:24 AM Amogh Jahagirdar <[email protected]> wrote: > >> Hey all, >> >> I largely agree with Anoop here, the format largely has biased towards >> cheap upgrades, where readers just have to be able to handle the older >> structures as well. The spec defines how to assign first row IDs for older >> manifest entries etc. I do think that this is the right tradeoff to >> continue making but we can always have implementations which do "eager" >> upgrades on write, which I think is a good idea for "not huge" tables. >> >> In other words, while the format *enables* cheap upgrades, it's not a >> requirement for implementations to *have to perform* a "cheap" upgrade. >> >> I do think it's valid to jump from v1/v2 to v4; I can't see an obvious >> reason to prevent that. >> >> A few syncs ago, we also talked about having leaf manifests for v3 and >> older manifests in a v4 root manifest, having the ability to also reference >> manifest DVs. This was so that for large updates/deletes which touch older >> manifests, there's not a significant post-upgrade performance cliff from >> metadata write amplification when trying to produce the combined entry >> structure that v4 requires. This is more of an issue when many delete >> manifests are touched, because finding the corresponding data file entries >> is a potentially large scan over larger data manifests. >> >> I think the concerns around having a proliferation of older manifests are >> somewhat valid but practically I think with nearly every write pattern >> there will be ways to age out the older manifest structures, either on the >> commit path of an implementation or just in the background as part of >> rewrite manifests. In the update/delete case I referenced, at some point >> we'd need to compact the MDV's anyways for those older manifests, and an >> implementation could produce the combined entries for those manifests at >> that point where the cost of producing combined entry is reduced. In an >> append only case, older manifests would largely just need to be rewritten >> into newer manifests (probably in the background), and there's no 2 phase >> planning that's needed anyways in this case. >> >> Again though, all of the above are just implementation choices and I >> think we have all the tools to address these cases; my recommendation is >> that we keep the spec open enough to enable implementations to move into >> the newer structures, whenever and however they choose. I don't think >> the spec needs to define anything here so long as we know post-upgrade >> there's no degenerate cases that we don't have a path through. >> >> Thanks, >> >> Amogh Jahagirdar >> >> On Tue, Aug 11, 2026 at 11:03 PM <[email protected]> wrote: >> >>> Hi Shawn & Anoop, >>> >>> Thank you Shawn for the clear write up. I am also worried that if we >>> don’t provide new ways to more intentionally upgrade tables sufficiently >>> significantly large pre-v4 tables may never fully complete their migrations. >>> >>> This concern for me personally is mostly forward looking. I am a little >>> anxious about what corners that would put us into come Table format v6 or >>> v7. The scenario I’m imaging is where query performance is no longer >>> consistent because a user will never know if their query is about to hit a >>> v3, or v4, or v5, or v6, etc etc manifest/data file. >>> >>> With Iceberg definitely becoming a solution for multi Petabyte sized >>> tables and ideally aiming to support multi Exabyte scale tables. Waiting >>> for the current maintenance options to rewrite (compact) files leaves us >>> with few options for more intentional, incremental and asynchronous ways of >>> catching a table up to the intended format version for all files. >>> >>> Anoop, question because I missed a few of the AMT syncs. Have we >>> discussed if it will be valid to jump straight from V1/V2 to V4 in the >>> proposed reference implementation or will users need to go V3 to V4? I’m >>> expecting that it would be valid for a V1 -> V4 jump, but I don’t want to >>> assume. >>> >>> Best Regards, >>> Kurtis >>> >>> On Aug 11, 2026, at 18:51, Anoop Johnson <[email protected]> wrote: >>> >>> >>> Hi, Shawn - >>> >>> Thanks for starting the discussion on this. Your summary of our current >>> thinking on upgrades is accurate. The upgrade is an O(1) operation that >>> leaves prior manifest files intact. New table writes will produce the v4 >>> metadata. This is by design because table upgrades that require expensive >>> metadata rewrites add a lot of friction to upgrade to v4. Indeed, tables >>> with older manifests don't get the performance benefits of v4, but they >>> produce the correct results and are no worse in performance than the v3 >>> table pre-upgrade. The table converges to the v4 metadata representation >>> over time as older data age out and maintenance kicks in. >>> >>> This is no different than any existing version upgrade: v1 to v2 or v2 >>> to v3 upgrades didn't require expensive metadata or data rewrites. V4 >>> doesn't make that upgrade process any worse. >>> >>> On the spec vs guidance. I lean towards guidance plus maintenance >>> utilities rather than a prescriptive upgrade protocol that forces >>> metadata rewrites. >>> >>> Best, >>> Anoop >>> >>> On Tue, Aug 11, 2026 at 5:45 PM Shawn Chang <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> As V4 takes better shape daily, I think some items regarding the >>>> upgrade path remain unclear. >>>> >>>> I'd like to better understand the expected V3 -> V4 upgrade path, >>>> particularly the operational expectations around pre-V4 manifests after a >>>> table is upgraded. >>>> >>>> From the current V4 work, my understanding is that an implementation >>>> can perform a lightweight upgrade by creating a V4 root manifest that >>>> references existing pre v4 manifests, and then writes new metadata in the >>>> V4 format going forward. >>>> >>>> What I'm less clear about is whether we have an expected or recommended >>>> lifecycle for those pre-V4 manifests after the upgrade. >>>> >>>> For example, is the expectation that an upgraded table may indefinitely >>>> retain the structure below, until the user explicitly runs a manifest >>>> rewrite/maintenance job? >>>> >>>> V4 root >>>> -> pre-V4 data/delete manifests >>>> -> V4 manifests >>>> >>>> If so, I worry that we may be shifting significant migration >>>> responsibility onto users. In practice, users often don't run optional >>>> table maintenance consistently. Leaving this undefined reminds me of >>>> equality deletes. >>>> >>>> In short, I believe we can either 1) spec it or 2) provide general >>>> guidance to avoid a situation where the format technically supports a clean >>>> migration path, but the practical path relies on users understanding V4. >>>> >>>> Have we considered solutions where the implementation eagerly converts >>>> metadata that is cheap to convert while explicitly leaving legacy data that >>>> is expensive to migrate alone?(applying EqDels) >>>> >>>> >>>> Best, >>>> >>>> Shawn >>>> >>>>
