Hi, Shawn - Thanks for starting the discussion on this. Your summary of our current thinking on upgrades is accurate. The upgrade is an O(1) operation that leaves prior manifest files intact. New table writes will produce the v4 metadata. This is by design because table upgrades that require expensive metadata rewrites add a lot of friction to upgrade to v4. Indeed, tables with older manifests don't get the performance benefits of v4, but they produce the correct results and are no worse in performance than the v3 table pre-upgrade. The table converges to the v4 metadata representation over time as older data age out and maintenance kicks in.
This is no different than any existing version upgrade: v1 to v2 or v2 to v3 upgrades didn't require expensive metadata or data rewrites. V4 doesn't make that upgrade process any worse. On the spec vs guidance. I lean towards guidance plus maintenance utilities rather than a prescriptive upgrade protocol that forces metadata rewrites. Best, Anoop On Tue, Aug 11, 2026 at 5:45 PM Shawn Chang <[email protected]> wrote: > Hi all, > > As V4 takes better shape daily, I think some items regarding the upgrade > path remain unclear. > > I'd like to better understand the expected V3 -> V4 upgrade path, > particularly the operational expectations around pre-V4 manifests after a > table is upgraded. > > From the current V4 work, my understanding is that an implementation can > perform a lightweight upgrade by creating a V4 root manifest that > references existing pre v4 manifests, and then writes new metadata in the > V4 format going forward. > > What I'm less clear about is whether we have an expected or recommended > lifecycle for those pre-V4 manifests after the upgrade. > > For example, is the expectation that an upgraded table may indefinitely > retain the structure below, until the user explicitly runs a manifest > rewrite/maintenance job? > > V4 root > -> pre-V4 data/delete manifests > -> V4 manifests > > If so, I worry that we may be shifting significant migration > responsibility onto users. In practice, users often don't run optional > table maintenance consistently. Leaving this undefined reminds me of > equality deletes. > > In short, I believe we can either 1) spec it or 2) provide general > guidance to avoid a situation where the format technically supports a clean > migration path, but the practical path relies on users understanding V4. > > Have we considered solutions where the implementation eagerly converts > metadata that is cheap to convert while explicitly leaving legacy data that > is expensive to migrate alone?(applying EqDels) > > > Best, > > Shawn > >
