Hi all, As V4 takes better shape daily, I think some items regarding the upgrade path remain unclear.
I'd like to better understand the expected V3 -> V4 upgrade path, particularly the operational expectations around pre-V4 manifests after a table is upgraded. >From the current V4 work, my understanding is that an implementation can perform a lightweight upgrade by creating a V4 root manifest that references existing pre v4 manifests, and then writes new metadata in the V4 format going forward. What I'm less clear about is whether we have an expected or recommended lifecycle for those pre-V4 manifests after the upgrade. For example, is the expectation that an upgraded table may indefinitely retain the structure below, until the user explicitly runs a manifest rewrite/maintenance job? V4 root -> pre-V4 data/delete manifests -> V4 manifests If so, I worry that we may be shifting significant migration responsibility onto users. In practice, users often don't run optional table maintenance consistently. Leaving this undefined reminds me of equality deletes. In short, I believe we can either 1) spec it or 2) provide general guidance to avoid a situation where the format technically supports a clean migration path, but the practical path relies on users understanding V4. Have we considered solutions where the implementation eagerly converts metadata that is cheap to convert while explicitly leaving legacy data that is expensive to migrate alone?(applying EqDels) Best, Shawn
