[
https://issues.apache.org/jira/browse/HUDI-7665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sagar Sumit updated HUDI-7665:
------------------------------
Status: In Progress (was: Open)
> Rolling upgrade to table version 8
> ----------------------------------
>
> Key: HUDI-7665
> URL: https://issues.apache.org/jira/browse/HUDI-7665
> Project: Apache Hudi
> Issue Type: Task
> Reporter: Sagar Sumit
> Assignee: Sagar Sumit
> Priority: Major
> Labels: hudi-1.0.0-beta2, pull-request-available
> Fix For: 1.0.0
>
>
>
> *Supported migration combinations*
> Obviously, within 1.x code line - writers, readers, table services all need
> to co-exist for both table versions (tv) 6 & 8.
> || ||Reader 0.x ||Reader 1.x tv=6/8||Table Services 0.x||Table Services 1.x
> tv=6||Table Services 1.x tv=8||
> |*Writer 1.x tv=6*|(/)|(/)|(x)|(/)|(x)|
> |*Writer 1.x tv=8*|(x)|(/)|(x)|(x)|(/)|
> |*Table Services 1.x tv=6*|(/)|(/)|(x)|(/)|(x)|
> |*Table Services 1.x tv=8*|(x)|(/)|(x)|(x)|(/)|
> We are making the practical restriction that table services 0.x, need to be
> stopped first, upgraded to new binary 1.x, before resuming operations even
> for tv=6 to be able to co-exist with a 1.x tv=6 writer.
> *Recommended migration steps*
> # Stop any async table services in 0.x completely.
> # Upgrade writers to 1.x with tv=6 (this won't auto-upgrade anything); 0.x
> readers will continue to work; writers can also be readers and will continue
> to read both tv=6.
> # Upgrade table services to 1.x with tv=6, and resume operations.
> # Upgrade all remaining readers to 1.x, with tv=6.
> # Redeploy writers with tv=8; table services and readers will adapt/pick up
> tv=8 on the fly.
> *Things to be done during rolling upgrade*
> the 6 to 8 upgrade handler, need to handle the following.
> | * Migrating table properties including key generators, payloads.
> * Migrating timeline to new layout
> * Full compact the table and clean, to get rid of log files?
> * Migration path for CDC and incremental queries.
> * Handle differences between 0.x and 1.x for stuff needed in upgrade e.g
> compaction (need to compact older file slice), rollback (marker differences
> b/w 0.14 and 0.15)|
>
> *Safe migration protocol :*
> <wip>
>
> *Downgrade tooling :*
> <wip>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
