[
https://issues.apache.org/jira/browse/HUDI-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447757#comment-17447757
]
Manoj Govindassamy commented on HUDI-2475:
------------------------------------------
h2. Test 1: Single Writer + InLine Clustering - Pass
* Setup
** Delta Streamer continuous mode
** InLine clustering
** Metadata table turned on
* Started with 0.9.0 version, had bunch of upserts along with inline
clustering
** Used CLI to do metadata listing
** Used spark shell to list records and count
* Later upgraded to 0.10.0 version, with the same single writer setup
** 0.10.0 bootstrapping kicked in, nuked the entire metadata table and
re-initialized it
** After the initial metadata bootstrap, upserts started to flow in
successfully
** Used spark shell to list records and count
h2. Test 2: Writer + Async Clustering - Fail
* Setup
** Writer 1 - Delta Streamer continuous mode
** Async clustering turned on
** Metadata table turned on
* Started with 0.9.0 version, had bunch of upserts along with async clustering
** Verified clustering, cleaning and upserts were successful at 0.9.0
** Shutdown the writers - Deltastreamer + along with its async clustering
** Timed the shutdown such a way that there was an inflight replace commit in
the timeline
* Later upgraded to 0.10.0
** With only the Single Writer - Delta Streamer Continuous mode
** Async clustering turned off
** Writer detected the previous metadata version, nuked it fully
** But, the follow on bootstrapping failed as it complained about the inflight
replace commit
** Upserts started flowing in successfully without the metadata table
* Finishing up the pending clustering
** Shutdown the Single Writer and brought up the Clustering Execution job to
let the inflight replace commit to complete
** This job detected the files part of the plan non-existing and so it failed
the clustering job
* Bring up the writer again
** Brought up the single writer again
** Bootstrapping failed as the inflight replace commit was still there
** upserts flowing in with no metadata table
> Rolling Upgrade downgrade story for 0.10 & enabling metadata
> ------------------------------------------------------------
>
> Key: HUDI-2475
> URL: https://issues.apache.org/jira/browse/HUDI-2475
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: sivabalan narayanan
> Assignee: Manoj Govindassamy
> Priority: Blocker
> Fix For: 0.10.0
>
>
> Upgrade downgrade infra for enabling metadata.
>
> If user is having a writer process and clustering/compaction running async.
>
> - New synchronous metadata design, has a constraint that once metadata table
> is bootstrapped, all commits will happen synchronously. In other words, there
> is no catch up business wrt datatable.
> So, it may not be feasible to do rolling upgrade (i.e. upgrade writer first
> while async compaction is running) and then upgrade async compaction.
> Bootstrap has to be done by stopping all processes and then we can restart
> all other processes one by one (by using the upgraded hudi library) w/
> metadata enabled.
> This is the only viable option I can think of.
> 1. Stop all processes. Upgrade to hudi to a version w/ synchronous metadata.
> bring up one writer process w/ metadata config enabled. this will bootstrap
> the metadata table. and from there on, any new commits by the writer will do
> synchronous updates to metadata.
> Note: users can choose to upgrade via hudi-cli if need be. but easier would
> be to just start the writer. Expect some delay for first commit since
> bootstrap will be happening.
> 2. Once first commit in previous writer process completes successfully, we
> can restart all other processes. Upgrade the async table service (to hudi
> version w/ metadata enabled) and restart it. *Ensure metadata table is
> enabled across all processes.* Even if missed on one, could result in data
> loss.
>
> By this, once metadata table is bootstrapped, any new commits from all
> processes will be synced to metadata.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
