Davis Zhang created HUDI-8917:
---------------------------------
Summary: Reset checkpoint handling
Key: HUDI-8917
URL: https://issues.apache.org/jira/browse/HUDI-8917
Project: Apache Hudi
Issue Type: Sub-task
Reporter: Davis Zhang
h2. Motivation
When target table upgrades to v2, it uses completion time to track the ckp. In
v1, it uses request time.
>From the user's perspective: * they only have 1 option for resetting the
>checkpoint. It does not imply it is completion time based or request time
>based.
* the interpretation of those single ckp options are dependent on which
version the target table is, which is beyond user's knowledge.
>From the system's standpoint: * It has no idea the user configured config is a
>request time or completion time. If we misread the semantics the behavior is
>totally different.
Target: * When user set the checkpoint, it should be easy for them to figure
out if they are setting completion time or request time.
* When system playing with configs, it should treat all configs with the same
semantics, we are only in danger when we compare request time with completion
time.
Acceptable case for simplifying the design: * Given the chance is low that 1
instant's start time is another instants' completion time, operations like
{{aCompletionTime == or != aRequestTimeConfig}} is allowed. But we should never
allow {{<}} {{>}} {{<=}} {{>=}} between {{aRequestTimeConfig}} and
{{aCompletionTime}}
* User streamer config:
** cfg.ckp needs {{<}} {{>}} {{<=}} {{>=}}
** cfg.ignore_ckp only needs {{== / !=}}
* Commit metadata
** ckp needs {{<}} {{>}} {{<=}} {{>=}}
** ignore_ckp and reset_ckp only needs {{== / !=}}
h3. Broken case 1 - User wants to override checkpoint while upgrading
{{cfg.checkpoint}}
As a result, when user would like to override the checkpoint while target table
upgrade is happening, they don't know if this is reset to some commit
completion time or request time.
*Auto Inference:* If it maps to request time of some instant, then we assume it
is request time. If it maps to some completion time, it is used as completion
time.
Cons: User might not always set a time whose value maps to any instant
request/completion time. Also {{cfg.checkpoint}} might involves in {{<}} {{>}}
{{<=}} {{>=}} operations, so inferencing is not acceptable.
*Introduce checkpoint v2 config:* Users make it explicit that I'm using v2
config and we should use completion time. If the table has not upgraded to v2,
we error out. If table has upgraded, we only allow v2.
Cons: 1 more user config. Require user action
*(Preferred) Do not allow user override ckp while upgrading:* If user set the
ckp and we are trying to upgrade, fail the ingestion.
needsUpgradeOrDowngrade(metaClient) && cfg.checkpoint/ignoreCheckpoint => throw
exception
Unless they turn off autoUpgrade
h3. Case 2 - Ckp config from the previous commit if we go with (Preferred) above
For reset ckp already taken effect before the upgrade, the reset ckp is
interpreted as a request time. Later when the upgrade completes, the reset ckp
is interpreted as completion time. Will this cause any issue?
timeline
t0.instant1.requested t0.instant2.completed t0_t1.instant1.completed
t1.instant3.requested t1_t4.instant3.completed t4.instant4.requested
prev commit metadata before upgrade, they are in the same semantic
{ ckp.v1: t60 <-- request time reset_ckp.v1: t50 <-- request time }
after upgrade, in the first ingestion we do the following: # first resolve
between reset_ckp.v1 and ckp.v1. They do not equal, so we continue with t1.
*Because we do comparison before translation, we can never compare reset_key as
a completion time to ckp as a request time.*
# then the translation happens for t1, for non USE_TRANSITION_TIME case we
convert to completion time t4. Then we start from there. Let's say it stop at
some instant t5. Then we will end up with something as follows
*Just no-op*
{ ckp.v2: t60 <-- completion time reset_ckp.v2: t50 <-- request time }
Please note here we failed to keep the 2 options under the same semantics, so
in the next ingestion, we might compare t0 and t5 and they are the same which
leads to issues.
*Just don't write* {{*reset_ckp.v1*}} *when upgrading to* {{*reset_ckp.v2*}} **
{ ckp.v2: t5 <-- completion time reset_ckp.v2: null }
----
1. getCheckpointToResumeFrom candidate ckp config to consider: -
commitMetadata.ckp (1234) -> set checkpoint.ckp as commitMetadata.ckp (1234) -
cfg.ckp (v1/v2:1234) -> set checkpoint.ckp as cfg.ckp (1234)
checkpoint.reset_ckp (raw value - v1/v2:1234) - cfg.ignore_ckp (1234) -> set
checkpoint.ckp as null
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
