Raymond Xu created HUDI-5906:
--------------------------------
Summary: Consolidate checkpointing logic with deltastreamer into
CheckpointStrategy model
Key: HUDI-5906
URL: https://issues.apache.org/jira/browse/HUDI-5906
Project: Apache Hudi
Issue Type: Improvement
Components: deltastreamer
Reporter: Raymond Xu
Fix For: 0.14.0
checkpointing logic with deltastreamer is unorganized. there are logic spread
in different places
- HoodieDeltaStreamer#CHECKPOINT_KEY
- HoodieDeltaStreamer#CHECKPOINT_RESET_KEY
- HoodieDeltaStreamer#CHECKPOINT_FORCE_SKIP_PROP
- org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#checkpoint
-
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#initialCheckpointProvider
-
org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.Config#allowCommitOnNoCheckpointChange
- MUTLI_WRITER_SOURCE_CHECKPOINT_ID
- org.apache.hudi.utilities.checkpointing.InitialCheckPointProvider
-
org.apache.hudi.utilities.sources.HoodieIncrSource.Config#MISSING_CHECKPOINT_STRATEGY
-
org.apache.hudi.utilities.sources.helpers.IncrSourceHelper.MissingCheckpointStrategy
There should be a consolidated model say HoodieCheckpointStrategy to manage all
these logic with pluggable APIs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
