[
https://issues.apache.org/jira/browse/HUDI-8726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-8726:
------------------------------
Description:
Three things that need to be fixed or revisited:
* When translating checkpoints from written by 0.x to 1.0 (V2 checkpoint based
on completion time), we need to consider
[hoodie.read.timeline.holes.resolution.policy|https://hudi.apache.org/docs/configurations#hoodiereadtimelineholesresolutionpolicy].
If USE_TRANSITION_TIME is configure, the checkpoint written by 0.x is already
using transition or completion time, so it should not go through the checkpoint
translation from requested to completion time. This need to be fixed.
* For the case of source table version 8 ingested to target table version 6,
make sure that all modes of handling hollow commit
([hoodie.read.timeline.holes.resolution.policy|https://hudi.apache.org/docs/configurations#hoodiereadtimelineholesresolutionpolicy])
are still handled (should be already done on master branch through the V1
source or relation logic). Need to add tests if not present.
*
For S3EventsHoodieIncrSource and GcsEventsHoodieIncrSource, make sure that V1
checkpoint is always used regardless of source and target table versions, and
the incremental read is based on request times. (should be already done on
master branch). Need to validate and add tests if needed.
> Test and fix gaps around checkpoint adjustments
> -----------------------------------------------
>
> Key: HUDI-8726
> URL: https://issues.apache.org/jira/browse/HUDI-8726
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Y Ethan Guo
> Priority: Blocker
> Fix For: 1.0.1
>
> Original Estimate: 20h
> Remaining Estimate: 20h
>
> Three things that need to be fixed or revisited:
> * When translating checkpoints from written by 0.x to 1.0 (V2 checkpoint
> based on completion time), we need to consider
> [hoodie.read.timeline.holes.resolution.policy|https://hudi.apache.org/docs/configurations#hoodiereadtimelineholesresolutionpolicy].
> If USE_TRANSITION_TIME is configure, the checkpoint written by 0.x is
> already using transition or completion time, so it should not go through the
> checkpoint translation from requested to completion time. This need to be
> fixed.
> * For the case of source table version 8 ingested to target table version 6,
> make sure that all modes of handling hollow commit
> ([hoodie.read.timeline.holes.resolution.policy|https://hudi.apache.org/docs/configurations#hoodiereadtimelineholesresolutionpolicy])
> are still handled (should be already done on master branch through the V1
> source or relation logic). Need to add tests if not present.
> *
> For S3EventsHoodieIncrSource and GcsEventsHoodieIncrSource, make sure that V1
> checkpoint is always used regardless of source and target table versions, and
> the incremental read is based on request times. (should be already done on
> master branch). Need to validate and add tests if needed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
