danny0405 commented on PR #19376: URL: https://github.com/apache/hudi/pull/19376#issuecomment-5114097964
> DefaultHoodieSplitProvider.state() persists only pendingSplits, all stamped UNASSIGNED — assigned and completed splits aren't in the checkpoint at all. splits that already got assigned will also be checkpointed if it is not finished, and each split will remember the position it consumes to, which can be used for job recovery where the next run consumes from. As a user, my understanding is that I can declare the start/end commit for the bounded streaming(batch) job, and the job will recover automatically from the state to ensure consistency. I'm assuming the case we want to fix in this patch is not for job auto recovery, but a manual restart of the job with an optional flink state, and you want to validate the state and the user config consume range is consistent, is that right? currenly Flink has no good API to read the state, so there is no good way to figure out the consumption offset/progress from the state, what we can do is probably validate whether the offset in the state is still in the consumption range that user specified? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
