ericyuan915 commented on PR #19376: URL: https://github.com/apache/hudi/pull/19376#issuecomment-5114582185
> > DefaultHoodieSplitProvider.state() persists only pendingSplits, all stamped UNASSIGNED — assigned and completed splits aren't in the checkpoint at all. > > splits that already got assigned will also be checkpointed if it is not finished, and each split will remember the position it consumes to, which can be used for job recovery where the next run consumes from. > > As a user, my understanding is that I can declare the start/end commit for the bounded streaming(batch) job, and the job will recover automatically from the state to ensure consistency. > > I'm assuming the case we want to fix in this patch is not for job auto recovery, but a manual restart of the job with an optional flink state, and you want to validate the state and the user config consume range is consistent, is that right? currenly Flink has no good API to read the state, so there is no good way to figure out the consumption offset/progress from the state, what we can do is probably validate whether the offset in the state is still in the consumption range that user specified? @danny0405 Hi Danny, yes your understanding is 100% correct. I might misunderstand your earlier. But yes, the issue I observed is that when user restarts the same job in a different bound range without cleaning up checkpoints (previously reading 01/01/2026 but the job fails somehow and has checkpoints written, then restarts the job with config of 01/02/2026) the job would silently and wrongly resume reading from 01/01/2026. So I'm thinking of failing this new run with clear errors, since logging a warning could likely be unnoticeable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
