cshuo commented on PR #19376: URL: https://github.com/apache/hudi/pull/19376#issuecomment-5115063179
> @danny0405 Hi Danny, yes your understanding is 100% correct. I might misunderstand your earlier. But yes, the issue I observed is that when user restarts the same job in a different bound range without cleaning up checkpoints (previously reading 01/01/2026 but the job fails somehow and has checkpoints written, then restarts the job with config of 01/02/2026) the job would silently and wrongly resume reading from 01/01/2026. So I'm thinking of failing this new run with clear errors, since logging a warning could likely be unnoticeable. Thanks for clarifying. This sounds more like a job-upgrade compatibility check on the platform side than source recovery semantics. For example, when a new job version is deployed with an existing checkpoint/savepoint, the platform should compare the old and new job specifications. If options that affect the source consumption scope have changed, it should reject the upgrade or require the job to start without the old state. This is also how other FLIP-27 sources are generally structured. KafkaSource checkpoints the assigned topic partitions and their current/stopping offsets because they are the actual execution state required to resume the splits. On restore, those offsets take precedence over the newly configured `OffsetsInitializer`; KafkaSource does not additionally persist the complete source configuration and compare it with the new configuration to decide whether the deployment is compatible. Table-format sources follow the same pattern. For example, Iceberg checkpoints pending splits and the last enumerated position, while Paimon checkpoints remaining splits and the next snapshot ID. They preserve enumeration progress needed for recovery, but do not copy the complete scan configuration into enumerator state solely for comparing it with the next job version. There is an important distinction here: persisting a bound as part of a split is reasonable when the reader needs it to finish that restored split, as Kafka does with its stopping offset. Persisting static source options only as a configuration fingerprint for deployment-time validation is different. It couples job-upgrade validation to the source checkpoint format and requires the state schema to keep expanding whenever another option affects split enumeration. Therefore, I don't think recording `read.start-commit` and `read.end-commit` in `HoodieSplitEnumeratorState` solely for comparing them with a future job configuration is the right layer for this feature. This compatibility check would be better handled by the job deployment platform before restoring the job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
