Davis-Zhang-Onehouse opened a new pull request, #12718: URL: https://github.com/apache/hudi/pull/12718
### Change Logs Please start reviewing from commit with title "basic code refactor". All previous commits comes from https://github.com/apache/hudi/pull/12688. The change is about how we handle checkpoint override configurations for delta streamer with hoodie incremental source during auto upgrade: - Initially we are in table version 6 - Turn on auto upgrade and set checkpoint override would leads to ingestion failure with user friendly error message. - Auto upgrade without checkpoint related config override (--checkpoint and --ignore-checkpoint) will succeed. - After upgrading to table version 8, --checkpoint for hoodie incremental source must be compliant with format ``` resumeFromInstantRequestTime: <checkpoint> ``` for request time based checkpoint reset. Internally delta streamer for hoodie incremental source would convert this to the completion time of the same instant and start ingestion following completion time based ordering. Alternatively, user can reset checkpoint based on completion time using: ``` resumeFromInstantCompletionTime: <checkpoint> ``` Please note, the config only controls the way of how checkpoint is overridden, delta streamer always follow the completion time based handing for version 8 target hoodie tables. ### Impact Avoid corner case that user/hudi confuse checkpoint semantics during auto upgrade. ### Risk level (write none, low medium or high below) none. ### Documentation Update After hoodie incremental source can support completion time based checkpoint, we should update the doc about these rules. As of now, the hoodie incremental sources are still using request time based handling in 1.x. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
