ericyuan915 commented on PR #19376: URL: https://github.com/apache/hudi/pull/19376#issuecomment-5098657916
@danny0405 Thanks for review! Regarding the 2 high-level questions: 1. Yeah we have use case that is backfilling 1 partition (day) of 55 billion records using batch source, and it takes about 4-5 hours. It takes days to backfill a few days of data. Due to a variety of issues, the batch backfill job could fail from time to time due to network issues, infra issues and so on. This would waste all the progress before failure. So we're looking to enable checkpoint for such use case, so the existing progress could be saved and resumed. 2. I agree on this actually. Users should make sure these are right. This PR would mainly focus on if user puts the different date ranges, and forgot to delete checkpoints, we should have the job fail fast instead of silently doing the wrong thing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
