nsivabalan edited a comment on pull request #2438: URL: https://github.com/apache/hudi/pull/2438#issuecomment-856862390
@liujinhui1994 : here is what we can do. If someone is running it just one, this should not be an issue. Issue arises when someone runs deltastreamer in a continuous manner. So, user is expected to set HoodieDeltaStreamer.Config.checkpoint or InitialCheckpointProvider. Also user sets the new config (hoodie.deltastreamer.source.kafka.checkpoint.type) to timestamp. KafkaOffset gen should be capable of parsing the checkpoint as timestamp. at the end write, deltaSync should reset this(...kafka.checkpoint.type) config (similar to how we reset the checkpoint). So, for subsequent runs, this(...kafka.checkpoint.type) config value will not be set. So, KafkaOffsetGen should parse checkpoint and fetch from source as a regular checkpoint. Let me know if you can understand the approach, and if it makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
