[
https://issues.apache.org/jira/browse/HUDI-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362688#comment-17362688
]
Vinay edited comment on HUDI-1910 at 6/14/21, 9:09 AM:
-------------------------------------------------------
[~nishith29] Make sense, so you are suggesting to include
COMMIT_OFFSET_TO_KAFKA config in KafkaOffsetGen.Config class so that users can
include it in property file like we pass topic name.
And then use it here -
[https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L474]
and call commitOffsetToKafka function. is that correct ?
If this approach looks good, I can test this change out and create a PR
was (Author: vinaypatil18):
[~nishith29] Make sense, so you suggesting to include COMMIT_OFFSET_TO_KAFKA
config in KafkaOffsetGen.Config class so that users can include it in property
file like we pass topic name.
And then use it here -
[https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L474]
and call commitOffsetToKafka function.
If this approach looks good, I can test this change out and create a PR
> Supporting Kafka based checkpointing for HoodieDeltaStreamer
> ------------------------------------------------------------
>
> Key: HUDI-1910
> URL: https://issues.apache.org/jira/browse/HUDI-1910
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: Nishith Agarwal
> Assignee: Vinay
> Priority: Major
> Labels: sev:normal, triaged
>
> HoodieDeltaStreamer currently supports commit metadata based checkpoint. Some
> users have requested support for Kafka based checkpoints for freshness
> auditing purposes. This ticket tracks any implementation for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
