[
https://issues.apache.org/jira/browse/HUDI-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17362600#comment-17362600
]
Nishith Agarwal commented on HUDI-1910:
---------------------------------------
[~vinaypatil18] Thanks for sharing your approach. The first level of configs in
deltastreamer are meant to be for generic use-cases that apply to general
purpose ingestion activities. Whenever we want to add a specific use-case, we
add configurable classes and then add a parameter something like this ->
[https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java#L158.]
This has 2 advantages
# If you want to dynamically enable or disable such configs without code
changes or deployment, it can be done if you are keeping your properties file
updated with configs.
# Keep users away from many custom configs (which most users might not care)
by not floating them as top level configs in deltasteramer (they way you
suggested).
I think we should consider the first approach.
> Supporting Kafka based checkpointing for HoodieDeltaStreamer
> ------------------------------------------------------------
>
> Key: HUDI-1910
> URL: https://issues.apache.org/jira/browse/HUDI-1910
> Project: Apache Hudi
> Issue Type: Improvement
> Components: DeltaStreamer
> Reporter: Nishith Agarwal
> Assignee: Vinay
> Priority: Major
> Labels: sev:normal, triaged
>
> HoodieDeltaStreamer currently supports commit metadata based checkpoint. Some
> users have requested support for Kafka based checkpoints for freshness
> auditing purposes. This ticket tracks any implementation for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
