[
https://issues.apache.org/jira/browse/HUDI-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532753#comment-17532753
]
Ji Qi commented on HUDI-4006:
-----------------------------
hi [~alexey.kudinkin], here's the proposal
allow user to configure failOnDataLoss, two options
* allow setting {{auto.offset.reset=none}} and use that to mean failOnDataLoss
* add a new config key
in
[here|https://github.com/apache/hudi/blob/20781a5fa6cfcb0069855e57a1974db435de37c9/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KafkaOffsetGen.java#L301]
there is already a logic detecting whether the checkpoint offsets are in
range, modify its behaviour as such
* if failOnDataLoss=false, keep current behaviour but log a warning (instead of
seeking silently)
* if failOnDataLoss=true throw
> Fail on data loss semantics for deltastreamer Kafka sources
> -----------------------------------------------------------
>
> Key: HUDI-4006
> URL: https://issues.apache.org/jira/browse/HUDI-4006
> Project: Apache Hudi
> Issue Type: New Feature
> Components: deltastreamer
> Reporter: Ji Qi
> Assignee: Ji Qi
> Priority: Minor
> Fix For: 0.12.0
>
>
> See https://github.com/apache/hudi/issues/5400 for more details
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
