Hello,
Thank you. I think it is a problem caused by Kafka configuration, not
Flink. I'll take a look and let you know if there's an issue in Flink.
BR,
Jung
2023년 8월 15일 (화) 오후 9:40, Hector Rios 님이 작성:
> Hi there
>
> It would be helpful if you could include the code for your pipeline. One
> suggestion, can you disable the "EXACTLY_ONCE" semantics on the producer.
> Using EXACTLY_ONCE will leverage Kafka transactions and thus add overhead.
> I would disable it to see if you still get the same situation.
>
> Also, can you look in the Flink UI for this job and see if checkpoints are
> in fact being taken?
>
> Hope that helps
> -Hector
>
> On Tue, Aug 15, 2023 at 11:36 AM Dennis Jung wrote:
>
>> Sorry, I've forgot putting title, so sending again.
>>
>> 2023년 8월 15일 (화) 오후 6:27, Dennis Jung 님이 작성:
>>
>>> (this is issue from Flink 1.14)
>>>
>>> Hello,
>>>
>>> I've set up following logic to consume messages from kafka, and produce
>>> them to another kafka broker. For producer, I've configured
>>> `Semantics.EXACTLY_ONCE` to send messages exactly once. (also setup
>>> 'StreamExecutionEnvironment::enableCheckpointing' as
>>> 'CheckpointingMode.EXACTLY_ONCE')
>>>
>>>
>>>
>>> kafka A -> FlinkKafkaConsumer -> ... -> FlinkKafkaProducer -> kafka B
>>>
>>>
>>>
>>> But though I've just produced only 1 message to 'kafka A', consumer
>>> consumes the same message repeatedly.
>>>
>>> When I remove `FlinkKafkaProducer` part and make it 'read only', it does
>>> not happen.
>>>
>>> Could someone suggest a way to debug or fix this?
>>>
>>> Thank you.
>>>
>>
