Muhammad Hilmi Al Fatih created FLINK-29270:
-----------------------------------------------
Summary: Wrong metrics kafka producer (FlinkKafkaProducer,
KafkaSink) in EXACTLY_ONCE
Key: FLINK-29270
URL: https://issues.apache.org/jira/browse/FLINK-29270
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka, Runtime / Metrics
Affects Versions: 1.14.4
Reporter: Muhammad Hilmi Al Fatih
Attachments: image-2022-09-13-11-41-51-759.png,
image-2022-09-13-11-42-45-297.png, image-2022-09-13-11-43-38-226.png,
image-2022-09-13-11-43-50-441.png
I did a very simple kafka-to-kafka pipeline with flink 1.14.4 with checkpoint.
I did 4 tests to confirm: for each old API
(FlinkKafkaConsumer&FlinkKafkaProducer) and new API (KafkaSource&KafkaSink), I
run AT_LEAST_ONCE and EXACLTY_ONCE.
However, the metrics shown by producer is not correct when doing EXACTLY_ONCE
semantics in both APIs.
To understand the situation, I also check the incoming rate metrics for both
source and sink topics. So here are the summary:
h3. Old API
h4. At Least Once
!image-2022-09-13-11-41-51-759.png|width=616,height=190!
h4. Exactly Once
!image-2022-09-13-11-42-45-297.png!
h3. New API (KafkaSource&KafkaSink)
h4. At Least Once
!image-2022-09-13-11-43-38-226.png!
h4. Exactly Once
!image-2022-09-13-11-43-50-441.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
