Mingliang Liu created FLINK-36780:
-------------------------------------
Summary: Kafka source disable partition discovery unexpectedly
Key: FLINK-36780
URL: https://issues.apache.org/jira/browse/FLINK-36780
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 2.0-preview, 1.19.1, 1.20.0, 1.18.1
Reporter: Mingliang Liu
Currently Kafka source enables partition discovery. This is set by
{{partition.discovery.interval.ms}}, aka
{{KafkaSourceOptions.PARTITION_DISCOVERY_INTERVAL_MS}}. The default value is 5
minutes, which is equal to the default value of {{metadata.max.age.ms}} in
Kafka.
However, it's disabled by default unexpectedly in the source builder
([code|https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L476-L480]).
The intention I believe was to only disable for bounded source.
We need a fix that is able to keep the default partition discovery. This could
cause data loss after Kafka retention if the new partitions are not consumed
silently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
