Hongshun Wang created FLINK-32020:
-------------------------------------
Summary: Enable Dynamic Partition Discovery by Default in Kafka
Source based on FLIP-288
Key: FLINK-32020
URL: https://issues.apache.org/jira/browse/FLINK-32020
Project: Flink
Issue Type: Sub-task
Reporter: Hongshun Wang
As described in
[FLIP-288|https://cwiki.apache.org/confluence/display/FLINK/FLIP-288%3A+Enable+Dynamic+Partition+Discovery+by+Default+in+Kafka+Source],
dynamic partition discovery is disabled by default, and users have to specify
the interval of discovery in order to turn it on.
This subtask is to enable Dynamic Partition Discovery by Default in Kafka
Source.
Partition discovery is performed on the KafkaSourceEnumerator, which
asynchronously fetches topic metadata from the Kafka cluster and checks if
there are any new topics and partitions. This should not cause performance
issues on the Flink side.
On the Kafka broker side, partition discovery sends a MetadataRequest to the
Kafka broker to fetch topic information. Considering that the Kafka broker has
its metadata cache and the default request frequency is relatively low (once
every 30 seconds), this is not a heavy operation, and the broker's performance
will not be significantly affected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
