Bowen Li created FLINK-39054:
--------------------------------
Summary: DynamicKafkaSourceEnumStateSerializer should preserve
full per-cluster properties instead of only bootstrap.servers
Key: FLINK-39054
URL: https://issues.apache.org/jira/browse/FLINK-39054
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: kafka-4.0.1
Reporter: Bowen Li
Fix For: kafka-4.1.0
Problem:
DynamicKafkaSourceEnumStateSerializer persisted only `bootstrap.servers` in V2
, dropping other cluster properties on restore.
v2 serialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L146]
v2 deserialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L179]
It causes loss of configs like auth/network after checkpoint restore and avoids
incorrect defaults.
Solution:
Create ser/de V3 in DynamicKafkaSourceEnumStateSerializer that can checkpoint
full per-cluster Properties in a new V3 format, and still restores V1/V2 states.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
