[
https://issues.apache.org/jira/browse/FLINK-39054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-39054:
-----------------------------
Description:
Problem:
DynamicKafkaSourceEnumStateSerializer persists only `bootstrap.servers` in V1,
dropping other cluster properties on restore. We added V2 in FLINK-38876 while
maintaining the same configs as V1
serialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L146]
deserialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L179]
Essentially, V1 state causes loss of configs like auth/network after checkpoint
restore and avoids incorrect defaults.
Solution:
Since DynamicKafkaSourceEnumStateSerializer V2 is not released with a new
flink-connector-kafka version yet as of 2/9/2026, we can update V2 of
DynamicKafkaSourceEnumStateSerializer to checkpoint full per-cluster
Properties, and still restores V1 states.
was:
Problem:
DynamicKafkaSourceEnumStateSerializer persisted only `bootstrap.servers` in V2
, dropping other cluster properties on restore.
v2 serialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L146]
v2 deserialization:
[https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L179]
It causes loss of configs like auth/network after checkpoint restore and avoids
incorrect defaults.
Solution:
Create ser/de V3 in DynamicKafkaSourceEnumStateSerializer that can checkpoint
full per-cluster Properties in a new V3 format, and still restores V1/V2 states.
> DynamicKafkaSourceEnumStateSerializer should preserve full per-cluster
> properties instead of only bootstrap.servers
> -------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-39054
> URL: https://issues.apache.org/jira/browse/FLINK-39054
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: kafka-4.0.1
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
> Fix For: kafka-4.1.0
>
>
> Problem:
> DynamicKafkaSourceEnumStateSerializer persists only `bootstrap.servers` in
> V1, dropping other cluster properties on restore. We added V2 in FLINK-38876
> while maintaining the same configs as V1
> serialization:
> [https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L146]
> deserialization:
> [https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L179]
> Essentially, V1 state causes loss of configs like auth/network after
> checkpoint restore and avoids incorrect defaults.
>
> Solution:
> Since DynamicKafkaSourceEnumStateSerializer V2 is not released with a new
> flink-connector-kafka version yet as of 2/9/2026, we can update V2 of
> DynamicKafkaSourceEnumStateSerializer to checkpoint full per-cluster
> Properties, and still restores V1 states.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
