gaoxin-oai opened a new pull request, #260: URL: https://github.com/apache/flink-connector-kafka/pull/260
## What changed - add `stream-metadata-removed-cluster-retention-ms` for Dynamic Kafka Source - keep removed-cluster reader split offsets in checkpointed split state until the retention expires - keep removed-cluster enumerator state separately so a re-added cluster can resume without duplicate assignment - serialize retained split and enumerator state with backward-compatible serializer version bumps - document the option in the English and Chinese dynamic Kafka docs ## Why When metadata removes a Kafka cluster, the next checkpoint currently drops that cluster from both reader and enumerator state. If the cluster is re-added or the job restores from a later checkpoint, the source no longer has the previous offsets for that cluster. This adds an opt-in retention window. Setting `stream-metadata-removed-cluster-retention-ms=604800000` keeps removed-cluster checkpoint state for 7 days. ## Validation - `DOCKER_HOST="unix:///Users/gaoxin/.colima/default/docker.sock" TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE="/var/run/docker.sock" TESTCONTAINERS_HOST_OVERRIDE="127.0.0.1" mvn -pl flink-connector-kafka -Dtest=DynamicKafkaSourceEnumStateSerializerTest,DynamicKafkaSourceSplitSerializerTest,DynamicKafkaSourceEnumeratorTest,DynamicKafkaSourceReaderTest -DskipITs -Drat.skip=true test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
