Bowen Li created FLINK-39940:
--------------------------------
Summary: DynamicKafkaSource should refresh restored cluster
properties from KafkaMetadataService
Key: FLINK-39940
URL: https://issues.apache.org/jira/browse/FLINK-39940
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Reporter: Bowen Li
Assignee: Bowen Li
{{DynamicKafkaSource}} restores {{KafkaStream}} metadata from the checkpointed
enumerator state when a job is resumed from a checkpoint or savepoint. Before
this change, restored sub-enumerators were rebuilt from the checkpointed
cluster properties without first refreshing them from
{{{}KafkaMetadataService{}}}. That means non-bootstrap Kafka client properties
such as authentication or security settings can be stale at restore time, and
the job can fail before the next periodic metadata refresh has a chance to
reconcile the latest metadata.
The restore path should use the latest metadata-service properties immediately,
while still preserving the checkpointed {{bootstrap.servers}} for restored
clusters so the enumerator reconnects to the same restored cluster identity.
Restore-time metadata lookup failures should also fail fast instead of being
deferred until a later discovery cycle.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
