Hi everyone, I'd like to start a discussion on FLIP-562 [1] to implement topic integrity checks on kafka connector, as currently the connector is blind to topic recreations, presenting risks to job consistency.
Kafka APIs traditionally rely on topic names, which do not guarantee uniqueness over time. Though both KIP-516 [2] and KIP-848 [3] discuss topicId based communication, client support is not on the roadmap [4]. The FLIP contains the new proposal to make flink kafka connector sensitive to topicId changes through integrity checks over the periodical metadata fetching (AKA topic partition discovery), and sets the grounds for future topicId based communication with both the user and kafka server. I'd appreciate your feedback on the proposed changes. Thanks, Efrat. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-562%3A+Topic+integrity+checks+in+Kafka+Connector [2] https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers [3] https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol [4] https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers/#KIP516:TopicIdentifiers-Clients
