Thanks Efrat for driving this valuable FLIP since it avoids the risks associated with a topic being recreated.
My only concern is that manually specifying the topic ID seems to be a bit costly for users. Why does it require flink users to pass the topic id manually or explicitly? It would be helpful for users if flink source or sink are able to fetch topic id automatically in the beginning, right? If so, then an option like topic-integrity-check.enabled would suffice. Please correct me if I have misunderstood, thanks Best, Rui On Thu, Jan 15, 2026 at 12:38 PM Efrat Levitan <[email protected]> wrote: > Hi everyone, I'd like to start a discussion on FLIP-562 [1] to implement > topic integrity checks on kafka connector, as currently the connector is > blind to topic recreations, presenting risks to job consistency. > > Kafka APIs traditionally rely on topic names, which do not guarantee > uniqueness over time. > Though both KIP-516 [2] and KIP-848 [3] discuss topicId based > communication, client support is not on the roadmap [4]. > > The FLIP contains the new proposal to make flink kafka connector sensitive > to topicId changes through integrity checks over the periodical metadata > fetching (AKA topic partition discovery), and sets the grounds for future > topicId based communication with both the user and kafka server. > > I'd appreciate your feedback on the proposed changes. > > Thanks, > Efrat. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-562%3A+Topic+integrity+checks+in+Kafka+Connector > > [2] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers > > [3] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol > > [4] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers/#KIP516:TopicIdentifiers-Clients >
