Hi Efrat, Thanks for kicking off this discussion — I’m also in favor of adding a check for the Kafka Connector.
> My only concern is that manually specifying the topic ID seems to be > a bit costly for users. Why does it require flink users to pass the topic > id > manually or explicitly? It would be helpful for users if flink source or > sink > are able to fetch topic id automatically in the beginning, right? I share Rui’s concern about requiring users to manually specify the topicId. Given that topicId-based communication isn’t part of Kafka’s current roadmap (as noted in your “Future Plan” section) and is starting available in Kafka 4.0–4.2, it seems premature—and potentially burdensome—to expose it in Flink’s user-facing API. Ideally, if needed at all, the source or sink should be able to resolve the topicId automatically during initialization. Additionally, I agree with Aleksandr that introducing “Partition Discovery” for sinks feels conceptually inconsistent with Flink’s existing design. Best, Leonard > > On Thu, Jan 15, 2026 at 12:38 PM Efrat Levitan <[email protected]> > wrote: > >> Hi everyone, I'd like to start a discussion on FLIP-562 [1] to implement >> topic integrity checks on kafka connector, as currently the connector is >> blind to topic recreations, presenting risks to job consistency. >> >> Kafka APIs traditionally rely on topic names, which do not guarantee >> uniqueness over time. >> Though both KIP-516 [2] and KIP-848 [3] discuss topicId based >> communication, client support is not on the roadmap [4]. >> >> The FLIP contains the new proposal to make flink kafka connector sensitive >> to topicId changes through integrity checks over the periodical metadata >> fetching (AKA topic partition discovery), and sets the grounds for future >> topicId based communication with both the user and kafka server. >> >> I'd appreciate your feedback on the proposed changes. >> >> Thanks, >> Efrat. >> >> [1] >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-562%3A+Topic+integrity+checks+in+Kafka+Connector >> >> [2] >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers >> >> [3] >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol >> >> [4] >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers/#KIP516:TopicIdentifiers-Clients >>
