Hi Efrat, Great work on FLIP. This is definitely a much-needed improvement to prevent silent failures!
I have a few questions: 1. +1 to Rui Fan's question regarding the manual specification of Topic IDs. The FLIP mentions "Coded into the JAR, topicId is ensured to be persistent". However, since Topic IDs are generated by the broker, they differ across environments (Dev/Staging/Prod). 2. On the Sink side, the proposal uses "setPartitionDiscoveryIntervalMs". Since "Partition Discovery" in Flink implies Source behavior (finding splits), and the Sink is doing a metadata update, have you considered "setMetadataRefreshIntervalMs"? Maybe that aligns better with the underlying Kafka Client terminology (Metadata Fetch/Lookup/Refresh). 3. Is it correct that enabling partition discovery is strictly required for the integrity check? 4. The disclaimer mentions an "inevitable short period of time" where the job reads/writes to the new topic before detection. If a checkpoint completes successfully during this window, do we risk to have a corrupted state? Is this a known limitation? 5. Will there be metrics exposed for monitoring the integrity checks? Could you please clarify? Thanks! Aleksandr Savonin On Mon, 19 Jan 2026 at 08:51, Rui Fan <[email protected]> wrote: > Thanks Efrat for driving this valuable FLIP since it avoids the risks > associated with a topic being recreated. > > My only concern is that manually specifying the topic ID seems to be > a bit costly for users. Why does it require flink users to pass the topic > id > manually or explicitly? It would be helpful for users if flink source or > sink > are able to fetch topic id automatically in the beginning, right? > > If so, then an option like topic-integrity-check.enabled would suffice. > > Please correct me if I have misunderstood, thanks > > Best, > Rui > > On Thu, Jan 15, 2026 at 12:38 PM Efrat Levitan <[email protected]> > wrote: > > > Hi everyone, I'd like to start a discussion on FLIP-562 [1] to implement > > topic integrity checks on kafka connector, as currently the connector is > > blind to topic recreations, presenting risks to job consistency. > > > > Kafka APIs traditionally rely on topic names, which do not guarantee > > uniqueness over time. > > Though both KIP-516 [2] and KIP-848 [3] discuss topicId based > > communication, client support is not on the roadmap [4]. > > > > The FLIP contains the new proposal to make flink kafka connector > sensitive > > to topicId changes through integrity checks over the periodical metadata > > fetching (AKA topic partition discovery), and sets the grounds for future > > topicId based communication with both the user and kafka server. > > > > I'd appreciate your feedback on the proposed changes. > > > > Thanks, > > Efrat. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-562%3A+Topic+integrity+checks+in+Kafka+Connector > > > > [2] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers > > > > [3] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol > > > > [4] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers/#KIP516:TopicIdentifiers-Clients > > >
