Hi, You’re correct that the FlinkKafkaProducer may emit duplicates to Kafka topics, as it currently only provides at-least-once guarantees. Note that this isn’t a restriction only in the FlinkKafkaProducer, but a general restriction for Kafka's message delivery. This can definitely be improved to exactly-once (no duplicates produced into topics) once Kafka supports transactional messaging.
On the consumer side, the FlinkKafkaConsumer doesn’t have built-in support to dedupe the messages read from topics. On the other hand this isn’t really feasible, as consumers could basically only view messages with different offsets as separate independent messages, unless identified by some user application-level logic. So in the end, we’ll need to rely on the assumption that messages produced into Kafka topics are not duplicated, which as explained above, will hopefully be available in the near future. Cheers, Gordon On January 14, 2017 at 6:12:29 PM, ljwagerfield (lawre...@dmz.wagerfield.com) wrote: As I understand it, the Flink Kafka Producer may emit duplicates to Kafka topics. How can I deduplicate these messages when reading them back with Flink (via the Flink Kafka Consumer)? For example, is there any out-the-box support for deduplicating messages, i.e. by incorporating something like "idempotent producers" as proposed by Jay Krepps (which, as I understand it, involves maintaining a "high watermark" on a message-by-message level)? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Deduplicate-messages-from-Kafka-topic-tp11051.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
