As I understand it, the Flink Kafka Producer may emit duplicates to Kafka topics.
How can I deduplicate these messages when reading them back with Flink (via the Flink Kafka Consumer)? For example, is there any out-the-box support for deduplicating messages, i.e. by incorporating something like "idempotent producers" as proposed by Jay Krepps (which, as I understand it, involves maintaining a "high watermark" on a message-by-message level)? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Deduplicate-messages-from-Kafka-topic-tp11051.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
