This is a good article on LinkedIn site - I think it's a good item to read
before hitting complicated designs
https://www.linkedin.com/pulse/exactly-once-delivery-message-distributed-system-arun-dhwaj/
On 29 May 2018 at 14:34, Thakrar, Jayesh
wrote:
> For more details, see
For more details, see https://www.slideshare.net/JayeshThakrar/kafka-68540012
While this is based on Kafka 0.9, the fundamental concepts and reasons are
still valid.
On 5/28/18, 12:20 PM, "Hans Jespersen" wrote:
Are you seeing 1) duplicate messages stored in a Kafka topic partition or
Are you seeing 1) duplicate messages stored in a Kafka topic partition or 2)
duplicate consumption and processing of a single message stored in a Kafka
topic?
If it’s #1 then you can turn on the idempotent producer feature to get Exactly
Once Semantics (EOS) while publishing.
If it’s #2 then
Duplication can happen if your producer or consumer are exiting uncleanly.
Like if producer just crashes before it receives ack from broker your logic
will fail to register that message got produced. And when it comes back up
it will try to send that batch again. Same with consumer, if it crashes
Hi,
Facing Duplication inconsistently while bouncing Kafka producer and
consumer in tomcat node. any help will be appreciated to find out the root
cause.
--
With Regards,
Karthick.K
