I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM,
1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition
in which messages contains the total of 3 Million messages. It took 5 days
to completely process all the messages and as soon as message got processed
i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the
topic I stopped the consumer .but after 6 hrs again it was showing the lag
of 2 million message which I found that were duplicate messages. This thing
is happening very frequently. My offsets are stored on Kafka broker itself.
My server configuration is:
Is there in the configuration that I am missing?
Any help is appreciated