Kafka details: Kafka Version: kafka_2.12-2.2.0 3 broker Kafka Cluster Kafka topic configs - Partitions 60, Replication Factor 3, min-insync-replica 2 Overview of Application from Kafka scenario: 1. The Collector application recieves data from Kafka topic (60 Partitions.) 2. Application has 4 instances running on 4 different server machines. 3. An application instantance initiates multiple consumer threads. 4. This is done through Java Executor service eg: ExecutorService es = Executors.newFixedThreadPool(NoOfconsumers); 5. An auto.offset.reset: 'earliest' property is also passed to each kafka consumer threads. 6. Each single partition in the topic is assigned to each single consumer thread using the Kafka Assign Method. 7. A "seekToBeginning" method is called inside the kafka Consumer thread to consume from first offset from the Kafka Partition. 8. The Kafka consumer group-id is kept constant throughout the application instances. 9. The Kafka consumer threads are closed using KafkaConsumer.close() method once all the records from its respective partition are consumed from the topic Issue Description: 1. On some occasions, One or more Kafka consumer threads, do not fetch from first offset of the kafka partitions, ie. they start to consume from between. 2. Thus some of the data from kafka topic is missed and not consumed. 3. This pattern of in-correct consumption can be observed on any of the 4 Servers and is not constant. 4. There are also instances where the all the threads have consumed from the first offsets of their assigned partitions successfully.
Is this issue related to the current version of Kafka we are using ( kafka_2.12-2.2.0) and its corresponding bug has been fixed in later versions ? Please advise on this. Thanks & Regards Debi Chowdhuri Tata Consultancy Services Limited Mumbai,Maharashtra India Mailto: [email protected] Website: http://www.tcs.com ____________________________________________ Experience certainty. IT Services Business Solutions Consulting ____________________________________________ =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you
