Hi all, In our implementation, instead of reading from kafka's automatic offsets everytime, we rely on the offsetsForTimes to get the offset for our interested timestamp and seek to it before we do our consumer.poll()
For quite some time, we could comfortably use 100ms for consumer.poll timeout. But recently we observe that suddely we require higher timeout values. and we have to poll 5 or 6 times with consumer.poll(1000) for the API to return data. My questions: 1. how to know what is the ideal timeout value that can be given to the consumer.poll API? 2. what can cause the sudden spike in the timeouts required ? (If the ingestion rates are too high in kafka, does that affect the required consumer poll timeout configuration?) 3. when we have already seeked to the offset where we are sure the data is present, shouldnt the consumer.poll API return data immediately. Why is that we are required to poll 5 or 6 times before we get data? thanks in advance for reply, -mugunthan
