Re: offsetsForTimes API performance

2018-01-24 Thread srimugunthan dhandapani
p > could be avoided if the broker could just auto-assign the consumer to the > offset for a timestamp. > > > On Mon, Jan 22, 2018 at 9:59 AM, srimugunthan dhandapani < > srimugunthan.dhandap...@gmail.com> wrote: > > > Hi all, > > > > We use kafka a

offsetsForTimes API performance

2018-01-22 Thread srimugunthan dhandapani
Hi all, We use kafka as our store and every one of our record is associated with a timeStamp. We pull data from kafka by seeking to a timeStamp offset everytime and then get the records by polling. We use KafkaConsumer's offsetsForTimes ( https://kafka.apache.org/0110/javadoc/org/apache/kafka/cl

timeout value for consumer.poll

2017-10-04 Thread srimugunthan dhandapani
Hi all, In our implementation, instead of reading from kafka's automatic offsets everytime, we rely on the offsetsForTimes to get the offset for our interested timestamp and seek to it before we do our consumer.poll() For quite some time, we could comfortably use 100ms for consumer.poll timeout.

fetchOffsetByTime: reading from multiple partitions

2017-09-30 Thread srimugunthan dhandapani
Hi all, I am using kafka 0.10.2 version and I have a kafka cluster with topics configured for 5 partitions. I am trying to use with Kafka consumers assigned manually to multiple partitions. I see that in the consumer poll loop , i am getting records from multiple partitions, but the number of reco