Hi there, >From 0.10 forward, the consumer supports a configuration option "max.poll.records" to limit the number of records returned from each call to poll(). Note also that if you use commitSync or commitAsync (with no arguments), the consumer assumes that you intend to commit offsets for all records returned from the previous poll. If that is not the case, you can also pass the exact offsets you need to commit using the overloaded versions of those APIs.
Thanks, Jason On Tue, Jun 21, 2016 at 5:31 AM, Asier Aranbarri Beldarrain < aranbar...@gmail.com> wrote: > Hi everyone! I'm new to this mailing list; Hope I can find solutions and > who knows, maybe some day help other Kafka developers. > > I'm facing an issue regarding the new (>=0.9) KafkaConsumer and its poll() > method: > > What would be the correct way to specify a maximum amount of records to be > processed without consuming more messages than needed? > > For example, I need to consume 10 and just 10 messages from a Kafka topic. > I start the consumer, and the poll method gets 15 messages. I process 10 of > them and then manually commit the offset. > > BUT: > > If I start the consumer again, it will start consuming from the 16th > message, not the 11th. This is a huge problem for me, as I need a reader > that consumes an specific number of messages. > > In Kafka 0.8, this was easy since you could use the Stream Iterator to > process the messages and stop in the exact moment you need to. But polling > isn't exact, so the consumer could consume a higher amount of records that > I need to process. > > Any way to solve this? Any help would be kindly appreciated. > > Thanks in advance! >
