Hi everyone! I'm new to this mailing list; Hope I can find solutions and who knows, maybe some day help other Kafka developers.
I'm facing an issue regarding the new (>=0.9) KafkaConsumer and its poll() method: What would be the correct way to specify a maximum amount of records to be processed without consuming more messages than needed? For example, I need to consume 10 and just 10 messages from a Kafka topic. I start the consumer, and the poll method gets 15 messages. I process 10 of them and then manually commit the offset. BUT: If I start the consumer again, it will start consuming from the 16th message, not the 11th. This is a huge problem for me, as I need a reader that consumes an specific number of messages. In Kafka 0.8, this was easy since you could use the Stream Iterator to process the messages and stop in the exact moment you need to. But polling isn't exact, so the consumer could consume a higher amount of records that I need to process. Any way to solve this? Any help would be kindly appreciated. Thanks in advance!
