Aah I didn't read long enough. I just read the subscribing to specific partitions, managing your own offsets and the controlling the consumer's position sections.
So a follow up question - when I am subscribing to specific partitions and managing my own offsets and controlling the consumer's position what does the consumer group do? Do I still have to specify it all the time? Thanks, Rajiv On Tue, Oct 13, 2015 at 1:14 PM, Rajiv Kurian <ra...@signalfx.com> wrote: > I was reading the documentation for the new Kafka consumer API at > https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java > and came across this: > > "Each Kafka consumer must specify a consumer group that it belongs to." > > Currently we use Kafka but WITHOUT the consumer group functionality since > I do the partition distribution based on my own app logic. I also persist > my own offsets in an external store. > I also sometimes need to consume from the beginning of a partition or just > some arbitrary offset. Currently I use the SimpleConsumer to do this. I > just give it the topics and partitions I want to consume from and I > maintain the partition -> broker mapping myself. I also maintain the offset > myself and when I need to consume from the beginning of a partition I ask > the broker for the earliest offset and start consuming from there. > > I am glad that the new consumer does the broker -> partition mapping > itself and I can get rid of a lot of code if I transitioned to it. But > given that the documentation suggests that I have to use the Consumer > group, it doesn't seem that it will support the lower level features that I > need. I do see the commitSync method which suggests that I am in control of > the offset management and for my use case I could just never call it. Is > there also a way for me to specify which partition to consume from exactly > (in addition to the topic) and also from which offset? > > Thanks, > Rajiv >
