I was reading the documentation for the new Kafka consumer API at https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java and came across this:
"Each Kafka consumer must specify a consumer group that it belongs to." Currently we use Kafka but WITHOUT the consumer group functionality since I do the partition distribution based on my own app logic. I also persist my own offsets in an external store. I also sometimes need to consume from the beginning of a partition or just some arbitrary offset. Currently I use the SimpleConsumer to do this. I just give it the topics and partitions I want to consume from and I maintain the partition -> broker mapping myself. I also maintain the offset myself and when I need to consume from the beginning of a partition I ask the broker for the earliest offset and start consuming from there. I am glad that the new consumer does the broker -> partition mapping itself and I can get rid of a lot of code if I transitioned to it. But given that the documentation suggests that I have to use the Consumer group, it doesn't seem that it will support the lower level features that I need. I do see the commitSync method which suggests that I am in control of the offset management and for my use case I could just never call it. Is there also a way for me to specify which partition to consume from exactly (in addition to the topic) and also from which offset? Thanks, Rajiv
