> For the kafka high level consumer, if I create exactly the number of > threads as the number of partitions, is there a guarantee that each > thread will be the only thread that reads from a particular partition? > I'm following this example > <https://github.com/bingoohuang/java-sandbox/blob/92318c6d3f2533bbadb253c59a201e4e70f72ad2/src/main/java/org/n3r/sandbox/kafka/ConsumerGroupExample.java>.
The documentation for that particular piece of code (https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example) states no such guarantee. The main point for the high level consumer is automatic balancing through zookeeper and the kafka brokers. Should one of your threads encounter an error, an external source add a partition to your topic a set of packets be magically dropped, the consumer group will rebalance the partition assignments and your exactly one thread per partition system will be lost. That said, the kafka people will be better able to shed light on this, you might want to consider posting this to the kafka mailing list (http://kafka.apache.org/contact.html) rather than the storm one :) Kind regards, Mattijs
