Ramkumar created KAFKA-6227:
-------------------------------
Summary: Kafka 0.11.01 New consumer - multiple consumers under
same group not working as expected
Key: KAFKA-6227
URL: https://issues.apache.org/jira/browse/KAFKA-6227
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 0.11.0.1
Reporter: Ramkumar
In Kafka 0.8 High level consumers, the consumer.id under group.id
differentiates the consumers connection and manage the rebalancing the
partitions. Our Service uses this logic and keeps the Kafka stream connection
in a cache (Concurrent Hashmap). so that consecutive http client connection
doesn’t have to make a stream connection, but takes from cache and read off the
messages. This also helps multiple consumers under same group.id can
simulatenously make connection to kafka and read off the message (load
balancing).
In Kafka 0.11.0.1, the New consumer API the design has changed. The
consumer.id properties are no more supported and the connections with zookeeper
are managed by Kafka itself. When 2 consumers instances under the same group
attempts to make a connection simulatenously , one connection waits on
consumer.poll() method until the other one drops the connection.
http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
That is at any point of only one active consumer instance is able to poll the
messages from the topic. This slightly would change the behavior of our service
that we have to restrict only one consumer connection for a group for a topic.
That is we couldn’t hold the connection in cache if multiple consumer under
same group needs to use the Kafka.
I couldn’t find any properties that aids to make multiple consumer connections
on the same group
The manual partition assignment may be a work around but this is way complex to
handle that in a service. This is complex because the service needs to track
the consumer connections and assign the partitions of the topic and do the
rebalancing (what Kafka 0.8 high level consumer does originally).
Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation
refers about performance degradation in using old apis in new versions.
Was there a solution devised how the highlevel consumer of kafka 0.8 can be
migrated with out any change to the behavior from the users perspective
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
