[
https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102363#comment-17102363
]
Ka Wah WONG commented on BEAM-9760:
-----------------------------------
Hi [~aromanenko] , my proposal applies on the case when there are multiple
partitions in a topic, and multiple consumers in a consumer group that
subscribe the same topic.
>From my understanding, to get a dynamically assigned partition with group
>management by Kafka coordinator, it needs to use KafkaConsumer's subscribe
>method. Using KafkaConsumer's assign method would manually assign a partition
>to the consumer and does not use the consumer's group management
>functionality. (Reference: Javaodc of
>org.apache.kafka.clients.consumer.KafkaConsumer).
If I want to have two separate Java applications both using Apache Beam
subscribing to the same Kafka topic through KafkaIO with same consumer group
defined, I would like there can be failover feature supported.
For example, when topic-partition-0 is assigned to App-0, and topic-partition-1
is assigned to App-1, if App-0 is down, the topic-partition-0 is then assigned
to the App-1 by the Kafka coordinator / broker. Then App-1 subscribes message
from both of topic-partition-0 and topic-partition-1 when App-0 is down. Note
that App-0 and App-1 defines same consumer group and subscribes same topic with
the 2 partitions.
> KafkaIO supports consumer group?
> --------------------------------
>
> Key: BEAM-9760
> URL: https://issues.apache.org/jira/browse/BEAM-9760
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Ka Wah WONG
> Priority: Minor
>
> It seems only assign method of Kafka Consumer class is called in
> org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation
> of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment
> through this assign method does not use the consumer's group management
> functionality.
> May I ask if KafkaIO will be enhanced to support consumer's group management
> with using Kafka consumer's subscribe method?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
