[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?
[ https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105641#comment-17105641 ] Alexey Romanenko commented on BEAM-9760: Hi [~wongkawah], thanks for details. I see your point but I'm not sure that we can use "subscribe()" method in KafkaIO because of some internal Beam limitations. We had a similar discussion a while ago ([see here|https://issues.apache.org/jira/browse/BEAM-5786?focusedCommentId=16655883=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16655883]) > KafkaIO supports consumer group? > > > Key: BEAM-9760 > URL: https://issues.apache.org/jira/browse/BEAM-9760 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ka Wah WONG >Priority: Minor > > It seems only assign method of Kafka Consumer class is called in > org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation > of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment > through this assign method does not use the consumer's group management > functionality. > May I ask if KafkaIO will be enhanced to support consumer's group management > with using Kafka consumer's subscribe method? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?
[ https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102363#comment-17102363 ] Ka Wah WONG commented on BEAM-9760: --- Hi [~aromanenko] , my proposal applies on the case when there are multiple partitions in a topic, and multiple consumers in a consumer group that subscribe the same topic. >From my understanding, to get a dynamically assigned partition with group >management by Kafka coordinator, it needs to use KafkaConsumer's subscribe >method. Using KafkaConsumer's assign method would manually assign a partition >to the consumer and does not use the consumer's group management >functionality. (Reference: Javaodc of >org.apache.kafka.clients.consumer.KafkaConsumer). If I want to have two separate Java applications both using Apache Beam subscribing to the same Kafka topic through KafkaIO with same consumer group defined, I would like there can be failover feature supported. For example, when topic-partition-0 is assigned to App-0, and topic-partition-1 is assigned to App-1, if App-0 is down, the topic-partition-0 is then assigned to the App-1 by the Kafka coordinator / broker. Then App-1 subscribes message from both of topic-partition-0 and topic-partition-1 when App-0 is down. Note that App-0 and App-1 defines same consumer group and subscribes same topic with the 2 partitions. > KafkaIO supports consumer group? > > > Key: BEAM-9760 > URL: https://issues.apache.org/jira/browse/BEAM-9760 > Project: Beam > Issue Type: Improvement > Components: io-java-kafka >Reporter: Ka Wah WONG >Priority: Minor > > It seems only assign method of Kafka Consumer class is called in > org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation > of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment > through this assign method does not use the consumer's group management > functionality. > May I ask if KafkaIO will be enhanced to support consumer's group management > with using Kafka consumer's subscribe method? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9760) KafkaIO supports consumer group?
[
https://issues.apache.org/jira/browse/BEAM-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091682#comment-17091682
]
Alexey Romanenko commented on BEAM-9760:
Could you elaborate more about your proposal?
For now, consumer group could be assigned by setting a consumer config option
with \{{.withConsumerConfigUpdates(ImmutableMap.of("group.id",
"my_beam_app_1"))}} method.
> KafkaIO supports consumer group?
>
>
> Key: BEAM-9760
> URL: https://issues.apache.org/jira/browse/BEAM-9760
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
>Reporter: Ka Wah WONG
>Priority: Minor
>
> It seems only assign method of Kafka Consumer class is called in
> org.apache.beam.sdk.io.kafka.ConsumerSpEL class. According to documentation
> of org.apache.kafka.clients.consumer.KafkaConsumer, manual topic assignment
> through this assign method does not use the consumer's group management
> functionality.
> May I ask if KafkaIO will be enhanced to support consumer's group management
> with using Kafka consumer's subscribe method?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
