[
https://issues.apache.org/jira/browse/FLINK-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542855#comment-15542855
]
Tzu-Li (Gordon) Tai edited comment on FLINK-4722 at 10/3/16 4:58 PM:
---------------------------------------------------------------------
I'm not sure if I fully understand your use case, but you could consider this
solution:
Use a single FlinkKafkaConsumer to read the topic, and key the input elements
by the partition id (you can do this by supplying your own
{{KeyedDeserializationSchema}} to the consumer; the deserialization schema
exposes the partition of the read record, so you can use that to key the data,
ex. output {{Tuple(partitionId,matrixData)}} or your own POJO / case class
directly from the FlinkKafkaConsumer09).
Then, on the keyed stream, you can perform windows on each key (i.e., in your
case, each matrix) like you mentioned.
Does this help?
was (Author: tzulitai):
I'm not sure if I fully understand your use case, but you could consider this
solution:
Use a single FlinkKafkaConsumer to read the topic, and key the input elements
by the partition id (you can do this by supplying your own
{{KeyedDeserializationSchema}} to the consumer; the deserialization schema
exposes the partition of the read record, so you can use that to key the data,
ex. output {{Tuple(partitionId,matrixData)}} or your own POJO / case class
directly from the source).
Then, on the keyed stream, you can perform windows on each key (i.e., in your
case, each matrix) like you mentioned.
Does this help?
> Consumer group concept not working properly with FlinkKafkaConsumer09
> -----------------------------------------------------------------------
>
> Key: FLINK-4722
> URL: https://issues.apache.org/jira/browse/FLINK-4722
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.1.2
> Reporter: Sudhanshu Sekhar Lenka
>
> When Kafka one Topic has 3 partition and 3 FlinkKafkaConsumer09 connected to
> that same topic using "group.id" ,"myGroup" property . Still flink consumer
> get all data which are push to each 3 partition . While it work properly
> with normal java consumer. each consumer get specific data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
