[
https://issues.apache.org/jira/browse/KYLIN-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173903#comment-17173903
]
ASF GitHub Bot commented on KYLIN-4683:
---------------------------------------
hit-lacus commented on a change in pull request #1351:
URL: https://github.com/apache/kylin/pull/1351#discussion_r467601221
##########
File path:
stream-source-kafka/src/main/java/org/apache/kylin/stream/source/kafka/consumer/KafkaConnector.java
##########
@@ -81,9 +81,15 @@ public void open() {
} else if (startMode == ConsumerStartMode.LATEST) {
kafkaConsumer.seekToEnd(topicPartitions);
} else {
+ List<TopicPartition> newTopicPartitions = Lists.newArrayList();
for (TopicPartition topicPartition : topicPartitions) {
Long offset = partitionOffsets.get(topicPartition.partition());
- kafkaConsumer.seek(topicPartition, offset);
+ if (offset != null) {
+ kafkaConsumer.seek(topicPartition, offset);
+ } else {
+ newTopicPartitions.add(topicPartition);
+ }
+ kafkaConsumer.seekToBeginning(newTopicPartitions);
Review comment:
I think `kafkaConsumer.seekToBeginning(newTopicPartitions);` should not
in **for-loop** .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Fail to consume kafka when partition number get larger
> ------------------------------------------------------
>
> Key: KYLIN-4683
> URL: https://issues.apache.org/jira/browse/KYLIN-4683
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v3.0.2
> Reporter: tianhui
> Priority: Major
> Attachments: image-2020-08-05-17-20-37-270.png
>
>
> I run a testing streaming cube with kafka. At first, the topic has 3
> partitions, and the cube running smoothly. But after I alter kafka topic to 7
> partitions, all receivers stop consuming. !image-2020-08-05-17-20-37-270.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
