fengli commented on a change in pull request #7726: [FLINK-10342] Filter
restored partitions with discovered partitions b…
URL: https://github.com/apache/flink/pull/7726#discussion_r260052216
##########
File path:
flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumerBase.java
##########
@@ -494,6 +514,12 @@ public void open(Configuration configuration) throws
Exception {
}
}
+ if (filterRestoredPartitionsWithDiscovered) {
+
subscribedPartitionsToStartOffsets.entrySet().removeIf(
+ entry ->
(!allPartitions.contains(entry.getKey()))
Review comment:
Thanks for the review @tzulitai , Appreciate it.
1.
- You are talking about the topicDescriptor will be different from what we
will discovered with Kafka? TopicDescriptor is user inputs, can be invalid
topics. I think it might be better to rely on what Kafka tells us (And it's
KafkaConsumerBase, relies heavily on Kafka).
- If we rely on topicDescriptor, we need to check against fixedtopics and
pattertopics. Will probably add a bit more complexity here.
What do you think? But I'm OK to use topicDescriptor if you think it's a
good direction to go.
2. Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
