[ https://issues.apache.org/jira/browse/FLINK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16389426#comment-16389426 ]
Tzu-Li (Gordon) Tai commented on FLINK-5479: -------------------------------------------- After some discussion with [~aljoscha] offline, we decided to not touch the Kafka connector code any more for Flink 1.5. Let's move this to 1.6.0, where we are planning a new implementation for the Kafka / Kinesis connectors. This feature can be implemented commonly for these connectors. > Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions > ------------------------------------------------------------------------------ > > Key: FLINK-5479 > URL: https://issues.apache.org/jira/browse/FLINK-5479 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector > Reporter: Tzu-Li (Gordon) Tai > Priority: Blocker > Fix For: 1.6.0 > > > Reported in ML: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-topic-partition-skewness-causes-watermark-not-being-emitted-td11008.html > Similar to what's happening to idle sources blocking watermark progression in > downstream operators (see FLINK-5017), the per-partition watermark mechanism > in {{FlinkKafkaConsumer}} is also being blocked of progressing watermarks > when a partition is idle. The watermark of idle partitions is always > {{Long.MIN_VALUE}}, therefore the overall min watermark across all partitions > of a consumer subtask will never proceed. > It's normally not a common case to have Kafka partitions not producing any > data, but it'll probably be good to handle this as well. I think we should > have a localized solution similar to FLINK-5017 for the per-partition > watermarks in {{AbstractFetcher}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)