[
https://issues.apache.org/jira/browse/FLINK-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096465#comment-17096465
]
Akshay Aggarwal commented on FLINK-11186:
-----------------------------------------
The other thread seems to be much more active, marking this ticket as duplicate.
> Event-time balancing for multiple Kafka partitions
> --------------------------------------------------
>
> Key: FLINK-11186
> URL: https://issues.apache.org/jira/browse/FLINK-11186
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream, Connectors / Kafka
> Reporter: Tom Schamberger
> Priority: Major
>
> Currently, it is not possible with Flink to back-pressure individual Kafka
> partitions, which are faster in terms of event-time. This leads to
> unnecessary memory consumption and can lead to deadlocks in the case of
> back-pressure.
> When multiple Kafka topics are consumed, succeeding event-time window
> operators have to wait until the last Kafka partition has produced a
> sufficient watermark to be triggered. If individual Kafka partitions differ
> in read performance or the event-time of messages within partitions is not
> monotonically distributed, this can lead to a situation, where 'fast'
> partitions (event-time makes fast progress) outperform slower partitions
> until back-pressuring prevents all partitions from being further consumed.
> This leads to a deadlock of the application.
> I suggest, that windows should be able to back-pressure individual
> partitions, which progress faster in terms of event-time, so that slow
> partitions can keep up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
