[FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.
This closes #5858. Project: http://git-wip-us.apache.org/repos/asf/flink/repo Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/fc58f987 Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/fc58f987 Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/fc58f987 Branch: refs/heads/release-1.5 Commit: fc58f987144a307e7208de0bdfe439922bed4b55 Parents: 4c3d018 Author: juhoautio <autioj...@gmail.com> Authored: Mon Apr 16 17:40:23 2018 +0300 Committer: Fabian Hueske <fhue...@apache.org> Committed: Mon Apr 16 21:42:21 2018 +0200 ---------------------------------------------------------------------- docs/dev/connectors/kafka.md | 5 +++++ 1 file changed, 5 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/flink/blob/fc58f987/docs/dev/connectors/kafka.md ---------------------------------------------------------------------- diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md index 27fca7a..47a6651 100644 --- a/docs/dev/connectors/kafka.md +++ b/docs/dev/connectors/kafka.md @@ -451,6 +451,11 @@ the `Watermark getCurrentWatermark()` (for periodic) or the `Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp)` (for punctuated) is called to determine if a new watermark should be emitted and with which timestamp. +**Note**: If a watermark assigner depends on records read from Kafka to advance its watermarks (which is commonly the case), all topics and partitions need to have a continuous stream of records. Otherwise, the watermarks of the whole application cannot advance and all time-based operations, such as time windows or functions with timers, cannot make progress. A single idle Kafka partition causes this behavior. +A Flink improvement is planned to prevent this from happening +(see [FLINK-5479: Per-partition watermarks in FlinkKafkaConsumer should consider idle partitions]( +https://issues.apache.org/jira/browse/FLINK-5479)). +In the meanwhile, a possible workaround is to send *heartbeat messages* to all consumed partitions that advance the watermarks of idle partitions. ## Kafka Producer
