Repository: flink
Updated Branches:
  refs/heads/master 27be32e8a -> b005ea353


[FLINK-9183] [docs] Add warning about idle partitions to Kafka connector docs.

This closes #5858.


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/b005ea35
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/b005ea35
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/b005ea35

Branch: refs/heads/master
Commit: b005ea35374f619298cebab649e0ba477aeaf860
Parents: afad30a
Author: juhoautio <autioj...@gmail.com>
Authored: Mon Apr 16 17:40:23 2018 +0300
Committer: Fabian Hueske <fhue...@apache.org>
Committed: Mon Apr 16 21:40:57 2018 +0200

----------------------------------------------------------------------
 docs/dev/connectors/kafka.md | 5 +++++
 1 file changed, 5 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/b005ea35/docs/dev/connectors/kafka.md
----------------------------------------------------------------------
diff --git a/docs/dev/connectors/kafka.md b/docs/dev/connectors/kafka.md
index 27fca7a..47a6651 100644
--- a/docs/dev/connectors/kafka.md
+++ b/docs/dev/connectors/kafka.md
@@ -451,6 +451,11 @@ the `Watermark getCurrentWatermark()` (for periodic) or the
 `Watermark checkAndGetNextWatermark(T lastElement, long extractedTimestamp)` 
(for punctuated) is called to determine
 if a new watermark should be emitted and with which timestamp.
 
+**Note**: If a watermark assigner depends on records read from Kafka to 
advance its watermarks (which is commonly the case), all topics and partitions 
need to have a continuous stream of records. Otherwise, the watermarks of the 
whole application cannot advance and all time-based operations, such as time 
windows or functions with timers, cannot make progress. A single idle Kafka 
partition causes this behavior. 
+A Flink improvement is planned to prevent this from happening 
+(see [FLINK-5479: Per-partition watermarks in FlinkKafkaConsumer should 
consider idle partitions](
+https://issues.apache.org/jira/browse/FLINK-5479)).
+In the meanwhile, a possible workaround is to send *heartbeat messages* to all 
consumed partitions that advance the watermarks of idle partitions.
 
 ## Kafka Producer
 

Reply via email to