[
https://issues.apache.org/jira/browse/FLINK-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203996#comment-15203996
]
ASF GitHub Bot commented on FLINK-3428:
---------------------------------------
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/1764#discussion_r56802608
--- Diff: docs/apis/streaming/event_timestamps_watermarks.md ---
@@ -308,6 +269,72 @@ class TimeLagWatermarkGenerator extends
AssignerWithPeriodicWatermarks[MyEvent]
</div>
+#### **With Ascending timestamps**
+
+The simplest special case for periodic watermark generation is the case
where timestamps within one source occur in ascending order.
+In that case, the current timestamp can always act as a watermark, because
no lower timestamps will occur any more.
+
+Note that it is only necessary that timestamps are ascending *per parallel
data source instance*. For example, if
+in a specific setup one Kafka partition is read by one parallel data
source instance, then it is only necessary that
+timestamps are ascending within each Kafka partition. Flink's Watermark
merging mechanism will generate correct
+watermarks whenever parallel streams are shuffled, unioned, connected, or
merged.
+
+<div class="codetabs" markdown="1">
+<div data-lang="java" markdown="1">
+{% highlight java %}
+DataStream<MyEvent> stream = ...
+
+DataStream<MyEvent> withTimestampsAndWatermarks =
+ stream.assignTimestampsAndWatermarks(new
AscendingTimestampExtractor<MyEvent>() {
+
+ @Override
+ public long extractAscendingTimestamp(MyEvent element) {
+ return element.getCreationTime();
+ }
+});
+{% endhighlight %}
+</div>
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+val stream: DataStream[MyEvent] = ...
+
+val withTimestampsAndWatermarks = stream.assignAscendingTimestamps(
_.getCreationTime )
+{% endhighlight %}
+</div>
+</div>
+
+#### **With Watermark of Fixed Allowed Lateness**
+
+Another example of periodic watermark generation is the one where the
watermark lags behind the maximum (event-time) timestamp
+seen in the stream, by a fixed amount of time. This case covers scenarios
where the maximum lateness that can be encountered in a
+stream is known in advance, e.g. when creating custom sources containing
elements with timestamps spread within a fixed period of
+time for testing. For these cases, Flink provides the
`FixedAllowedLatenessWatermarkEmitter` which takes as an argument
+the `maxAllowedLateness`, i.e. the maximum amount of time an element is
allowed to be late, before being dropped from the stream.
--- End diff --
Late elements are not dropped from the stream! (In #1819 this is also
mentioned multiple times).
Its just that late elements won't make it into their time-window anymore
(or anything else an operator implements with watermarks).
> Add fixed time trailing timestamp/watermark extractor
> -----------------------------------------------------
>
> Key: FLINK-3428
> URL: https://issues.apache.org/jira/browse/FLINK-3428
> Project: Flink
> Issue Type: Improvement
> Reporter: Robert Metzger
> Assignee: Kostas Kloudas
>
> Flink currently provides only one build-in timestamp extractor, which assumes
> strictly ascending timestamps. In real world use cases, timestamps are almost
> never strictly ascending.
> Therefore, I propose to provide an utility watermark extractor which is
> generating watermarks with a fixed-time trailing.
> The implementation should keep track of the highest event-time seen so far
> and subtract a fixed amount of time from that event time.
> This way, users can for example specify that the watermarks should always
> "lag behind" 10 minutes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
