Dawid Wysakowicz created FLINK-37025:
----------------------------------------
Summary: Periodic SQL watermarks can travel back in time
Key: FLINK-37025
URL: https://issues.apache.org/jira/browse/FLINK-37025
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 2.0-preview, 1.20.0
Reporter: Dawid Wysakowicz
Fix For: 2.0.0, 1.20.1
If watermarks are generated through SQL, e.g. using `WATERMARK FOR ts AS ts` is
used
https://github.com/apache/flink/blob/e9353319ad625baa5b2c20fa709ab5b23f83c0f4/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/generated/GeneratedWatermarkGeneratorSupplier.java#L86
The code in {{onEvent}} can move the {{currentWatermark}} back in time, because
it does not check if it advanced. This is fine in case of `on-event`
watermarks`, because underlying {{WatermarkOutput}} will, but it has confusing
outcomes for `on-periodic`.
Example:
If events with timestamps:
```
{"id":1,"ts":"2024-01-01 00:00:00"}
{"id":3,"ts":"2024-01-03 00:00:00"}
{"id":2,"ts":"2024-01-02 00:00:00"}
```
come within the periodic emit interval, the generated watermark will be
"2024-01-02 00:00:00" instead of "2024-01-03 00:00:00". As a result the
watermark for "2024-01-03 00:00:00" is swallowed and a new event is required to
progress the processing of that event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
