[
https://issues.apache.org/jira/browse/FLINK-40503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-40503:
-----------------------------------
Description:
{{WatermarkStrategy#withIdleness}} documents: "if no records flow in a
partition of a
stream for that amount of time" the output is marked idle. The actual detection
in
{{WatermarksWithIdleness.IdlenessTimer}} anchors its countdown at the *first
quiet
periodic probe* (not at the last record) and uses a strict {{>}} comparison
({{WatermarksWithIdleness.java:140}}).
With timeout T=10ms and probes every 5ms (event at t=4):
probe t=5 observes activity and resets; t=10 is the first quiet probe and only
*starts*
the timer; t=15 elapsed 5ms; t=20 elapsed exactly 10ms fails the strict {{>}};
*idle fires at t=25 — 21ms after the last event, 2.1× the configured timeout.*
General form: idleness is declared between idleTimeout and idleTimeout + 3
probe intervals after the last event - up to two intervals for the countdown to
be anchored at the first quiet periodic probe, plus up to one more because the
strict > comparison only fires at the next probe after the timeout is exceeded.
The worst case is attained when the timeout is an exact multiple of the
auto-watermark interval, as in the example (T=10ms, probes every 5ms → idle
21ms after the last event).
This is distinct from FLINK-35886, which fixed *what time counts* toward the
timeout
(blocked/backpressured time); this issue is about *when the countdown starts
and fires*.
Proposed resolution: document the actual behavior (javadoc of {{withIdleness}}
and the
config documentation) rather than change the timing — silently changing
detection timing
has a regression track record (cf. the FLIP-471 fallout). Optionally, anchoring
the
timer at the last activity observation could be discussed separately.
was:
{{WatermarkStrategy#withIdleness}} documents: "if no records flow in a
partition of a
stream for that amount of time" the output is marked idle. The actual detection
in
{{WatermarksWithIdleness.IdlenessTimer}} anchors its countdown at the *first
quiet
periodic probe* (not at the last record) and uses a strict {{>}} comparison
({{WatermarksWithIdleness.java:140}}).
With timeout T=10ms and probes every 5ms (event at t=4):
probe t=5 observes activity and resets; t=10 is the first quiet probe and only
*starts*
the timer; t=15 elapsed 5ms; t=20 elapsed exactly 10ms fails the strict {{>}};
*idle fires at t=25 — 21ms after the last event, 2.1× the configured timeout.*
General form: last event + T + up to 2 probe intervals — one to anchor the
timer, one
extra whenever the timeout is an exact multiple of the auto-watermark interval,
which is
the common production configuration. The same overshoot exists in every
consumer of
{{IdlenessTimer}} (DataStream V1 sources, {{assignTimestampsAndWatermarks}},
DataStream V2).
This is distinct from FLINK-35886, which fixed *what time counts* toward the
timeout
(blocked/backpressured time); this issue is about *when the countdown starts
and fires*.
Proposed resolution: document the actual behavior (javadoc of {{withIdleness}}
and the
config documentation) rather than change the timing — silently changing
detection timing
has a regression track record (cf. the FLIP-471 fallout). Optionally, anchoring
the
timer at the last activity observation could be discussed separately.
> WatermarksWithIdleness effective timeout is idleTimeout plus up to 3 probe
> intervals, contradicting the withIdleness documentation
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40503
> URL: https://issues.apache.org/jira/browse/FLINK-40503
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Reporter: Martijn Visser
> Priority: Major
>
> {{WatermarkStrategy#withIdleness}} documents: "if no records flow in a
> partition of a
> stream for that amount of time" the output is marked idle. The actual
> detection in
> {{WatermarksWithIdleness.IdlenessTimer}} anchors its countdown at the *first
> quiet
> periodic probe* (not at the last record) and uses a strict {{>}} comparison
> ({{WatermarksWithIdleness.java:140}}).
> With timeout T=10ms and probes every 5ms (event at t=4):
> probe t=5 observes activity and resets; t=10 is the first quiet probe and
> only *starts*
> the timer; t=15 elapsed 5ms; t=20 elapsed exactly 10ms fails the strict {{>}};
> *idle fires at t=25 — 21ms after the last event, 2.1× the configured timeout.*
> General form: idleness is declared between idleTimeout and idleTimeout + 3
> probe intervals after the last event - up to two intervals for the countdown
> to be anchored at the first quiet periodic probe, plus up to one more because
> the strict > comparison only fires at the next probe after the timeout is
> exceeded. The worst case is attained when the timeout is an exact multiple of
> the auto-watermark interval, as in the example (T=10ms, probes every 5ms →
> idle 21ms after the last event).
> This is distinct from FLINK-35886, which fixed *what time counts* toward the
> timeout
> (blocked/backpressured time); this issue is about *when the countdown starts
> and fires*.
> Proposed resolution: document the actual behavior (javadoc of
> {{withIdleness}} and the
> config documentation) rather than change the timing — silently changing
> detection timing
> has a regression track record (cf. the FLIP-471 fallout). Optionally,
> anchoring the
> timer at the last activity observation could be discussed separately.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
