[
https://issues.apache.org/jira/browse/FLINK-40235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18099035#comment-18099035
]
Weiqing Yang commented on FLINK-40235:
--------------------------------------
Hi Jing Hung,
This issue seems similar to FLINK-38477:
https://issues.apache.org/jira/browse/FLINK-38477
Could you please take a look when you get a chance? I proposed a PR for it
earlier, but was suggested to come up with a FLIP instead.
Thanks,
Weiqing
> FLIP-27 source subtask stalls the watermark after all its splits finish
> ------------------------------------------------------------------------
>
> Key: FLINK-40235
> URL: https://issues.apache.org/jira/browse/FLINK-40235
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common
> Affects Versions: 1.19.3, 2.1.0, 1.20.3
> Reporter: Jing Hung
> Priority: Major
>
> *Problem Summary*
> In an unbounded FLIP-27 source, if a subtask finishes all of its assigned
> splits while the job keeps running, it's left with zero active splits but is
> not finished. Its watermark freezes at the last emitted value and holds back
> the global (operator-min) watermark indefinitely.
> *Reproduction*
> - Unbounded Kinesis (or Kafka w/ rebalancing) source, `withIdleness(...)` set.
> - Parallelism high enough that a subtask can reach zero active splits after a
> reshard.
> - Observe: operator `currentOutputWatermark` stalls; the affected subtask
> reports a frozen watermark while others advance.
> *Note*
> - withIdleness() does not help: idleness is tracked per-split (each split has
> its own timer, advanced only while the split is live), so with zero splits
> there is no generator left to ever mark the subtask idle.
> - FLIP-27 has no reader/enumerator-level equivalent of the legacy
> SourceContext.markAsTemporarilyIdle(), so there is currently no way to signal
> that such a subtask should stop holding back the watermark. Reintroducing
> that capability may be one direction for a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
