[
https://issues.apache.org/jira/browse/FLINK-40235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Hung updated FLINK-40235:
------------------------------
Description:
*Problem Summary*
In an unbounded FLIP-27 source, if a subtask finishes all of its assigned
splits while the job keeps running, it's left with zero active splits but is
not finished. Its watermark freezes at the last emitted value and holds back
the global (operator-min) watermark indefinitely.
*Reproduction*
- Unbounded Kinesis (or Kafka w/ rebalancing) source, `withIdleness(...)` set.
- Parallelism high enough that a subtask can reach zero active splits after a
reshard.
- Observe: operator `currentOutputWatermark` stalls; the affected subtask
reports a frozen watermark while others advance.
*Note*
- withIdleness() does not help: idleness is tracked per-split (each split has
its own timer, advanced only while the split is live), so with zero splits
there is no generator left to ever mark the subtask idle.
- FLIP-27 has no reader/enumerator-level equivalent of the legacy
SourceContext.markAsTemporarilyIdle(), so there is currently no way to signal
that such a subtask should stop holding back the watermark. Reintroducing that
capability may be one direction for a fix.
was:
*Problem Summary*
In an unbounded FLIP-27 source, if a subtask finishes all of its assigned
splits while the job keeps running, it's left with zero active splits but is
not finished. Its watermark freezes at the last emitted value and holds back
the global (operator-min) watermark indefinitely.
*Reproduction*
- Unbounded Kinesis (or Kafka w/ rebalancing) source, `withIdleness(...)` set.
- Parallelism high enough that a subtask can reach zero active splits after a
reshard.
- Observe: operator `currentOutputWatermark` stalls; the affected subtask
reports a frozen watermark while others advance.
*Note*
- `withIdleness()` does not help: idleness is tracked per-split (each split has
its own timer, advanced only while the split is live), so with zero splits
there is no generator left to ever mark the subtask idle.
- FLIP-27 has no reader/enumerator-level equivalent of the legacy
SourceContext.markAsTemporarilyIdle(), so there is currently no way to signal
that such a subtask should stop holding back the watermark. Reintroducing that
capability may be one direction for a fix.
> FLIP-27 source subtask stalls the watermark after all its splits finish
> ------------------------------------------------------------------------
>
> Key: FLINK-40235
> URL: https://issues.apache.org/jira/browse/FLINK-40235
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / Common
> Affects Versions: 1.19.3, 2.1.0, 1.20.3
> Reporter: Jing Hung
> Priority: Major
>
> *Problem Summary*
> In an unbounded FLIP-27 source, if a subtask finishes all of its assigned
> splits while the job keeps running, it's left with zero active splits but is
> not finished. Its watermark freezes at the last emitted value and holds back
> the global (operator-min) watermark indefinitely.
> *Reproduction*
> - Unbounded Kinesis (or Kafka w/ rebalancing) source, `withIdleness(...)` set.
> - Parallelism high enough that a subtask can reach zero active splits after a
> reshard.
> - Observe: operator `currentOutputWatermark` stalls; the affected subtask
> reports a frozen watermark while others advance.
> *Note*
> - withIdleness() does not help: idleness is tracked per-split (each split has
> its own timer, advanced only while the split is live), so with zero splits
> there is no generator left to ever mark the subtask idle.
> - FLIP-27 has no reader/enumerator-level equivalent of the legacy
> SourceContext.markAsTemporarilyIdle(), so there is currently no way to signal
> that such a subtask should stop holding back the watermark. Reintroducing
> that capability may be one direction for a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
