[
https://issues.apache.org/jira/browse/FLINK-13798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911428#comment-16911428
]
Stephan Ewen commented on FLINK-13798:
--------------------------------------
FYI: I believe one outcome of FLIP-27 will be to drop the
{{TimestampsAndPeriodicWatermarksOperator}} and have Watermark Generation
strictly in the source operator.
The reason is that this allows for transparent "push down" of watermark
generation to the source partition level (like Kafka topic partition) and we
should avoid having two slightly different ways to generate watermarks.
> Refactor the process of checking stream status while emitting watermark in
> source
> ---------------------------------------------------------------------------------
>
> Key: FLINK-13798
> URL: https://issues.apache.org/jira/browse/FLINK-13798
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Task
> Reporter: zhijiang
> Assignee: zhijiang
> Priority: Minor
>
> As we know, the watermark could be emitted to downstream only when the stream
> status is active. For the downstream task we already have the component of
> StatusWatermarkValve in StreamInputProcessor to handle this logic. But for
> the source task the current implementation of this logic seems a bit tricky.
> There are two scenarios for the source case:
> * Emit watermark via source context: In the specific WatermarkContext, it
> would toggle the stream status as active before collecting/emitting
> records/watermarks. Then in the implementation of RecordWriterOutput, it
> would check the status always active before really emitting watermark.
> * TimestampsAndPeriodicWatermarksOperator: The watermark is triggered by
> timer in interval time. When it happens, it would call output stack to emit
> watermark. Then the RecordWriterOutput could take the role of checking status
> before really emitting watermark.
> So we can see that the checking status logic in RecordWriterOutput only works
> for above second scenario, and this logic seems redundant for the first
> scenario because WatermarkContext always toggle active status before
> emitting. Even worse, the logic is RecordWriterOutput would bring cycle
> dependency with StreamStatusMaintainer, which is a blocker for the following
> work of integrating source processing on runtime side.
> The solution is that we could migrate the checking logic from
> RecordWriterOutput to TimestampsAndPeriodicWatermarksOperator.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
