[
https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776546#comment-16776546
]
Tzu-Li (Gordon) Tai commented on FLINK-11286:
---------------------------------------------
Some thoughts regarding the discussion whether or not we want to remove the
ability to have arbitrary watermark assigners in the middle of the pipeline:
First of all, I agree that removing this will definitely simplify things. as we
can see in my above comment. Stream source operators and arbitrary downstream
watermark generating operators are essentially the two places where idleness
detection is required.
If we don't allow arbitrary watermark assigners downstream, then we essentially
only have to detect idleness in sources.
Second of all, the arguments so far for keeping the ability to have arbitrary
watermark assigners in the middle of the pipeline, is that sometimes the
timestamp can only be extracted after some parsing on the source records.
I feel like this isn't a strong argument, because this should be doable via the
deserialization schemas already.
So, I would give a +1 to disallow mid-pipeline arbitrary watermark assigners in
the future.
All watermark assigners must be provided within the source, or directly chained
after it (might need some specific APIs for adding sources to enforce that).
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many
> times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows:
> {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism
> of 10. Among them, the watermark is not sent by the source operator, but is
> downstream, such as flatmap. Every source subtask will not be idle. However,
> after the filter, some pipelines generate "idle". For example, there are 3
> pipelines that will no longer have data sent downstream. At this time, we
> can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline
> in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
