[
https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776585#comment-16776585
]
Tzu-Li (Gordon) Tai edited comment on FLINK-11286 at 2/25/19 7:33 AM:
----------------------------------------------------------------------
[~yanghua]
Ok. I think I was misguided by this sentence you mentioned:
{quote}
I think that assigning a watermark in a non-source operator is a reasonable
requirement, which can reduce the complexity of the source logic
{quote}
because this implies that the proposal is to remove the idleness detection from
{{StreamSource}}, and instead do it at the {{AbstractStreamOperator}} level.
For what I proposed above, the {{StreamSource}} still needs to do the idleness
detection. The proposal is that additionally
{{TimestampsAndPeriodicWatermarksOperator}} and
{{TimestampsAndPunctuatedWatermarksOperator}} should also do it.
was (Author: tzulitai):
[~yanghua]
Ok. I think I was misguided by this sentence you mentioned:
{quote}
I think that assigning a watermark in a non-source operator is a reasonable
requirement, which can reduce the complexity of the source logic
{quote}
because this implies that the proposal is to do the idleness detection at the
{{AbstractStreamOperator}} level instead of in the {{StreamSource}}.
For what I proposed above, the {{StreamSource}} still needs to do the idleness
detection. The proposal is that additionally
{{TimestampsAndPeriodicWatermarksOperator}} and
{{TimestampsAndPunctuatedWatermarksOperator}} should also do it.
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many
> times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows:
> {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism
> of 10. Among them, the watermark is not sent by the source operator, but is
> downstream, such as flatmap. Every source subtask will not be idle. However,
> after the filter, some pipelines generate "idle". For example, there are 3
> pipelines that will no longer have data sent downstream. At this time, we
> can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline
> in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
