[
https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776563#comment-16776563
]
vinoyang commented on FLINK-11286:
----------------------------------
[~tzulitai] When I looked at the history of this issue, I didn't mention that
I hope the "ALL" operators can mark them as IDLE. I just said "non-source
operator" and those operators which the watermark could be assigned. Maybe I
didn't express it clearly enough here, causing misunderstanding.
I have encountered this problem with [~jark] , which means that it is not
accidental. If IDLE can only be set in the source, then a lot of business logic
(preprocessing of the original data) must be compressed into the source
operator. But without this limitation, the business logic can be processed in
other operators with higher parallelism. This will help improve processing
performance.
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many
> times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows:
> {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism
> of 10. Among them, the watermark is not sent by the source operator, but is
> downstream, such as flatmap. Every source subtask will not be idle. However,
> after the filter, some pipelines generate "idle". For example, there are 3
> pipelines that will no longer have data sent downstream. At this time, we
> can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline
> in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
