[
https://issues.apache.org/jira/browse/FLINK-11286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776597#comment-16776597
]
Jark Wu commented on FLINK-11286:
---------------------------------
Hi [~tzulitai], thanks for your feedback.
I agree with you that the watermark-generating operators should support
timeout-based automatic idleness detection (and this is what the issue need to
do). We can continue the discussion about proposing an API in this JIRA or move
to mailing list.
Regarding to the "disallow mid-pipeline arbitrary watermark assigners".
I don't feel strongly about it, but I'm okay with this. We also only allow
watermark assigner from source in Flink SQL / Table API, and this simplifies
much things.
I want to re-elaborate what I mean about decoupling source logic and watermark
logic. IMO, it should be better NOT combining event-time assigner and watermark
assigner into every specific source implementation. We can limit watermark
assigner to be only followed after source, by moving the
{{assignTimestampsAndWatermarks}} to {{DataStreamSource}}. But in physical,
they can be two operators but chained together.
However, on the other hand, [~yanghua] raised a good problem which is what we
are also facing.
For example, the parser/extraction may be heavy (e.g. using regex to extract
event-time). But if we enforce the limitation, the parser is in the source or
the parallelism of the parser is enforced to be the same with the source (say
only 4 partitions). As a result, the parser become the bottleneck of the job,
and we can't scale out the job. But without the limitation, we can increase the
parallelism of the parser (say 128 tasks) to get a better throughput.
> Support to send StreamStatus.IDLE for non-source operators
> -----------------------------------------------------------
>
> Key: FLINK-11286
> URL: https://issues.apache.org/jira/browse/FLINK-11286
> Project: Flink
> Issue Type: Improvement
> Components: DataStream API
> Reporter: vinoyang
> Assignee: vinoyang
> Priority: Major
>
> Currently, only stream source tasks can be marked as temporary idle. But many
> times, this approach has limitations.
> Considering such a scenario, there is a DAG as follows:
> {{source->map->filter->flatmap->keyBy->window}}, with a degree of parallelism
> of 10. Among them, the watermark is not sent by the source operator, but is
> downstream, such as flatmap. Every source subtask will not be idle. However,
> after the filter, some pipelines generate "idle". For example, there are 3
> pipelines that will no longer have data sent downstream. At this time, we
> can't call the {{markAsTemporarilyIdle}} method to mark the current pipeline
> in the idle state. This will affect the downstream window.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
