Fabian Hueske commented on FLINK-7245:

Great that we are on the same page!

Let's see if we want to / need to touch the public DataStream API. This is 
always a bit delicate. 
We can also first implement a custom operator that we only use internally for 
the Table API. 

[~aljoscha] what are your thoughts about adding such an operator to the 
DataStream API or extending the {{ProcessFunction}} context and 
{{AbstractStreamOperator}}? The changes for a constant watermark delay should 
be moderate. But I'm not sure if we want this in the public API. What do you 

> Enhance the operators to support holding back watermarks
> --------------------------------------------------------
>                 Key: FLINK-7245
>                 URL: https://issues.apache.org/jira/browse/FLINK-7245
>             Project: Flink
>          Issue Type: New Feature
>          Components: DataStream API
>            Reporter: Xingcan Cui
>            Assignee: Xingcan Cui
> Currently the watermarks are applied and emitted by the 
> {{AbstractStreamOperator}} instantly. 
> {code:java}
> public void processWatermark(Watermark mark) throws Exception {
>       if (timeServiceManager != null) {
>               timeServiceManager.advanceWatermark(mark);
>       }
>       output.emitWatermark(mark);
> }
> {code}
> Some calculation results (with timestamp fields) triggered by these 
> watermarks (e.g., join or aggregate results) may be regarded as delayed by 
> the downstream operators since their timestamps must be less than or equal to 
> the corresponding triggers. 
> This issue aims to add another "working mode", which supports holding back 
> watermarks, to current operators. These watermarks should be blocked and 
> stored by the operators until all the corresponding new generated results are 
> emitted.

This message was sent by Atlassian JIRA

Reply via email to