[ 
https://issues.apache.org/jira/browse/FLINK-11454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771340#comment-16771340
 ] 

Rong Rong commented on FLINK-11454:
-----------------------------------

Yes Thanks [~fhueske] for the feedback.

 

*Regarding moving the logic:* I also see that the wrapped function in TableAPI 
used by the OVER operation as actually a rich function with complex APIs 
related to the table operations (forwarding fields, etc). So far I wasn't sure 
that can be done in one shot but we can also move it bit by bit if necessary.

*Regarding the OVER implementation in DataStreamAPI*: I was actually thinking 
of making it similar to how WindowOperator works: As of now, if the 
GeneratedAggregate in the OVER operator is heavy in retract() but light in 
accumulate(). this will not actually help because OVER aggregate always try to 
"retract()" in the bounded cases.

With how WindowOperator works: we can make it flexible: e.g. let user (or 
higher level API) decide whether to:
 # use a AggregatingState with a PassThroughFunction: this is the current OVER 
aggregate approach.
 # a ListState with a one-off Aggregating WindowFunction: this is similar to 
how EvictingWindowOperator does.

*Regarding the "flexibility":* yes, there's only time-sliding and 
count-sliding. which corresponding to RangOver and RowOver. 
 However, I would like to ask for some comments regarding how the "PaneState" 
gets store/carry between operators, as you can see, there must be a state 
between slice-stage and merge-state. currently I am implementing it with a 
special container type call "Slice", see the 
[PR|https://github.com/apache/flink/pull/7678/files#diff-48f54083076d21c54f8d49c36c3647a8R34]
 in FLINK-11453. However I wasn't sure if creating a dedicated StateDescriptor 
for this would've been a better idea as [~jark] have suggested in the design 
doc.

 

 

> Support MergedStream operation
> ------------------------------
>
>                 Key: FLINK-11454
>                 URL: https://issues.apache.org/jira/browse/FLINK-11454
>             Project: Flink
>          Issue Type: Sub-task
>          Components: DataStream API
>            Reporter: Rong Rong
>            Assignee: Rong Rong
>            Priority: Major
>
> {{Following SlicedStream, the mergedStream operator merges results from 
> sliced stream and produces windowing results.
> {code:java}
> val slicedStream: SlicedStream = inputStream
>   .keyBy("key")
>   .sliceWindow(Time.seconds(5L))       // new “slice window” concept: to 
> combine 
>                                        // tumble results based on discrete
>                                        // non-overlapping windows.
>   .aggregate(aggFunc)
> val mergedStream1: MergedStream = slicedStream
>   .slideOver(Time.second(10L))         // combine slice results with same     
>            
>                                        // windowing function, equivalent to 
>                                        // WindowOperator with an aggregate 
> state 
>                                        // and derived aggregate function.
> val mergedStream2: MergedStream = slicedStream
>   .slideOver(Count.of(5))
>   .apply(windowFunction)               // apply a different window function 
> over  
>                                        // the sliced results.{code}
> MergedStream are produced by MergeOperator:
> {{slideOver}} and {{apply}} can be combined into a {{OVER AGGREGATE}} 
> implementation similar to the one in TableAPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to