[
https://issues.apache.org/jira/browse/FLINK-11454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771340#comment-16771340
]
Rong Rong commented on FLINK-11454:
-----------------------------------
Yes Thanks [~fhueske] for the feedback.
*Regarding moving the logic:* I also see that the wrapped function in TableAPI
used by the OVER operation as actually a rich function with complex APIs
related to the table operations (forwarding fields, etc). So far I wasn't sure
that can be done in one shot but we can also move it bit by bit if necessary.
*Regarding the OVER implementation in DataStreamAPI*: I was actually thinking
of making it similar to how WindowOperator works: As of now, if the
GeneratedAggregate in the OVER operator is heavy in retract() but light in
accumulate(). this will not actually help because OVER aggregate always try to
"retract()" in the bounded cases.
With how WindowOperator works: we can make it flexible: e.g. let user (or
higher level API) decide whether to:
# use a AggregatingState with a PassThroughFunction: this is the current OVER
aggregate approach.
# a ListState with a one-off Aggregating WindowFunction: this is similar to
how EvictingWindowOperator does.
*Regarding the "flexibility":* yes, there's only time-sliding and
count-sliding. which corresponding to RangOver and RowOver.
However, I would like to ask for some comments regarding how the "PaneState"
gets store/carry between operators, as you can see, there must be a state
between slice-stage and merge-state. currently I am implementing it with a
special container type call "Slice", see the
[PR|https://github.com/apache/flink/pull/7678/files#diff-48f54083076d21c54f8d49c36c3647a8R34]
in FLINK-11453. However I wasn't sure if creating a dedicated StateDescriptor
for this would've been a better idea as [~jark] have suggested in the design
doc.
> Support MergedStream operation
> ------------------------------
>
> Key: FLINK-11454
> URL: https://issues.apache.org/jira/browse/FLINK-11454
> Project: Flink
> Issue Type: Sub-task
> Components: DataStream API
> Reporter: Rong Rong
> Assignee: Rong Rong
> Priority: Major
>
> {{Following SlicedStream, the mergedStream operator merges results from
> sliced stream and produces windowing results.
> {code:java}
> val slicedStream: SlicedStream = inputStream
> .keyBy("key")
> .sliceWindow(Time.seconds(5L)) // new “slice window” concept: to
> combine
> // tumble results based on discrete
> // non-overlapping windows.
> .aggregate(aggFunc)
> val mergedStream1: MergedStream = slicedStream
> .slideOver(Time.second(10L)) // combine slice results with same
>
> // windowing function, equivalent to
> // WindowOperator with an aggregate
> state
> // and derived aggregate function.
> val mergedStream2: MergedStream = slicedStream
> .slideOver(Count.of(5))
> .apply(windowFunction) // apply a different window function
> over
> // the sliced results.{code}
> MergedStream are produced by MergeOperator:
> {{slideOver}} and {{apply}} can be combined into a {{OVER AGGREGATE}}
> implementation similar to the one in TableAPI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)