[
https://issues.apache.org/jira/browse/SAMZA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350680#comment-14350680
]
Jay Kreps commented on SAMZA-552:
---------------------------------
I think we are in agreement about pushdown. Incremental operations can store
the value so far and in that case the changelog is the output. This will cover
the common case of sums, counts, max, min, and other projections. What is the
right mathematical term for this property? I think a full outer join has this
property but other joins don't. Medians and quantiles don't have the property
as you say though they are not too important.
However in those common cases the aggregation would just be a normal state
store containing the result so far, right?
In the non-optimizable cases we are still talking about using the existing
key-value store, right? It is just that the content of this store isn't the
output of the operator. This would be the case for an inner join, I think,
where the contents of the store are the rows in the window but the output is
the join computed off this state.
I think it would be good to use (or improve) the existing abstraction so that
we can put some time into really nailing performance on that and have it pay
off for both direct Java usage of the key value store and also the operator
usage.
> Tuple or time window semantics in physical operator
> ---------------------------------------------------
>
> Key: SAMZA-552
> URL: https://issues.apache.org/jira/browse/SAMZA-552
> Project: Samza
> Issue Type: Sub-task
> Components: sql
> Affects Versions: 0.9.0
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
> Attachments: DESIGN-SAMZA-552-3.md, DESIGN-SAMZA-552-3.pdf
>
>
> The discussion is based on how to support tuple and/or time based window
> operators in Samza physical operator layer.
> Here are the few observations:
> # Tuple represents the “physical ordering” of events while time-based window
> has semantic meanings to users
> # Total ordering between tuples are possible within Samza/Kafka given a
> deterministic MessageSelector on all input streams and offsets within each
> stream
> # No matter whether tuple or time is used to measure the window size, the
> window termination condition is needed to close a window to avoid the job to
> be wedged forever
> The following questions have to be answered to fully implement a window
> operator:
> # how to determine that a window is closed and no new tuples will be added?
> ## For tuple based, how do we close the window if messages do not come or get
> delayed?
> ## For time based, how do we close the window if
> ### the messages are not strictly in order w/ the time?
> ### the message w/ timestamp greater than the window boundary does not come
> or gets delayed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
