[
https://issues.apache.org/jira/browse/SAMZA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350604#comment-14350604
]
Yi Pan (Data Infrastructure) commented on SAMZA-552:
----------------------------------------------------
Hi, [~jkreps], let me comment below:
# You are right about "window close". What I really meant for is that an event
or a timer to trigger *window update* for a particular window. Some types of
windows can truly be closed, such as a fixed length tuple window. But that
would simply means no more updates for that window to the downstream operators.
Regarding to the "single canonical output for a window", I am not very strong
about that either. The window update should be triggered either a) some system
time governed deadline demands a timely output; Or b) there is "enough" data in
the window to trigger a meaningful calculation. The whole purpose of "waiting
till the end" is trying to avoid too many window updates that could be
consolidated otherwise.
# As for the content stored in the state store, here are the thoughts that I
have now: a) "the aggregates == key value store changelog" probably means that
the key value store has a unique key per window aggregation, and we just update
the store for every update comes in? I agree that would be much more efficiency
in state management. However, as you already commented, it does not work for
the following cases:
## aggregations that need all samples in the window, such as median,
99-percentile
## joins that need to access each messages in a past window
Hence, I would rather focus on designing a general state store for the window
operator first. The aggregations that does not need full sample set in the
window can be considered later as an optimization.
# I agree with you that some aggregation can be pushed down to the window
operator as an optimization. But for the above reasons, I don't think that we
can make a general case for that.
Thanks!
> Tuple or time window semantics in physical operator
> ---------------------------------------------------
>
> Key: SAMZA-552
> URL: https://issues.apache.org/jira/browse/SAMZA-552
> Project: Samza
> Issue Type: Sub-task
> Components: sql
> Affects Versions: 0.9.0
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
> Attachments: DESIGN-SAMZA-552-3.md, DESIGN-SAMZA-552-3.pdf
>
>
> The discussion is based on how to support tuple and/or time based window
> operators in Samza physical operator layer.
> Here are the few observations:
> # Tuple represents the “physical ordering” of events while time-based window
> has semantic meanings to users
> # Total ordering between tuples are possible within Samza/Kafka given a
> deterministic MessageSelector on all input streams and offsets within each
> stream
> # No matter whether tuple or time is used to measure the window size, the
> window termination condition is needed to close a window to avoid the job to
> be wedged forever
> The following questions have to be answered to fully implement a window
> operator:
> # how to determine that a window is closed and no new tuples will be added?
> ## For tuple based, how do we close the window if messages do not come or get
> delayed?
> ## For time based, how do we close the window if
> ### the messages are not strictly in order w/ the time?
> ### the message w/ timestamp greater than the window boundary does not come
> or gets delayed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
