[
https://issues.apache.org/jira/browse/SAMZA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350779#comment-14350779
]
Yi Pan (Data Infrastructure) commented on SAMZA-552:
----------------------------------------------------
Yes, I agree that it just seems as that the content of the window would differ
at a high-level view. The issue that makes me deviate from this high-level
abstraction is the case of sliding windows. In that case, many entries in two
adjacent sliding windows are the same and if we follow the abstraction of
storing the following:
(windowKey, windowOutput) where windowOutput can be optimized to an aggregated
result. For sliding windows in a inner-join, there would be huge amount of
redundant data. And in my mind, sliding windows for inner-join seems to be a
common use case. In this case, it is more efficient to use a single store that
keyed by eventtime / offset and allow access to the past windowOutput as a
range query to the store. The actual difference in the underlying store may be
hidden by an interface of getWindowOutput(windowKey).
> Tuple or time window semantics in physical operator
> ---------------------------------------------------
>
> Key: SAMZA-552
> URL: https://issues.apache.org/jira/browse/SAMZA-552
> Project: Samza
> Issue Type: Sub-task
> Components: sql
> Affects Versions: 0.9.0
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
> Attachments: DESIGN-SAMZA-552-3.md, DESIGN-SAMZA-552-3.pdf
>
>
> The discussion is based on how to support tuple and/or time based window
> operators in Samza physical operator layer.
> Here are the few observations:
> # Tuple represents the “physical ordering” of events while time-based window
> has semantic meanings to users
> # Total ordering between tuples are possible within Samza/Kafka given a
> deterministic MessageSelector on all input streams and offsets within each
> stream
> # No matter whether tuple or time is used to measure the window size, the
> window termination condition is needed to close a window to avoid the job to
> be wedged forever
> The following questions have to be answered to fully implement a window
> operator:
> # how to determine that a window is closed and no new tuples will be added?
> ## For tuple based, how do we close the window if messages do not come or get
> delayed?
> ## For time based, how do we close the window if
> ### the messages are not strictly in order w/ the time?
> ### the message w/ timestamp greater than the window boundary does not come
> or gets delayed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
