[
https://issues.apache.org/jira/browse/SAMZA-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343512#comment-14343512
]
Geoffry Sumter commented on SAMZA-552:
--------------------------------------
Jay, when you're talking about a rejection horizon, is that wall-clock-time?
I'm curious how you see this method working when replaying old data. Let's say
I have a month of raw input data in kafka and whenever I create a new model I
run it over that month and compare output to some source of ground truth. It
seems like this would cause two problems:
1. The local state becomes unreasonably large as you keep around weeks worth of
5 minute windows, since you're going through that month much faster than
'real-time'.
2. The processing of some partitions could proceed significantly faster than
other partitions, so if there's some sort of join/merge you can pile up state
waiting for a partition doing heavier processing to catch up.
> Tuple or time window semantics in physical operator
> ---------------------------------------------------
>
> Key: SAMZA-552
> URL: https://issues.apache.org/jira/browse/SAMZA-552
> Project: Samza
> Issue Type: Sub-task
> Components: sql
> Affects Versions: 0.9.0
> Reporter: Yi Pan (Data Infrastructure)
> Assignee: Yi Pan (Data Infrastructure)
>
> The discussion is based on how to support tuple and/or time based window
> operators in Samza physical operator layer.
> Here are the few observations:
> # Tuple represents the “physical ordering” of events while time-based window
> has semantic meanings to users
> # Total ordering between tuples are possible within Samza/Kafka given a
> deterministic MessageSelector on all input streams and offsets within each
> stream
> # No matter whether tuple or time is used to measure the window size, the
> window termination condition is needed to close a window to avoid the job to
> be wedged forever
> The following questions have to be answered to fully implement a window
> operator:
> # how to determine that a window is closed and no new tuples will be added?
> ## For tuple based, how do we close the window if messages do not come or get
> delayed?
> ## For time based, how do we close the window if
> ### the messages are not strictly in order w/ the time?
> ### the message w/ timestamp greater than the window boundary does not come
> or gets delayed?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
