[
https://issues.apache.org/jira/browse/SPARK-61510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-61510:
---------------------------------
Affects Version/s: 4.4.0
> Evict shorter stored windows in stream-stream joins by a cutoff lowered by
> the window length difference
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61510
> URL: https://issues.apache.org/jira/browse/SPARK-61510
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> h3. Problem
> Since SPARK-61348, a whole-window comparison such as {{leftWindow >
> rightWindow}} gives an eviction
> cutoff only when the stored windows last at least as long as the other
> input's windows. When the
> stored windows are shorter, the solver gives no cutoff:
> * inner joins keep those windows for the life of the query and hold back the
> join's output
> watermark for that input;
> * outer and left semi joins that depend on the bound fail analysis.
> The rule is safe, because it never loses a match, but it keeps more state
> than necessary.
> h3. Proposal
> Use a cutoff that is lowered by the difference of the window lengths. A
> stored window S matches a
> later window O only if O starts before S ends. A later O passes late
> filtering only if O ends above
> the watermark W, so O starts after W minus the length of O. So S can be
> evicted when
> {{S.end <= W - (len(O) - len(S))}}.
> h3. Why a separate change
> The change touches the shared cutoff solver that the analysis check and
> eviction both use, the
> analysis error message, the join guide and the migration guide, and the
> window tests. A union of
> window columns must record the length that the cutoff uses (SPARK-61348 round
> 3 makes a union record
> the length only when every child agrees). It needs its own review instead of
> growing a correctness
> fix.
> Related: SPARK-61348.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]