[ 
https://issues.apache.org/jira/browse/SPARK-61510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-61510:
---------------------------------
    Affects Version/s: 4.4.0

> Evict shorter stored windows in stream-stream joins by a cutoff lowered by 
> the window length difference
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61510
>                 URL: https://issues.apache.org/jira/browse/SPARK-61510
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> h3. Problem
> Since SPARK-61348, a whole-window comparison such as {{leftWindow > 
> rightWindow}} gives an eviction
> cutoff only when the stored windows last at least as long as the other 
> input's windows. When the
> stored windows are shorter, the solver gives no cutoff:
> * inner joins keep those windows for the life of the query and hold back the 
> join's output
>   watermark for that input;
> * outer and left semi joins that depend on the bound fail analysis.
> The rule is safe, because it never loses a match, but it keeps more state 
> than necessary.
> h3. Proposal
> Use a cutoff that is lowered by the difference of the window lengths. A 
> stored window S matches a
> later window O only if O starts before S ends. A later O passes late 
> filtering only if O ends above
> the watermark W, so O starts after W minus the length of O. So S can be 
> evicted when
> {{S.end <= W - (len(O) - len(S))}}.
> h3. Why a separate change
> The change touches the shared cutoff solver that the analysis check and 
> eviction both use, the
> analysis error message, the join guide and the migration guide, and the 
> window tests. A union of
> window columns must record the length that the cutoff uses (SPARK-61348 round 
> 3 makes a union record
> the length only when every child agrees). It needs its own review instead of 
> growing a correctness
> fix.
> Related: SPARK-61348.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to