Jungtaek Lim created SPARK-61510:
------------------------------------

             Summary: Evict shorter stored windows in stream-stream joins by a 
cutoff lowered by the window length difference
                 Key: SPARK-61510
                 URL: https://issues.apache.org/jira/browse/SPARK-61510
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

Since SPARK-61348, a whole-window comparison such as {{leftWindow > 
rightWindow}} gives an eviction
cutoff only when the stored windows last at least as long as the other input's 
windows. When the
stored windows are shorter, the solver gives no cutoff:

* inner joins keep those windows for the life of the query and hold back the 
join's output
  watermark for that input;
* outer and left semi joins that depend on the bound fail analysis.

The rule is safe, because it never loses a match, but it keeps more state than 
necessary.

h3. Proposal

Use a cutoff that is lowered by the difference of the window lengths. A stored 
window S matches a
later window O only if O starts before S ends. A later O passes late filtering 
only if O ends above
the watermark W, so O starts after W minus the length of O. So S can be evicted 
when
{{S.end <= W - (len(O) - len(S))}}.

h3. Why a separate change

The change touches the shared cutoff solver that the analysis check and 
eviction both use, the
analysis error message, the join guide and the migration guide, and the window 
tests. A union of
window columns must record the length that the cutoff uses (SPARK-61348 round 3 
makes a union record
the length only when every child agrees). It needs its own review instead of 
growing a correctness
fix.

Related: SPARK-61348.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to