Jungtaek Lim created SPARK-61510:
------------------------------------
Summary: Evict shorter stored windows in stream-stream joins by a
cutoff lowered by the window length difference
Key: SPARK-61510
URL: https://issues.apache.org/jira/browse/SPARK-61510
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
Since SPARK-61348, a whole-window comparison such as {{leftWindow >
rightWindow}} gives an eviction
cutoff only when the stored windows last at least as long as the other input's
windows. When the
stored windows are shorter, the solver gives no cutoff:
* inner joins keep those windows for the life of the query and hold back the
join's output
watermark for that input;
* outer and left semi joins that depend on the bound fail analysis.
The rule is safe, because it never loses a match, but it keeps more state than
necessary.
h3. Proposal
Use a cutoff that is lowered by the difference of the window lengths. A stored
window S matches a
later window O only if O starts before S ends. A later O passes late filtering
only if O ends above
the watermark W, so O starts after W minus the length of O. So S can be evicted
when
{{S.end <= W - (len(O) - len(S))}}.
h3. Why a separate change
The change touches the shared cutoff solver that the analysis check and
eviction both use, the
analysis error message, the join guide and the migration guide, and the window
tests. A union of
window columns must record the length that the cutoff uses (SPARK-61348 round 3
makes a union record
the length only when every child agrees). It needs its own review instead of
growing a correctness
fix.
Related: SPARK-61348.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]