Jungtaek Lim created SPARK-61350:
------------------------------------
Summary: Reject negative watermark terms in stream-stream join
eviction bounds
Key: SPARK-61350
URL: https://issues.apache.org/jira/browse/SPARK-61350
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
The eviction-bound inference for stream-stream joins accepts a condition in
which the opposite watermark appears with a negative sign after normalization.
Example: for {{CAST(leftTime AS BIGINT) + CAST(rightTime AS BIGINT) > 100}}, a
later right row with a larger {{rightTime}} can match an arbitrarily old left
row. Spark treats the right watermark as an upper bound and removes left rows
that can still match. Valid matches are lost in V2, V3 and V4.
A normalized negative opposite-watermark term must not produce an eviction
bound. Outer and semi joins without any other valid bound must then fail
analysis.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]