Jungtaek Lim created SPARK-61985:
------------------------------------
Summary: Stream-stream join derives range cutoffs from conjuncts
that do not advance with the other input's watermark
Key: SPARK-61985
URL: https://issues.apache.org/jira/browse/SPARK-61985
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
The join solves each range conjunct for a state cutoff by substituting the other
input's watermark. Conjuncts without the watermarked column ({{CAST(lt AS LONG)
> 10}}) or with
it negated ({{CAST(lt AS LONG) + CAST(rt AS LONG) > 100}}) still produce a
cutoff. The first
caps the cutoff of a valid bound, so a left outer join never emits its null
rows. The second
evicts state that later rows still match, so an inner join loses matches. Fix:
use only
conjuncts where the watermark term is positive.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]