Jungtaek Lim created SPARK-61985:
------------------------------------

             Summary: Stream-stream join derives range cutoffs from conjuncts 
that do not advance with the other input's watermark
                 Key: SPARK-61985
                 URL: https://issues.apache.org/jira/browse/SPARK-61985
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


The join solves each range conjunct for a state cutoff by substituting the other
input's watermark. Conjuncts without the watermarked column ({{CAST(lt AS LONG) 
> 10}}) or with
it negated ({{CAST(lt AS LONG) + CAST(rt AS LONG) > 100}}) still produce a 
cutoff. The first
caps the cutoff of a valid bound, so a left outer join never emits its null 
rows. The second
evicts state that later rows still match, so an inner join loses matches. Fix: 
use only
conjuncts where the watermark term is positive.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to