[ 
https://issues.apache.org/jira/browse/SPARK-61985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-61985:
---------------------------------
    Affects Version/s: 4.3.0
                       4.4.0

> Stream-stream join derives range cutoffs from conjuncts that do not advance 
> with the other input's watermark
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61985
>                 URL: https://issues.apache.org/jira/browse/SPARK-61985
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> The join solves each range conjunct for a state cutoff by substituting the 
> other
> input's watermark. Conjuncts without the watermarked column ({{CAST(lt AS 
> LONG) > 10}}) or with
> it negated ({{CAST(lt AS LONG) + CAST(rt AS LONG) > 100}}) still produce a 
> cutoff. The first
> caps the cutoff of a valid bound, so a left outer join never emits its null 
> rows. The second
> evicts state that later rows still match, so an inner join loses matches. 
> Fix: use only
> conjuncts where the watermark term is positive.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to