[
https://issues.apache.org/jira/browse/SPARK-61985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61985:
-----------------------------------
Labels: pull-request-available (was: )
> Stream-stream join derives range cutoffs from conjuncts that do not advance
> with the other input's watermark
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61985
> URL: https://issues.apache.org/jira/browse/SPARK-61985
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> The join solves each range conjunct for a state cutoff by substituting the
> other
> input's watermark. Conjuncts without the watermarked column ({{CAST(lt AS
> LONG) > 10}}) or with
> it negated ({{CAST(lt AS LONG) + CAST(rt AS LONG) > 100}}) still produce a
> cutoff. The first
> caps the cutoff of a valid bound, so a left outer join never emits its null
> rows. The second
> evicts state that later rows still match, so an inner join loses matches.
> Fix: use only
> conjuncts where the watermark term is positive.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]