[
https://issues.apache.org/jira/browse/SPARK-61987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61987:
-----------------------------------
Labels: pull-request-available (was: )
> Stream-stream outer and semi joins are admitted when the watermark cannot
> finalize their output
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-61987
> URL: https://issues.apache.org/jira/browse/SPARK-61987
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> The analysis check for stream-stream left outer, right outer, full outer and
> left semi joins accepts any watermarked equality key, on either input. A left
> outer join on
> {{lt = rt}} with a watermark only on {{lt}} is accepted. It outputs {{(1, 5,
> null)}} and then misses
> the match with a later right row {{(1, 5)}}, which is not late because the
> right input has no
> watermark. Backward-shifted keys lose matches the same way. Fix: require the
> other input's
> watermark to bound the state of every input whose rows the join outputs once.
> Add an internal
> conf to turn the check off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]