[ 
https://issues.apache.org/jira/browse/SPARK-61987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61987:
-----------------------------------
    Labels: pull-request-available  (was: )

> Stream-stream outer and semi joins are admitted when the watermark cannot 
> finalize their output
> -----------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61987
>                 URL: https://issues.apache.org/jira/browse/SPARK-61987
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> The analysis check for stream-stream left outer, right outer, full outer and
> left semi joins accepts any watermarked equality key, on either input. A left 
> outer join on
> {{lt = rt}} with a watermark only on {{lt}} is accepted. It outputs {{(1, 5, 
> null)}} and then misses
> the match with a later right row {{(1, 5)}}, which is not late because the 
> right input has no
> watermark. Backward-shifted keys lose matches the same way. Fix: require the 
> other input's
> watermark to bound the state of every input whose rows the join outputs once. 
> Add an internal
> conf to turn the check off.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to