Jungtaek Lim created SPARK-61987:
------------------------------------

             Summary: Stream-stream outer and semi joins are admitted when the 
watermark cannot finalize their output
                 Key: SPARK-61987
                 URL: https://issues.apache.org/jira/browse/SPARK-61987
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


The analysis check for stream-stream left outer, right outer, full outer and
left semi joins accepts any watermarked equality key, on either input. A left 
outer join on
{{lt = rt}} with a watermark only on {{lt}} is accepted. It outputs {{(1, 5, 
null)}} and then misses
the match with a later right row {{(1, 5)}}, which is not late because the 
right input has no
watermark. Backward-shifted keys lose matches the same way. Fix: require the 
other input's
watermark to bound the state of every input whose rows the join outputs once. 
Add an internal
conf to turn the check off.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to