Jungtaek Lim created SPARK-61987:
------------------------------------
Summary: Stream-stream outer and semi joins are admitted when the
watermark cannot finalize their output
Key: SPARK-61987
URL: https://issues.apache.org/jira/browse/SPARK-61987
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
The analysis check for stream-stream left outer, right outer, full outer and
left semi joins accepts any watermarked equality key, on either input. A left
outer join on
{{lt = rt}} with a watermark only on {{lt}} is accepted. It outputs {{(1, 5,
null)}} and then misses
the match with a later right row {{(1, 5)}}, which is not late because the
right input has no
watermark. Backward-shifted keys lose matches the same way. Fix: require the
other input's
watermark to bound the state of every input whose rows the join outputs once.
Add an internal
conf to turn the check off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]