Cheng Su created SPARK-32863:
--------------------------------
Summary: Full outer stream-stream join
Key: SPARK-32863
URL: https://issues.apache.org/jira/browse/SPARK-32863
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 3.1.0
Reporter: Cheng Su
Current stream-stream join supports inner, left outer and right outer join
([https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingSymmetricHashJoinExec.scala#L166]
). With current design of stream-stream join (which marks whether the row is
matched or not in state store), it would be very easy to support full outer
join as well.
Full outer stream-stream join will work as followed:
(1).for left side input row, check if there's a match on right side state
store. If there's a match, output all matched rows. Put the row in left side
state store.
(2).for right side input row, check if there's a match on left side state
store. If there's a match, output all matched rows and update left side rows
state with "matched" field to set to true. Put the right side row in right side
state store.
(3).for left side row needs to be evicted from state store, output the row if
"matched" field is false.
(4).for right side row needs to be evicted from state store, output the row if
"matched" field is false.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
