arunmahadevan commented on issue #23634: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows URL: https://github.com/apache/spark/pull/23634#issuecomment-458277908 Thanks for the explanation, so it seems the right side rows are retained until the global watermark (min of left and right watermarks). But the left side rows are evicted later due to the join condition and then it joins with a 'null' in the right side since the right side got evicted before. Then what you are proposing (storing the matched state) makes sense as long as it handles the different join types and conditions. If it does not, may be we need to retain the right side rows till the left side rows are evicted.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
