HeartSaVioR commented on issue #23634: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows URL: https://github.com/apache/spark/pull/23634#issuecomment-464911031 > So it actually makes sense to add the support for matched flag as a purely optional feature within the existing SymmetricHashJoinStateManager. I would still suggest to have matched flag at any chance: suppose we don't touch anything but change the query from inner join to left outer join - then theoretically the result could be incorrect because we don't record `matched` field and all rows in left state can be emitted as right-null even they've matched. If I'm not missing here, the change of query is allowed, or even disallowed we don't fail the query before it gets processed so it would incur no error but incorrect result, right?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
