sigmod commented on pull request #32210: URL: https://github.com/apache/spark/pull/32210#issuecomment-856160864
> > PS, there's some additional complexity to get left/full-outer joins done right (assuming probe side is on the left) -- for a LOJ/FOJ, it seems that you need to tag each probe side row with "matched" v.s. "not-matched-yet", and then use this information in the customized merge join logic. > > @sigmod - this is valid point. For outer join, current SMJ cannot work out of box, need to push the matching info into sorter as well. This does add complexity for SMJ. There seem two ways to keep SMJ unchanged: - Option 1: pass an additional `condition` to check the tag field is "matched" v.s. "not-matched-yet"; - Option 2: only support this kind of fallback for inner joins, initially. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
