cloud-fan commented on pull request #29104: URL: https://github.com/apache/spark/pull/29104#issuecomment-660182641
> we can always change buildSide into a HashSet, and streamedSide just need to lookup in the HashSet, then the calculation will be optimized into M*log(N). Taking the example you give, `Anti Or((ps_suppkey=s_suppkey), IsNull(ps_suppkey=s_suppkey))`, it means we need to find matches that: `ps_suppkey != s_suppkey` and `ps_suppkey not null` and `s_suppkey not null`. Assume `s_suppkey` is the build side, then we collect all the non-null values of `s_suppkey` and create a hash set. Then we iterate all the non-null values of `ps_suppkey`, and return a row if we can't find a match in the hash set. Do I understand correctly? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
