cloud-fan commented on PR #38464: URL: https://github.com/apache/spark/pull/38464#issuecomment-1316611596
I agree with using bloom filters, as the size estimation can be wrong and the build size can be too large that `InSubquery` can't work. However, this PR contains another optimization that forces shuffle reuse when building the subquery to build bloom filter. Can we do it later with more discussions? This is a general optimization that can apply in other places as well: InSubquery DPP, bloom filter join. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
