marcuslin123 opened a new pull request, #57705: URL: https://github.com/apache/spark/pull/57705
### What changes were proposed in this pull request? This PR updates `InjectRuntimeFilter` to use `CollationAwareXxHash64` when building and probing runtime Bloom filters. It also updates the existing-filter check to recognize the collation-aware hash expression. A regression test covers a `UTF8_LCASE` equi-join whose application-side key is a `substr` expression. The test verifies both the query result and the hash expressions in the optimized plan. ### Why are the changes needed? Runtime Bloom filters currently use the collation-agnostic `XxHash64`. Values that compare equal under a non-binary collation can therefore produce different hashes. This can cause the Bloom filter to reject valid join rows before the collation-aware join is evaluated. For example, the regression query returns 5000 rows with runtime Bloom filters disabled but 0 rows when they are enabled because `"abc"` and `"ABC"` hash differently despite being equal under `UTF8_LCASE`. ### Does this PR introduce _any_ user-facing change? Yes. Equi-joins on collated keys, including non-attribute expressions, no longer silently lose matching rows when a runtime Bloom filter is injected. ### How was this patch tested? Added `SPARK-58486: runtime bloom filters use collation-aware hashing` to `InjectRuntimeFilterSuite`. Ran: ``` build/sbt 'sql/testOnly org.apache.spark.sql.InjectRuntimeFilterSuite' build/sbt catalyst/scalastyle sql/Test/scalastyle ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (used for code assistance). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
