zhengruifeng opened a new pull request, #57845: URL: https://github.com/apache/spark/pull/57845
### What changes were proposed in this pull request? This PR optimizes `HashingTF.transform` by selecting the binary/counting UDF and hash function from `($(binary), hashFuncVersion)` before executing the UDF. The UDF calculates feature indexes from the captured hash function and local `numFeatures` value instead of calling the instance `indexOf` method. Binary mode uses `OpenHashMap.update`, while counting mode uses `changeValue`. Both store integer term counts and convert them to `Double` only when constructing the output sparse vector. The compatibility test for HashingTF models saved before Spark 3.0 now also verifies `transform`, including the saved binary mode and legacy hash function. ### Why are the changes needed? Calling `indexOf` from the UDF captures the `HashingTF` transformer and performs a parameter lookup and hash-version match for every term. Selecting the behavior before constructing the UDF reduces the closure to the values it needs, removes repeated branching and parameter lookups, and reduces the term-count map value size from `Double` to `Int`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 'mllib/testOnly org.apache.spark.ml.feature.HashingTFSuite'` All 6 tests passed. The compiled UDF helper signatures were also inspected to confirm that they do not retain a `HashingTF` receiver. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
