zhengruifeng opened a new pull request, #58278: URL: https://github.com/apache/spark/pull/58278
### What changes were proposed in this pull request? Replace the immutable `Map[String, Int]` used by `CountVectorizerModel` for vocabulary lookup with `OpenHashMap[String, Int]`. Add `Utils.toOpenHashMapWithIndex`, which accepts an `Array` so the initial size is known. `CountVectorizerModel` stores vocabulary indexes with an offset of one, reserving zero for missing terms and avoiding an `Option` allocation on each lookup. ### Why are the changes needed? `CountVectorizerModel.transform` performs one vocabulary lookup per input token. The specialized `Int` values in `OpenHashMap` avoid boxed map values, and direct lookup with the zero sentinel avoids allocating `Option` objects. In a local Java 17 microbenchmark with 262,144 string keys, 2 million lookups, and a 90% hit rate, immutable `Map.get` took about 69.5 ns/op while `OpenHashMap.apply` with the zero sentinel took about 37.6 ns/op, a roughly 46% improvement. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added coverage for `Utils.toOpenHashMapWithIndex` and ran: - `build/sbt "core/testOnly *OpenHashMapSuite"` - `build/sbt "mllib/testOnly *CountVectorizerSuite"` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
