Ruifeng Zheng created SPARK-58993:
-------------------------------------
Summary: Use OpenHashMap for CountVectorizer vocabulary lookup
Key: SPARK-58993
URL: https://issues.apache.org/jira/browse/SPARK-58993
Project: Spark
Issue Type: Sub-task
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
CountVectorizerModel.transform builds and broadcasts an immutable Map[String,
Int] for vocabulary
lookups. Replace it with Spark's OpenHashMap[String, Int], storing index + 1 so
that zero can be
used as the missing-key sentinel.
A Java 17 microbenchmark with the default 262,144-entry vocabulary and 2
million lookups (90%
hits) measured approximately 69.5 ns/lookup for the current immutable Map.get
match and 37.6
ns/lookup for OpenHashMap.apply with the sentinel representation. The change
should also avoid
boxed vocabulary indexes and per-lookup Option allocation while preserving
transform behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
