Ruifeng Zheng created SPARK-58637:
-------------------------------------
Summary: Optimize HashingTF transform closure
Key: SPARK-58637
URL: https://issues.apache.org/jira/browse/SPARK-58637
Project: Spark
Issue Type: Sub-task
Components: ML
Affects Versions: 5.0.0
Reporter: Ruifeng Zheng
HashingTF.transform calls the instance indexOf method from its UDF, which
captures the transformer and looks up numFeatures for every term. Select the
hash and binary update functions before constructing the UDF, calculate indexes
from captured local values, store term counts as Int, and convert them to
Double only when constructing the output vector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
