zhengruifeng commented on a change in pull request #29852:
URL: https://github.com/apache/spark/pull/29852#discussion_r494713601
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/HashingTF.scala
##########
@@ -91,20 +90,13 @@ class HashingTF @Since("3.0.0") private[ml] (
@Since("2.0.0")
override def transform(dataset: Dataset[_]): DataFrame = {
val outputSchema = transformSchema(dataset.schema)
- val localNumFeatures = $(numFeatures)
- val localBinary = $(binary)
+ val n = $(numFeatures)
+ val updateFunc = if ($(binary)) (v: Double) => 1.0 else (v: Double) => v +
1.0
val hashUDF = udf { terms: Seq[_] =>
- val termFrequencies = mutable.HashMap.empty[Int,
Double].withDefaultValue(0.0)
- terms.foreach { term =>
- val i = indexOf(term)
- if (localBinary) {
- termFrequencies(i) = 1.0
- } else {
- termFrequencies(i) += 1.0
- }
- }
- Vectors.sparse(localNumFeatures, termFrequencies.toSeq)
+ val map = new OpenHashMap[Int, Double]()
Review comment:
yes, the comment in `util.collection.OpenHashMap` only refer to java
hashmap, but to my knowledge, scala hashmap is slower than java hashmap.
and there is was also a [performance
test](https://gist.github.com/rxin/44657c3f40d4e4294857) between java hashmap
and scala hashmap: scala hashmap < java hashmap < scala openhashmap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
