This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c2c335a80f2b [SPARK-58912][ML] Use integer counts in CountVectorizer
transform
c2c335a80f2b is described below
commit c2c335a80f2b676438889da8ab81ef2a567ff626
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Aug 21 08:13:52 2026 +0800
[SPARK-58912][ML] Use integer counts in CountVectorizer transform
### What changes were proposed in this pull request?
This PR changes the temporary per-row count map used by
`CountVectorizerModel.transform` from
`OpenHashMap[Int, Double]` to `OpenHashMap[Int, Int]`.
The transform path only increments word counts by one while scanning the
input document. The
counts are converted to `Double` when constructing the output sparse vector.
### Why are the changes needed?
Using integer values in the temporary map reduces memory used while
counting words for each input
document. The output vector still uses `Double` values, so the external
behavior is unchanged.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not run; change is covered by existing `CountVectorizerSuite` transform
coverage.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #58170 from
zhengruifeng/spark-58912-count-vectorizer-transform-int-counts.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit e7470b4b41ce72d6164cbc784b0a459d5deb6851)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../org/apache/spark/ml/feature/CountVectorizer.scala | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
index 033c53d5afd7..f7d63faeea79 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
@@ -326,25 +326,23 @@ class CountVectorizerModel(
broadcastDict = Some(dataset.sparkSession.sparkContext.broadcast(dict))
}
val dictBr = broadcastDict.get
- // SPARK-48837: capture parameter values here so that we only evaulate
once-per-transform
- // rather than once-per-row:
- val minTf = $(minTF)
- val isBinary = $(binary)
+ val localMinTF = $(minTF)
+ val localBinary = $(binary)
val vectorizer = udf { document: Seq[String] =>
- val termCounts = new OpenHashMap[Int, Double]
+ val termCounts = new OpenHashMap[Int, Int]
var tokenCount = 0L
document.foreach { term =>
dictBr.value.get(term) match {
- case Some(index) => termCounts.changeValue(index, 1.0, _ + 1.0)
+ case Some(index) => termCounts.changeValue(index, 1, _ + 1)
case None => // ignore terms not in the vocabulary
}
tokenCount += 1
}
- val effectiveMinTF = if (minTf >= 1.0) minTf else tokenCount * minTf
- val effectiveCounts = if (isBinary) {
+ val effectiveMinTF = if (localMinTF >= 1.0) localMinTF else tokenCount *
localMinTF
+ val effectiveCounts = if (localBinary) {
termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1, 1.0)).toSeq
} else {
- termCounts.filter(_._2 >= effectiveMinTF).toSeq
+ termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1,
p._2.toDouble)).toSeq
}
Vectors.sparse(dictBr.value.size, effectiveCounts)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
