This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0f19599f463a [SPARK-58912][ML][FOLLOWUP] Avoid intermediate
collections in CountVectorizer transform
0f19599f463a is described below
commit 0f19599f463a5f4c99e4a635504419059881b902
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Aug 28 09:57:29 2026 +0800
[SPARK-58912][ML][FOLLOWUP] Avoid intermediate collections in
CountVectorizer transform
### What changes were proposed in this pull request?
This follow-up to #58170 starts the `CountVectorizerModel` count filtering
and conversion
pipelines from an iterator for both binary and non-binary output.
### Why are the changes needed?
`OpenHashMap` is a strict `Iterable`, so calling `filter` and `map` on it
materializes intermediate
collections for every transformed document. Using its iterator keeps those
operations lazy and
materializes only the sequence required by `Vectors.sparse`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Not run. This only changes collection evaluation from strict to lazy;
existing
`CountVectorizerSuite` transform tests cover the output behavior. `git diff
--check` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #58325 from zhengruifeng/ml-countvectorizer-iterator-followup-dev-6.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit b9b9facbae181bdd90cc131fe3bf32241497d3c7)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
index a0da92fa7d7e..2ef1bf1f43a3 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala
@@ -326,9 +326,10 @@ class CountVectorizerModel(
}
val effectiveMinTF = if (localMinTF >= 1.0) localMinTF else tokenCount *
localMinTF
val effectiveCounts = if (localBinary) {
- termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1, 1.0)).toSeq
+ termCounts.iterator.filter(_._2 >= effectiveMinTF).map(p => (p._1,
1.0)).toSeq
} else {
- termCounts.filter(_._2 >= effectiveMinTF).map(p => (p._1,
p._2.toDouble)).toSeq
+ termCounts.iterator
+ .filter(_._2 >= effectiveMinTF).map(p => (p._1, p._2.toDouble)).toSeq
}
Vectors.sparse(dictSize, effectiveCounts)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
