Github user ymazari commented on a diff in the pull request:
https://github.com/apache/spark/pull/20367#discussion_r164260027
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/CountVectorizer.scala ---
@@ -160,6 +187,11 @@ class CountVectorizer @Since("1.5.0") (@Since("1.5.0")
override val uid: String)
} else {
$(minDF) * input.cache().count()
}
+ val maxDf = if ($(maxDF) >= 1.0) {
+ $(maxDF)
+ } else {
+ $(maxDF) * input.cache().count()
+ }
--- End diff --
Good points.
- I added a check that maxDF >= minDF
- Changed the code so that counting (and caching) is done only once
- I refactored the code so that "filter()" is only invoked if minDF or
maxDF is set
- Added an un-persisting the input after the counting is done.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
