[
https://issues.apache.org/jira/browse/SPARK-8703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606700#comment-14606700
]
Feynman Liang commented on SPARK-8703:
--------------------------------------
This seems to extend HashingTF by adding
* a user-specified vocabulary
* filtering for words above a minimum frequency
* no possibility of hash collisions
I agree with [~viirya] that it would be nice to reuse HashingTF if possible.
> Add CountVectorizer as a ml transformer to convert document to words count
> vector
> ---------------------------------------------------------------------------------
>
> Key: SPARK-8703
> URL: https://issues.apache.org/jira/browse/SPARK-8703
> Project: Spark
> Issue Type: New Feature
> Components: ML
> Reporter: yuhao yang
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Converts a text document to a sparse vector of token counts. Similar to
> http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
> I can further add an estimator to extract vocabulary from corpus if that's
> appropriate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
