[
https://issues.apache.org/jira/browse/SPARK-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14364219#comment-14364219
]
Joseph K. Bradley commented on SPARK-6340:
------------------------------------------
It is possible to keep track, as Ronald suggests in the user list thread: Split
the RDD of LabeledPoints into 2 RDDs (labels and feature vectors); transform
the feature vectors; and zip the label RDD with the transformed feature vector
RDD. You can see an example of zipping in many places, such as here:
[https://github.com/apache/spark/blob/e3f315ac358dfe4f5b9705c3eac76e8b1e24f82a/examples/src/main/scala/org/apache/spark/examples/mllib/LinearRegression.scala#L124]
We are currently working on making this easier via the DataFrame API (in the
experimental spark.ml package which provides a higher-level API for ML
workflows).
If you don't mind, I'm going to close this JIRA.
> mllib.IDF for LabelPoints
> -------------------------
>
> Key: SPARK-6340
> URL: https://issues.apache.org/jira/browse/SPARK-6340
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.3.0
> Environment: python 2.7.8
> pyspark
> OS: Linux Mint 17 Qiana (Cinnamon 64-bit)
> Reporter: Kian Ho
> Priority: Minor
> Labels: feature
>
> as per:
> http://apache-spark-user-list.1001560.n3.nabble.com/Using-TF-IDF-from-MLlib-td19429.html#a19528
> Having the IDF.fit accept LabelPoints would be useful since, correct me if
> i'm wrong, there currently isn't a way of keeping track of which labels
> belong to which documents if one needs to apply a conventional tf-idf
> transformation on labelled text data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
