[
https://issues.apache.org/jira/browse/SPARK-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243133#comment-14243133
]
Reza Zadeh commented on SPARK-4823:
-----------------------------------
Given that we're talking about RowMatrices, computing rowSimilarities the same
way as columnSimilarities would require transposing the matrix, which is
dangerous when the original matrix has many rows. RowMatrix assumes a single
row should fit in memory on a single machine, but this might not happen after
transposing a RowMatrix.
> rowSimilarities
> ---------------
>
> Key: SPARK-4823
> URL: https://issues.apache.org/jira/browse/SPARK-4823
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: Reza Zadeh
>
> RowMatrix has a columnSimilarities method to find cosine similarities between
> columns.
> A rowSimilarities method would be useful to find similarities between rows.
> This is JIRA is to investigate which algorithms are suitable for such a
> method, better than brute-forcing it. Note that when there are many rows (>
> 10^6), it is unlikely that brute-force will be feasible, since the output
> will be of order 10^12.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
