[
https://issues.apache.org/jira/browse/SPARK-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021527#comment-15021527
]
Sen Fang commented on SPARK-2336:
---------------------------------
I finally took a crack on the hybrid spill tree for kNN and results so far
appear to be promising. For anyone who is still interested, you can find it as
a spark package at: https://github.com/saurfang/spark-knn
The implementation is written for ml API and scales well in terms of both
number of observations and number of vector dimensions. The KNN itself is
flexible and the package comes with KNNClassifier and KNNRegression for
(optionally weighted) classification and regression.
There are a few implementation details I am still trying to iron out. Otherwise
I look forward to benchmark it against other implementations such as KNN-join,
KD-Tree, and LSH.
> Approximate k-NN Models for MLLib
> ---------------------------------
>
> Key: SPARK-2336
> URL: https://issues.apache.org/jira/browse/SPARK-2336
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Brian Gawalt
> Priority: Minor
> Labels: clustering, features
>
> After tackling the general k-Nearest Neighbor model as per
> https://issues.apache.org/jira/browse/SPARK-2335 , there's an opportunity to
> also offer approximate k-Nearest Neighbor. A promising approach would involve
> building a kd-tree variant within from each partition, a la
> http://www.autonlab.org/autonweb/14714.html?branch=1&language=2
> This could offer a simple non-linear ML model that can label new data with
> much lower latency than the plain-vanilla kNN versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
