Hmm. I have a lot of code on the local linear algebra operations using Spark's Matrix and Vector representations done for https://issues.apache.org/jira/browse/SPARK-6442.
I can make a Spark package with that code if people are interested. Best, Burak On Tue, Aug 25, 2015 at 10:54 AM, Kristina Rogale Plazonic <[email protected] > wrote: > However I do think it's easier than it seems to write the implicits; >> it doesn't involve new classes or anything. Yes it's pretty much just >> what you wrote. There is a class "Vector" in Spark. This declaration >> can be in an object; you don't implement your own class. (Also you can >> use "toBreeze" to get Breeze vectors.) > > > The implicit conversion with the implicit def happens for the first vector > in the sum, but not the second vector (see below). > > At this point I give up, because I spent way too much time. I am so > disappointed. So many times I heard "Spark makes simple things easy and > complicated things possible". Well, here is the simplest thing you can > imagine in linear algebra, but heck, it is not easy or intuitive. It was > easier to run a DeepLearning algo (from another library) than add two > vectors. > > If anybody has a workaround other than implementing your own > add/substract/scalarMultiply, PLEASE let me know. > > Here is the code and error from (freshly started) spark-shell: > > scala> import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, > Vector => BV} > import breeze.linalg.{DenseVector=>BDV, SparseVector=>BSV, Vector=>BV} > > scala> import org.apache.spark.mllib.linalg.Vectors > import org.apache.spark.mllib.linalg.Vectors > > scala> val v1 = Vectors.dense(1.0, 2.0, 3.0) > v1: org.apache.spark.mllib.linalg.Vector = [1.0,2.0,3.0] > > scala> import org.apache.spark.mllib.linalg.{Vector =>SparkVector} > import org.apache.spark.mllib.linalg.{Vector=>SparkVector} > > scala> object MyUtils { > | implicit def toBreeze(v:SparkVector) = BV(v.toArray) > | } > warning: there were 1 feature warning(s); re-run with -feature for details > defined module MyUtils > > scala> import MyUtils._ > import MyUtils._ > > scala> v1:BV[Double] > res2: breeze.linalg.Vector[Double] = DenseVector(1.0, 2.0, 3.0) > > scala> v1 + v1 > <console>:30: error: could not find implicit value for parameter op: > breeze.linalg.operators.OpAdd.Impl2[breeze.linalg.Vector[Double],org.apache.spark.mllib.linalg.Vector,That] > v1 + v1 > ^ > > >
