thanks reza. i'm not familiar with the "block matrix multiplication", but is it a good fit for "very large dimension, but extremely sparse" matrix?
if not, what is your recommendation on implementing matrix multiplication in spark on "very large dimension, but extremely sparse" matrix? On Thu, Nov 6, 2014 at 5:50 PM, Reza Zadeh <r...@databricks.com> wrote: > See this thread for examples of sparse matrix x sparse matrix: > https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA > > We thought about providing matrix multiplies on CoordinateMatrix, however, > the matrices have to be very dense for the overhead of having many little > (i, j, value) objects to be worth it. For this reason, we are focused on > doing block matrix multiplication first. The goal is version 1.3. > > Best, > Reza > > On Wed, Nov 5, 2014 at 11:48 PM, Wei Tan <w...@us.ibm.com> wrote: > >> I think Xiangrui's ALS code implement certain aspect of it. You may want >> to check it out. >> Best regards, >> Wei >> >> --------------------------------- >> Wei Tan, PhD >> Research Staff Member >> IBM T. J. Watson Research Center >> >> >> [image: Inactive hide details for Xiangrui Meng ---11/05/2014 01:13:40 >> PM---You can use breeze for local sparse-sparse matrix multiplic]Xiangrui >> Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse >> matrix multiplication and then define an RDD of sub-matri >> >> From: Xiangrui Meng <men...@gmail.com> >> To: Duy Huynh <duy.huynh....@gmail.com> >> Cc: user <u...@spark.incubator.apache.org> >> Date: 11/05/2014 01:13 PM >> Subject: Re: sparse x sparse matrix multiplication >> ------------------------------ >> >> >> >> You can use breeze for local sparse-sparse matrix multiplication and >> then define an RDD of sub-matrices >> >> RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) >> >> and then use join and aggregateByKey to implement this feature, which >> is the same as in MapReduce. >> >> -Xiangrui >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >> >
