Re: sparse x sparse matrix multiplication
thanks reza. i'm not familiar with the block matrix multiplication, but is it a good fit for very large dimension, but extremely sparse matrix? if not, what is your recommendation on implementing matrix multiplication in spark on very large dimension, but extremely sparse matrix? On Thu, Nov 6, 2014 at 5:50 PM, Reza Zadeh r...@databricks.com wrote: See this thread for examples of sparse matrix x sparse matrix: https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA We thought about providing matrix multiplies on CoordinateMatrix, however, the matrices have to be very dense for the overhead of having many little (i, j, value) objects to be worth it. For this reason, we are focused on doing block matrix multiplication first. The goal is version 1.3. Best, Reza On Wed, Nov 5, 2014 at 11:48 PM, Wei Tan w...@us.ibm.com wrote: I think Xiangrui's ALS code implement certain aspect of it. You may want to check it out. Best regards, Wei - Wei Tan, PhD Research Staff Member IBM T. J. Watson Research Center [image: Inactive hide details for Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplic]Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matri From: Xiangrui Meng men...@gmail.com To: Duy Huynh duy.huynh@gmail.com Cc: user u...@spark.incubator.apache.org Date: 11/05/2014 01:13 PM Subject: Re: sparse x sparse matrix multiplication -- You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matrices RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) and then use join and aggregateByKey to implement this feature, which is the same as in MapReduce. -Xiangrui - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
If you're have very large and very sparse matrix represented as (i, j, value) entries, then you can try the algorithms mentioned in the post https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA brought up earlier. Reza On Fri, Nov 7, 2014 at 8:31 AM, Duy Huynh duy.huynh@gmail.com wrote: thanks reza. i'm not familiar with the block matrix multiplication, but is it a good fit for very large dimension, but extremely sparse matrix? if not, what is your recommendation on implementing matrix multiplication in spark on very large dimension, but extremely sparse matrix? On Thu, Nov 6, 2014 at 5:50 PM, Reza Zadeh r...@databricks.com wrote: See this thread for examples of sparse matrix x sparse matrix: https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA We thought about providing matrix multiplies on CoordinateMatrix, however, the matrices have to be very dense for the overhead of having many little (i, j, value) objects to be worth it. For this reason, we are focused on doing block matrix multiplication first. The goal is version 1.3. Best, Reza On Wed, Nov 5, 2014 at 11:48 PM, Wei Tan w...@us.ibm.com wrote: I think Xiangrui's ALS code implement certain aspect of it. You may want to check it out. Best regards, Wei - Wei Tan, PhD Research Staff Member IBM T. J. Watson Research Center [image: Inactive hide details for Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplic]Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matri From: Xiangrui Meng men...@gmail.com To: Duy Huynh duy.huynh@gmail.com Cc: user u...@spark.incubator.apache.org Date: 11/05/2014 01:13 PM Subject: Re: sparse x sparse matrix multiplication -- You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matrices RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) and then use join and aggregateByKey to implement this feature, which is the same as in MapReduce. -Xiangrui - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
See this thread for examples of sparse matrix x sparse matrix: https://groups.google.com/forum/#!topic/spark-users/CGfEafqiTsA We thought about providing matrix multiplies on CoordinateMatrix, however, the matrices have to be very dense for the overhead of having many little (i, j, value) objects to be worth it. For this reason, we are focused on doing block matrix multiplication first. The goal is version 1.3. Best, Reza On Wed, Nov 5, 2014 at 11:48 PM, Wei Tan w...@us.ibm.com wrote: I think Xiangrui's ALS code implement certain aspect of it. You may want to check it out. Best regards, Wei - Wei Tan, PhD Research Staff Member IBM T. J. Watson Research Center [image: Inactive hide details for Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplic]Xiangrui Meng ---11/05/2014 01:13:40 PM---You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matri From: Xiangrui Meng men...@gmail.com To: Duy Huynh duy.huynh@gmail.com Cc: user u...@spark.incubator.apache.org Date: 11/05/2014 01:13 PM Subject: Re: sparse x sparse matrix multiplication -- You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matrices RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) and then use join and aggregateByKey to implement this feature, which is the same as in MapReduce. -Xiangrui - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
local matrix-matrix multiplication or distributed? On Tue, Nov 4, 2014 at 11:58 PM, ll duy.huynh@gmail.com wrote: what is the best way to implement a sparse x sparse matrix multiplication with spark? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparse-x-sparse-matrix-multiplication-tp18163.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
distributed. something like CordinatedMatrix.multiply(CoordinatedMatrix). thanks xiangrui! On Wed, Nov 5, 2014 at 4:24 AM, Xiangrui Meng men...@gmail.com wrote: local matrix-matrix multiplication or distributed? On Tue, Nov 4, 2014 at 11:58 PM, ll duy.huynh@gmail.com wrote: what is the best way to implement a sparse x sparse matrix multiplication with spark? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparse-x-sparse-matrix-multiplication-tp18163.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
in case, this won't be available anytime soon with spark. what would be a good way to implement this multiplication feature in spark? On Wed, Nov 5, 2014 at 4:59 AM, Duy Huynh duy.huynh@gmail.com wrote: distributed. something like CordinatedMatrix.multiply(CoordinatedMatrix). thanks xiangrui! On Wed, Nov 5, 2014 at 4:24 AM, Xiangrui Meng men...@gmail.com wrote: local matrix-matrix multiplication or distributed? On Tue, Nov 4, 2014 at 11:58 PM, ll duy.huynh@gmail.com wrote: what is the best way to implement a sparse x sparse matrix multiplication with spark? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparse-x-sparse-matrix-multiplication-tp18163.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matrices RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) and then use join and aggregateByKey to implement this feature, which is the same as in MapReduce. -Xiangrui - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparse x sparse matrix multiplication
I think Xiangrui's ALS code implement certain aspect of it. You may want to check it out. Best regards, Wei - Wei Tan, PhD Research Staff Member IBM T. J. Watson Research Center From: Xiangrui Meng men...@gmail.com To: Duy Huynh duy.huynh@gmail.com Cc: user u...@spark.incubator.apache.org Date: 11/05/2014 01:13 PM Subject:Re: sparse x sparse matrix multiplication You can use breeze for local sparse-sparse matrix multiplication and then define an RDD of sub-matrices RDD[(Int, Int, CSCMatrix[Double])] (blockRowId, blockColId, sub-matrix) and then use join and aggregateByKey to implement this feature, which is the same as in MapReduce. -Xiangrui - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
