Re: Is there any Sparse Matrix implementation in Spark/MLib?
GML is a fast, distributed, in-memory sparse (and dense) matrix libraries. It does not use RDDs for resilience. Instead we have examples that use Resilient X10 (which provides recovery of distributed control structures in case of node failure) and Hazelcast for stable storage. We are looking to benchmark with RDDs to compare overhead, and also looking to see how the same ideas could be realized on top of RDDs. On 2/28/15 7:25 PM, Joseph Bradley wrote: Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix The Spark 1.3 RC includes a new one: BlockMatrix. But since these are distributed, they are represented using RDDs, so they of course will not be as fast as computations on smaller, locally stored matrices. Joseph On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh riteshoneinamill...@gmail.com mailto:riteshoneinamill...@gmail.com wrote: try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com mailto:shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org mailto:vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Thanks Vijay, but the setup requirement for GML was not straightforward for me at all, so I put it aside for a while. best, /Shahab On Sun, Mar 1, 2015 at 9:34 AM, Vijay Saraswat vi...@saraswat.org wrote: GML is a fast, distributed, in-memory sparse (and dense) matrix libraries. It does not use RDDs for resilience. Instead we have examples that use Resilient X10 (which provides recovery of distributed control structures in case of node failure) and Hazelcast for stable storage. We are looking to benchmark with RDDs to compare overhead, and also looking to see how the same ideas could be realized on top of RDDs. On 2/28/15 7:25 PM, Joseph Bradley wrote: Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix The Spark 1.3 RC includes a new one: BlockMatrix. But since these are distributed, they are represented using RDDs, so they of course will not be as fast as computations on smaller, locally stored matrices. Joseph On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Thanks Josef for the comments, I think I need to do some benchmarking. best, /Shahab On Sun, Mar 1, 2015 at 1:25 AM, Joseph Bradley jos...@databricks.com wrote: Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix The Spark 1.3 RC includes a new one: BlockMatrix. But since these are distributed, they are represented using RDDs, so they of course will not be as fast as computations on smaller, locally stored matrices. Joseph On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global- matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Hi Shahab, There are actually a few distributed Matrix types which support sparse representations: RowMatrix, IndexedRowMatrix, and CoordinateMatrix. The documentation has a bit more info about the various uses: http://spark.apache.org/docs/latest/mllib-data-types.html#distributed-matrix The Spark 1.3 RC includes a new one: BlockMatrix. But since these are distributed, they are represented using RDDs, so they of course will not be as fast as computations on smaller, locally stored matrices. Joseph On Fri, Feb 27, 2015 at 4:39 AM, Ritesh Kumar Singh riteshoneinamill...@gmail.com wrote: try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global- matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Thanks, But do you know if access to Coordinated matrix elements is almost as fast as a normal matrix or it has access time similar to RDD ( relatively slow)? I am looking for some fast access sparse matrix data structure. On Friday, February 27, 2015, Peter Rudenko petro.rude...@gmail.com wrote: Yes, it's called Coordinated Matrix( http://spark.apache.org/docs/latest/mllib-data-types.html#coordinatematrix) you need to fill it with elemets of type MatrixEntry( (Long, Long, Double)) Thanks, Peter Rudenko On 2015-02-27 14:01, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab
Re: Is there any Sparse Matrix implementation in Spark/MLib?
try using breeze (scala linear algebra library) On Fri, Feb 27, 2015 at 5:56 PM, shahab shahab.mok...@gmail.com wrote: Thanks a lot Vijay, let me see how it performs. Best Shahab On Friday, February 27, 2015, Vijay Saraswat vi...@saraswat.org wrote: Available in GML -- http://x10-lang.org/x10-community/applications/global-matrix-library.html We are exploring how to make it available within Spark. Any ideas would be much appreciated. On 2/27/15 7:01 AM, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Is there any Sparse Matrix implementation in Spark/MLib?
Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab
Re: Is there any Sparse Matrix implementation in Spark/MLib?
Yes, it's called Coordinated Matrix(http://spark.apache.org/docs/latest/mllib-data-types.html#coordinatematrix) you need to fill it with elemets of type MatrixEntry( (Long, Long, Double)) Thanks, Peter Rudenko On 2015-02-27 14:01, shahab wrote: Hi, I just wonder if there is any Sparse Matrix implementation available in Spark, so it can be used in spark application? best, /Shahab
