Hi All, Sorry for my late reply!
Yu Ishikawa,Thanks for your interests in Saury project. You are welcomed to try that out. If you have questions about that, please email me. We are keeping improving performance/adding features for the project. Xiangrui, thanks for your encouragement. If you have any problems with my CSDN reports, please feel free to contact me. We had some design for Saury on our lab's private JIRA which is in Chinese. I will translate into English then share it to you these days. Acutally, I also have surveyed the related algorithms/systems before we started the Saury project. The survey is attached in this email, not on CSDN report. We also had considered the 2.5D algorithm for reducing communication. However, at that time, MLlib did not have a distributed block matrix representation. So, we decided to firstly implement the distributed matrix multiplication on the IndexRowMatrix as time is limited for the Summer Code project. Also, as far as we know, nobody had tried that at that time. Actually, adopting 2.5D algorithm to reduce network communication is on our roadmap. We are also planning to do that in the next days. Best, Rong 2014-09-08 15:31 GMT+08:00 Xiangrui Meng <men...@gmail.com>: > Sorry for my late reply! I'm also very interested in the > implementation of distributed matrix multiplication. As Shivaram > mentioned, the communication is the concern here. But maybe we can > start with a reasonable implementation and then iterate on its > performance. It would be great if eventually we can implement an > algorithm close to the 2.5D algorithm > (http://www.netlib.org/lapack/lawnspdf/lawn248.pdf). > > I created two JIRAs for this topic: > > 1. Distributed block matrix: > https://issues.apache.org/jira/browse/SPARK-3434 > 2. Distributed matrix multiplication: > https://issues.apache.org/jira/browse/SPARK-3435 > > We can move our discussion there. > > Rong, I'm really happy to see the Saury project. It would be great if > you can share your design and experience (maybe on the JIRA page so it > is easier to track). I will read the reports on CSDN and ping you if I > ran into problems. Thanks! > > Best, > Xiangrui > > On Sat, Sep 6, 2014 at 1:28 AM, Yu Ishikawa > <yuu.ishikawa+sp...@gmail.com> wrote: > > Hi Rong, > > > > Great job! Thank you for let me know your work. > > I will read the source code of saury later. > > > > Although AMPLab is working to implement them, would you like to merge it > > into Spark? > > > > Best, > > > > -- Yu Ishikawa > > > > > > > > > > -- > > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/mllib-Add-multiplying-large-scale-matrices-tp8291p8310.html > > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > > -- ------------------ Rong Gu Department of Computer Science and Technology State Key Laboratory for Novel Software Technology Nanjing University Phone: +86 15850682791 Email: gurongwal...@gmail.com Homepage: http://pasa-bigdata.nju.edu.cn/people/ronggu/
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
