Hi
Can we please have the discussion in other thread on R4ML integration with SystemML :) sincerely Alok ________________________________ From: Niketan Pansare <npan...@us.ibm.com> Sent: Friday, September 22, 2017 10:51 AM To: dev@systemml.apache.org Subject: Re: [DISCUSS] R-Interface to SystemML Hi Brendan/Alok, Thanks for taking time to answer our questions and also for considering to help SystemML community. I have couple more questions: 1. In case there is inconsistency, do you (as R4ML developers) feel comfortable changing R4ML interface to be compatible with our other APIs ? May be you can go over the below two links and imagine adding a corresponding R tab: - MLContext Programming guide: http://apache.github.io/systemml/spark-mlcontext-programming-guide apache.github.io<http://apache.github.io/systemml/spark-mlcontext-programming-guide> apache.github.io Spark MLContext Programming Guide. Overview; Spark Shell Example. Start Spark Shell with SystemML; Create MLContext; Hello World; LeNet on MNIST Example; DataFrame ... - Algorithm wrappers: http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression 2. Classification - GitHub Pages<http://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression> apache.github.io SystemML Algorithms Reference 2. Classification 2.1. Multinomial Logistic Regression Description. The MultiLogReg.dml script performs both binomial and multinomial ... 2. Other than providing R interface to SystemML as the above APIs, what additional features/code R4ML plans to add in SystemML ? Just like we want the R API to be functionally complete with our Python and Scala API, we want Python and Scala APIs to be functionally complete with the R API. So a discussion on supporting the additional features in Python and Scala APIs is required :) Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar [http://researcher.watson.ibm.com/researcher/photos/3531.jpg]<http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar> Niketan Pansare - IBM<http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar> researcher.watson.ibm.com Niketan Pansare is a Senior Software Engineer at IBM Research Almaden, where he works on advanced information management systems that include analytics, distributed ... [Inactive hide details for Deron Eriksson ---09/22/2017 08:59:58 AM---Hi Brendan, Thank you for the detailed description. At a h]Deron Eriksson ---09/22/2017 08:59:58 AM---Hi Brendan, Thank you for the detailed description. At a high level that sounds From: Deron Eriksson <deroneriks...@gmail.com> To: dev@systemml.apache.org Date: 09/22/2017 08:59 AM Subject: Re: [DISCUSS] R-Interface to SystemML ________________________________ Hi Brendan, Thank you for the detailed description. At a high level that sounds feasible. Also, offering to help maintain the R codebase is extremely helpful. Please let us know if you have any questions so that we can assist you and Alok in your efforts, since as I said I think an R interface to SystemML makes a lot of sense. Deron On Thu, Sep 21, 2017 at 4:36 PM, Brendan Dwyer <brendan.dw...@ibm.com> wrote: > Sorry for not responding sooner. I had some issues with my email client. > > > > I will do my best to address as many as the points that have been raised > as I can. Hopefully Alok will be able to jump in as well once he resolves > his email issues. > > > > - I would be happy to help maintain R4ML in SystemML and I’m sure Alok > would too. > > - R4ML does allow arbitrary DML script to be executed via the > `sysml.execute()` function. > > - I think we would like to merge the entire R4ML github repository into > SystemML. We could do this the same way SparkR was merged into Spark ( > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e= [https://avatars3.githubusercontent.com/u/47359?v=4&s=400]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e=> apache/spark<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_tree_master_R&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=-AsxMGFLlXCro0OSEJTWLLca2vPsQwTwBxAY8Bzp6kw&e=> urldefense.proofpoint.com spark - Mirror of Apache Spark ) > > - Currently the code is not ready to be merged into SystemML because we > are still on the old ML context. We have a PR in the works that will update > to the newest ML context. Once that happens we won’t need to duplicate the > DML scripts. > > - Documentation is generated automatically with the R package “roxygen”. > We would need to discuss how to incorporate this into the SystemML > documentation. Perhaps we could look to Spark/SparkR for ideas. > > - Test are done using the R testthat package. I can work with Alan to get > that integrated into the systemml Jenkins server > > Sent from IBM Verse > > > Matthias Boehm --- Re: [DISCUSS] R-Interface to SystemML --- > From:"Matthias Boehm" <mboe...@googlemail.com>To: > dev@systemml.apache.org, deron@apache.orgDate:Thu, Sep 21, 2017 4:13 > PMSubject:Re: [DISCUSS] R-Interface to SystemML > > I pretty much agree with Niketan and Deron. In general, it would be > usefulto provide an R API as well. However, I'm a bit concerned for two > reasons:* Looking over the github repo, apparently R4ML is not under > activedevelopment/maintenance anymore (last commit Jul 20). So who would > bewilling to maintain and extend it?* Providing wrappers for our algorithm > scripts would be just a startbecause it hides our core value proposition of > custom large-scale ML.Hence, we would also need an MLContext equivalent > that allows to executearbitrary DML scripts or R functions. Is there > already a tentative designof such an API and if not, who would like to take > it over?Regards,MatthiasOn Thu, Sep 21, 2017 at 3:43 PM, Deron Eriksson < > deroneriks...@gmail.com>wrote:> I agree with Niketan. An R interface > definitely makes sense for SystemML.> DML itself is based on R, so it's > surprising we have Java/Scala/Python> interfaces to SystemML but we don't > have an R interface.>> Perhaps R4ML committers could supply a little more > info? For instance:> 1) Would they like to merge R4ML code into the main > SystemML project> itself? (Currently we have no modules.)> 2) What would > they like to merge?> 3) If so, how do they propose to do so?> 4) Who will > do the majority of the work to add R4ML code to SystemML? Or> who would > like to volunteer to do this?> 5) Who will maintain the contributed code? > Or who would like to volunteer> to do this?> 6) Documentation is needed > (fit in SystemML documentation framework).> 7) Testing is needed (fit into > SystemML testing framework).> 8) How is this packaged?>> From a technology > standpoint, I think an R interface totally makes sense.> As for a minor > criticism (which I apply to other parts of SystemML too), I> see script > wrappers at https://urldefense.proofpoint.com/v2/url?u=https-3A__github. > com_SparkTC_r4ml_tree_master_R4ML_R&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=_qGcWSDggH-K3E_ > wTPXBBjOClp2Jub4KtvBgfeW1kbU&e= .> This tightly binds the existing DML > scripts to R, which means DML> input/output modifications could potentially > require modifications to R> code.>> Deron>>>> On Thu, Sep 21, 2017 at 11:00 > AM, Niketan Pansare <npan...@us.ibm.com>> wrote:>> > Janardhan: I believe > this is the R4ML repo: > https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e= > . > com/v2/url?u=https-3A__github.com_SparkTC_&d=DwIBaQ&c=jf_ > iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1Xd > JBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4& > s=vj-Ogu1F2fnli1JwDjU1-S-Pauc7SSmSNG0g5sjgwUk&e= > > r4ml . Arvind: > please correct me if I am wrong.> >> > Overall, having a R interface for > SystemML is an awesome idea. Since I am> > not an R4ML expert, may be R4ML > committers can comment on how they> envision> > "two code streams to work > together".> >> > Also, comparing the features of R4ML with that of our > Python APIs will be> > useful as it might make a stronger case for R4ML.> > >> > As an FYI, here are different ways Python users can use SystemML:> > - > Using MLContext to invoke DML script > (https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=W-mdQ-T1cM2C_v9AGp-9_8RfDTP54iwaFGrXNwx-B1c&e= > . > proofpoint.com/v2/url?u=http-3A__apache.github.io_&d= > DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=oU0Hd6PZBeEjeAVlZmb0utBefJN1Xd > JBEF8eiZhdECk&m=kVcfoxaRYrbaD_gb_hA_jn4bjiKe_gcUpc6mF1xbEd4& > s=7ng5kh3VaiQ045UZ-8xQtNmZS1KvD7LQ2e5gCM7aRmQ&e= > > > systemml/beginners-guide-python#invoking-dmlpydml-> > scripts-using-mlcontext> > and > https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e= > . > com/v2/url?u=http-3A__apache.github.io_systemml_spark- > 2Dmlcontext-2D&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=PuQtoa13gaSfRk95Bl5jvZnBeEhyI_ > Rae7ZB8-H5970&e= > > programming-guide.html)> > - Python algorithms > wrappers (https://urldefense.proofpoint.com/v2/url?u=http- > 3A__apache.github.io_&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=7ng5kh3VaiQ045UZ- > 8xQtNmZS1KvD7LQ2e5gCM7aRmQ&e= > > systemml/beginners-guide- > python#invoke-systemmls-algorithms)> > - (not important for R4ML > discussion): Python DSL (> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__urldefense.proofpoint&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=zZv3hzGXYPsASGxiAIkKxrx6RwULOnp_PhMvzsrnwGs&s=s0malh3u3v57rOz_0fpFab5WlWAJ735Cb6cGOWtiS1w&e= > . > com/v2/url?u=http-3A__apache.github.io_systemml_beginners- > 2Dguide-2D&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=7W3j9VgE_HckySvu_5- > NDjtmgHTcdtjxB_1syxRMNRY&e= > python#matrix-operations)> >> > Thanks,> >> > > Niketan Pansare> > IBM Almaden Research Center> > E-mail: npansar At > us.ibm.com> > http://researcher.watson.ibm.com/researcher/view.php? > person=us-npansar> >> > [image: Inactive hide details for Janardhan > ---09/21/2017 04:44:02> AM---Hi> > Arvind, This is a great idea. One > question: the R4ML generat]Janardhan> > ---09/21/2017 04:44:02 AM---Hi > Arvind, This is a great idea. One> question:> > the R4ML generates any plan > like the SystemML with `D> >> > From: Janardhan <j...@protonmail.com>> > > To: Arvind Surve <ac...@yahoo.com.INVALID>, "dev@systemml.apache.org" <> > > dev@systemml.apache.org>> > Date: 09/21/2017 04:44 AM> > Subject: Re: > [DISCUSS] R-Interface to SystemML> > ------------------------------> >> > >> >> > Hi Arvind,> >> > This is a great idea. One question: the R4ML > generates any plan like the> > SystemML with `DML` Or with providing some > interface we leverage this> > feature ?. Community effort in the sense of > collaborative algorithm> > implementation.(?)> >> > Is this the Spark-R > repo ( https://urldefense.proofpoint.com/v2/url?u=https-3A__ > urldefense.proofpoint&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=IsTGjUhAgkp2oVtWd0RUsJcAcrSa0m > zr9AbpU4Q86zU&e= .> > com/v2/url?u=https-3A__github. > com_rstudio_sparklyr&d=DwIGaQ&> > > c=jf_iaSHvJObTbx-siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_> > > 6a6SaOkZrfiSrYZVby0&m=uxG7P-4VuICwg6yatnAEX5JBdZ-> > PSwyvQzq5gwX1GL0&s=6VRs_> > J7zXj9jZEavEP8iNvVfISAjDJeM8wFL2sBnb0g&e= ) > ?> >> > Thanks,> > Janardhan> >> > Sent with [ProtonMail](https:// > urldefense.proofpoint.com/v2/> > url?u=https-3A__protonmail. > com&d=DwIGaQ&c=jf_iaSHvJObTbx-> > > siA1ZOg&r=HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=> > > uxG7P-4VuICwg6yatnAEX5JBdZ-PSwyvQzq5gwX1GL0&s=khkGV3oXz1W5m_> > > ueQRuKWlNMVOXXCVhV_ytNCINjJWY&e= ) Secure Email.> >> > > -------- > Original Message --------> > > Subject: [DISCUSS] R-Interface to SystemML> > > > Local Time: September 20, 2017 12:50 PM> > > UTC Time: September 20, > 2017 4:50 PM> > > From: ac...@yahoo.com.INVALID> > > To: > dev@systemml.apache.org <dev@systemml.apache.org>> > >> > > Hi,> > > R4ML > is an open source project which provides a R interface to> > SystemML.Its a > bridge between SystemML and Spark-R.> > > Lets discuss here if and how we > can get two code streams work together> > to benefit development/community > effort.> > >> > > Arvind Surve | Spark Technology Center |> > https://urldefense.proofpoint.com/v2/url?u=https-3A__ > urldefense.proofpoint&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r= > oU0Hd6PZBeEjeAVlZmb0utBefJN1XdJBEF8eiZhdECk&m=kVcfoxaRYrbaD_ > gb_hA_jn4bjiKe_gcUpc6mF1xbEd4&s=IsTGjUhAgkp2oVtWd0RUsJcAcrSa0m > zr9AbpU4Q86zU&e= .> > > com/v2/url?u=http-3A__www.spark.tc_&d=DwIGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=> > > HzVC6v79boGYQrpc383_Kao_6a6SaOkZrfiSrYZVby0&m=uxG7P-> > 4VuICwg6yatnAEX5JBdZ-> > > PSwyvQzq5gwX1GL0&s=Yj8qfo7sjGBxX547UMGfLHRZFUxCAjZDTHWe8B7jLxI&e=> > >> >> >> > >
