Hi Niketan, Thank you for telling me many things. I understand the plan of SystemML to support REPL.
> Like Matthias, I would recommend that implementing an interpreter for DML might not be a good task to explore SystemML. I will start contributing SystemML from small tasks. > We do have a "mini-interpreter" in the form of a debugger and it might be a good place to explore before implementing an interpreter. This is so great. I will try to use it and read the source code. Thank you again for your giving me the overview and some references. Thanks, - Tatsuya -- From: Niketan Pansare [mailto:[email protected]] Sent: Thursday, December 03, 2015 2:30 AM To: [email protected] Subject: RE: How to contribute SystemML Hi Tatsuya, Thanks for your interest in the SystemML project. Just so that we are on same page, there are two interpretation of REPL with respect to SystemML: 1. Using SystemML in Spark REPL. - Scala Spark-shell: http://apache.github.io/incubator-systemml/mlcontext-programming-guide.html. This also involves the integration with MLPipeline (https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/ ibm/bi/dml/api/ml/LogisticRegression.java) APIs. - PySpark shell (using SystemML.py) - Jupyter (using SystemML.py) 2. Having a "DML kernel/interpreter". - A DML kernel for Jupyter (will require a command-line interpreter) - A command-line interpreter where data scientists can type DML commands. Like Matthias, I would recommend that implementing an interpreter for DML might not be a good task to explore SystemML. Still, I will try to give you an overview and please pardon me for hand-waving a bit in below paragraph for sake of simplicity. We do have a "mini-interpreter" in the form of a debugger and it might be a good place to explore before implementing an interpreter. Please note: the debugger works at level of instruction, not at the level of DML statement. Also, the debugger takes as input a DML script and hence SystemML's optimizer has full scope for optimization. However, to allow for tasks such as "step into", we disable certain optimizations and have a special level for the debugger (O5_DEBUG_MODE). For example: algrebraic simplification, interprocedural analysis, branch removal, dynamic recompilations, sum product rewrites, etc are disabled. Browsing through the debugger code should give you a fair idea about the machinery required in SystemML for implementing a full-fledged interpreter (such as runtime instructions, symbol table, instructions-to-DML mapping, piggybacking, etc). Then one must step back and look at the high-level design decisions as well, such as lazy evaluation. To get started with the debugger, please look at http://apache.github.io/incubator-systemml/debugger-guide.html and also in the com.ibm.bi.dml.debug package. Please note the URL might change soon when we change our package names to org.apache.sysml. Reference: https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/i bm/bi/dml/api/MLContext.java https://github.com/apache/incubator-systemml/blob/master/src/main/java/com/i bm/bi/dml/api/python/SystemML.py https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages https://github.com/apache/incubator-systemml/tree/master/src/main/java/com/i bm/bi/dml/debug Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar "Tatsuya Nishiyama" ---12/02/2015 01:06:33 AM---Hi Matthias, Thanks for your reply. From: "Tatsuya Nishiyama" <[email protected]> To: <[email protected]> Date: 12/02/2015 01:06 AM Subject: RE: How to contribute SystemML ________________________________________ Hi Matthias, Thanks for your reply. > Such a feature would, however, require substantial discussions, especially with regard to optimization scope etc and hence might not be a good starting task. > How about we select some smaller tasks on our existing APIs and once you feel comfortable get started on a larger feature like that? OK, I understand it. I will try to find such smaller tasks, and tackle it. If you already know any started tasks , please tell me. Best regards, - Tatsuya -- From: Matthias Boehm [mailto:[email protected]] Sent: Wednesday, December 02, 2015 4:47 PM To: [email protected] Subject: Re: How to contribute SystemML Hi Tatsuya, thanks for your interest, we'd love to help you get started. Although we do have various APIs, including MLContext that allows you to invoke DML scripts from Spark's interactive shell, we don't have an actual REPL interface yet. Niketan built an initial prototype of a related API. @Niketan: Do you want to comment on that? Such a feature would, however, require substantial discussions, especially with regard to optimization scope etc and hence might not be a good starting task. How about we select some smaller tasks on our existing APIs and once you feel comfortable get started on a larger feature like that? Regards, Matthias "Tatsuya Nishiyama" ---12/01/2015 10:50:41 PM---Hi, I'm interested in contributing SystemML. I've checked the documentation, and From: "Tatsuya Nishiyama" <[email protected]> To: <[email protected]> Date: 12/01/2015 10:50 PM Subject: How to contribute SystemML ________________________________________ Hi, I'm interested in contributing SystemML. I've checked the documentation, and it looks like SystemML doesn't have REPL for interactive execution. Is it correct? If SystemML doesn't have REPL, I would like to contribute it. BTW, JIRA looks to be unavailable for now: https://issues.apache.org/jira/browse/SYSTEMML How can I start contribution? Thanks, - Tatsuya
