GPU hardware support for SystemML bleeding edge.

Greetings, What are the hardware(hw) configuration that SystemML (an opensource project) needs to support? i.e., do we have support for all of these? - Kepler - Maxwell - Pascal - Volta Lets say, 1. we have tuned a kernel only for a specific hw configuration(eg. Maxwell) with `asm()` code, c

Factorization Machines addition to our library.

Hi all, we recently added factorization machines, core layer to our `nn` library, along with regression and classification scripts. What is great about it? -- 1. The factorization machines handles the cases where SVM fails i.e., where the information is so sparse. 2. I

UDF in expression support for DML

Hi all, The support for function calls in expression has been addressed by Matthias. Till now tmp = my_udf_func(x) out = tmp + 4 >From now on --- out = my_udf_func(x) + 4 Thanks, Janardhan

gpu dev environment on windows.

Hi, Anybody would volunteer to help me (would like to spare a 15 mins) setup gpu dev environment on my windows. 1. IDE: intellij, visual studio 2017 2. OS: windows 10 3. cuda: 8.0 4. cudnn: I downloaded the package, but doesn't know how to integrate with cuda. questions: 1. where to edit the c++

[TESTS] What are the best practices for dml script testing. Thanks.

Hi prithvirajsen, If a script contains the gradients, then we could compare them with numerical gradients, as we have done in the `grad_check.dml`, file. How can we place a test script with a main script (dml), so that we can be confident that the script will work. (just like junit test for java

[review request] for the bayesian optimization support

Hi all, this is in reference to [systemml-979], after 4 months of going through the design preliminary implementation of each component. Now, we need to take up the review of the each component one by one and can be merged after LGTM. Please find that, we've done some modification to syntax, bui

Can BITWISE_XOR be added. Thanks.

Hi all, I am using the following line code operation ` x_tmp = bitwise_xor( v_index, as.scalar(V[v_index,]) ) ` Can this `bitwise_xor` be added?. I will try to take up the task, if there it fits into dml. Thanks, Janardhan

Re: [DISCUSS] Roadmap SystemML 1.1 and beyond

Hi all, my 0.02$ I am working on one by one. Please add to the above list.. 0. Algorithms * Factorization machines, with regression & classification capabities with the help of nn layers.[ 1437] * A test suite for the nn optimization, with well known optimization test functions. [1974] 1. Deep L

add. of matrices of different dim. (the bias term)

Hi Matthias, When adding the biases... For example., * Inputs: * - X: Inputs, of shape (N, D). * - W: Weights, of shape (D, M). * - b: Biases, of shape (1, M). * * Outputs: * - out: Outputs, of shape (N, M). */ out = (X %*% W) + b; # This script works out = b + (X %*

Re: dev environment on windows

s Boehm wrote: > > > I'm using eclipse on win and simply defined two maven run configurations > > for > > > > (1) the default build (base=${workspace_loc:/systemml}, goals=package), > > and > > (2) the distribution build ((base=${workspace_loc:/systemml},

dev environment on windows

Hi all, 1. How to build systemml on windows, as there is a backslash problem. 2. When build systemml on intellij idea, it has the compilation error with caffe2dml class. Thanks.

Distribution functions such as gamma, weibull etc.

Hi, I am wondering, whether we have builtin functions for distributions such as weibull, laplace, gamma. (looking at some top level algorithms, says that we do have!). Would like to use in the following case: ``` lp = -gamma(k) - k*log(t) + (k-1)*lx - x/t; ``` Thanks, Janardhan

My life made easier, now!

Hi all, After 5 months of struggle for installation of Apache Systemml ( After, more than 150 failed installations...), I finally, got this message "Welcome to Apache SystemML!". ​ Now, I am not going to bother our Jenkins for testing my patches. Thanks everyone, Janardhan

CI test environment configuration

Hi all, Any body knows how to set the TRAVIS CI configuration for test 1. spark backend, for executing a script like this (spark-submit SystemML.jar -f nn/test/run_tests.dml) 2. hadoop I have some success in building for the maven script here ( https://github.c

Re: Regarding `enable remote hyperparameter tuning`[BLOCKER issue]. Thanks.

are wrote: > Hi Janardhan, > > I am available anytime on thursday or friday this week works for me. I > would recommend sending an agenda before scheduling the meeting. > > Thanks, > > Niketan. > > - Original message - > From: Janardhan Pulivarthi > To:

Regarding `enable remote hyperparameter tuning`[BLOCKER issue]. Thanks.

@niketan - I don't have a time preference, please give me any time (or) date for meeting at your convenience. Thanks. Hi Mike, This issue [https://issues.apache.org/jira/browse/SYSTEMML-1159] has been marked as a blocker. I've gone through the reference paper you have attached there. In this pap

Minor script changes for SVM with `MLContext`, `spark_submit` etc.

Hi Matthias, Based on your comment here[ https://github.com/apache/systemml/pull/529#issuecomment-316253791], I've updated the SVM scripts(I believe). Can you please have a look at whether the changeset[ https://github.com/apache/systemml/pull/673/files] needs to be updated. :) Thanks, Janardhan

[Question] SVM & GLM test with MLContext

Hi Deron, I would like to resolve the testing of algorithms through `MLContext` in 2 phases. 1st phase - For the time being I am try to test the input/output behaviour of `Algorithms` through `MLContext`. 2nd phase - Here, with the help of domain expert I try to validate the output matrices or s

file location for `import org.apache.sysml.parser.dml.DmlParser.WhileStatementContext`

Hi, Can somebody point me out the `DmlParser` class? Thanks.

Re: [QUESTION] XOR operations in SystemML. Thanks.

our AND, OR, NOT, (and XOR) operations to matrices, because > these are currently only supported over scalars. It would get you in touch > with the parser validation, compilation, and runtime of rather simple > operations. > > Regards, > Matthias > > On Mon, Sep 4, 2017 at 8:57 PM

Re: file location for `import org.apache.sysml.parser.dml.DmlParser.WhileStatementContext`

DmlListener.java, > DmlParser.java ---> parse foo.dml > > On Thu, Sep 21, 2017 at 9:01 AM, Janardhan Pulivarthi < > janardhan.pulivar...@gmail.com> wrote: > > > Hi, > > > > Can somebody point me out the `DmlParser` class?. where this file is > being > > imported from `import > > org.apache.sysml.parser.dml.DmlParser.WhileStatementContext` > > > > Thanks. > > >

file location for `import org.apache.sysml.parser.dml.DmlParser.WhileStatementContext`

Hi, Can somebody point me out the `DmlParser` class?. where this file is being imported from `import org.apache.sysml.parser.dml.DmlParser.WhileStatementContext` Thanks.

Re: [QUESTION] about initialization of outputs , depends on if-else execution. Thanks.

g? > > Thanks > > Niketan > > > On Sep 5, 2017, at 9:32 AM, Janardhan Pulivarthi > wrote: > > > > Hi, > > > > 1. *Parser Error:*This is one problem encountered during testing of GLM > > scripts. I have fixed this problem by i

Recommendation use case for Factorization Machines

Hi all, I would like to add a notebook example to showcase scalability of factorization machines (fm), just like the ALS recommendation system with netflix dataset, https://github.com/apache/systemml/blob/master/samples/jupyter-notebooks/ALS_python_demo.ipynb But, would like to implement other ki

Re: [QUESTION] about initialization of outputs , depends on if-else execution. Thanks.

Hi Niketan, 1. Please correct me here, -> https://github.com/apache/ systemml/pull/651/files#diff-ef5e4f92ced39a334cdce976800ace26R139 2. please refer this comment -> https://github.com/apache/ systemml/pull/651#issuecomment-326902771 Thank you, Janardhan

[QUESTION] about initialization of outputs , depends on if-else execution. Thanks.

Hi, 1. *Parser Error:*This is one problem encountered during testing of GLM scripts. I have fixed this problem by initializing before the script, but can we do it the other way? @ https://github.com/apache/systemml/blob/1cbfdef3fad5235b692c372ec94bcade7f1b9ff9/scripts/algorithms/GLM-predict.dml#L

Re: [QUESTION] XOR operations in SystemML. Thanks.

> Matthias > > On Mon, Sep 4, 2017 at 7:55 AM, Janardhan Pulivarthi < > janardhan.pulivar...@gmail.com> wrote: > > > Hi all, [XOR symbol] > > > > Now, I gave a sample try for the XOR operator, with caret ` ^ ` symbol. > > But, this have been re

Re: Bayesian optimizer support for SystemML.

ic dataset record. Each heuristic targets > certain situations that arise in some records. Not sure how to take > advantage of this. > > Thanks, > Sasha > > > > From:Janardhan Pulivarthi > To:Alexandre V Evfimievski , npan...@us.ibm.com, > dev@sy

Re: [QUESTION] XOR operations in SystemML. Thanks.

cumulative operations such as cumsum, cumprod, cummax, to arbitrary cell > computations and aggregation functions, which would be useful for quite a > number of applications. > > Regards, > Matthias > > On Thu, Aug 31, 2017 at 5:59 AM, Janardhan Pulivarthi < > janardhan.

Re: Memory estimates equal to zero

Hi Nantia, Yes, even the negative estimates happened sometimes, as follows (although, this was fixed) : ``` DEBUG opt.Optimizer: --- RULEBASED OPTIMIZER --- DEBUG opt.Optimizer: RULEBASED OPT: Optimize w/ max_mem=12743MB/525MB/525MB, max_k=16/144/144). WARN opt.CostEstimator: Cannot get memor

[QUESTION] XOR operations in SystemML. Thanks.

Hi, The following is an equation (2.4) from the algorithm for the generation of sobol sequences. The authors of the paper have utilized the bitwise operations of C++ to calculate this efficiently. *Now, the question is:* Can we do this at script level (in dml) or we should do it in the `java` its

Gentle ping for help on all my PRs. Thanks.

Dear committers, I am feeling that my contributions are not in an ordered way. So, I am listing them here. And, also listed the help required from volunteers. 1. [SYSTEMML-1437] - factorization machines [*in progress*] - till now, I have implemen

Re: Numerical accuracy of DML.

; it helps. > > Regards, > Matthias > > > On Fri, Aug 18, 2017 at 11:39 PM, Janardhan Pulivarthi < > janardhan.pulivar...@gmail.com> wrote: > > > Dear committers, > > > > May I know the numerical accuracy of dml at present, and are you planning > > t

Numerical accuracy of DML.

Dear committers, May I know the numerical accuracy of dml at present, and are you planning to increase it. It seems for comparison operators we have depended upon java numerical floating point accuracy. Thank you very much, Janardhan

Re: svd( ) implementation

x. Please have a look at equation 1.3 in this paper: > > > > http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.127.115&rank=1 > > > > I think this is worth trying out. The distributed QR is already > > implemented in SystemlML, so it may quick to try out. > >

Re: svd( ) implementation

tored. > > (One of the other contributors can clarify, if I've missed something or > have said something wrong). > > Given this context, can you please think about whether missing sparse > matrix support is still a problem? > > > -Nakul > > > > > > >

Re: svd( ) implementation

t; > > > > > Maybe we could try another algorithm for scalable svd() : > > Sure. But before you do that, it may be worth our while to understand what > is exactly misleading about the paper that Imran talks about. > > > > > > -Nakul > > > > > > >

Re: Bayesian optimizer support for SystemML.

:41 PM, Janardhan Pulivarthi < janardhan.pulivar...@gmail.com> wrote: > Dear committers, > > We will be planning to add bayesian optimizer support for both the ML and > Deep learning tasks for the SystemML. Relevant jira link: > https://issues.apache.org/jira/browse/SYSTEMML-979

Re: Update Spark Configuration to improve SystemML performance

Hi Himanshu! We feel great you are here. SystemML is a very friendly community. To get started. 1. see the components of the SystemML, here in this link: https://issues.apache.org/jira/projects/SYSTEMML?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page 2. Through this link you

Bayesian optimizer support for SystemML.

Dear committers, We will be planning to add bayesian optimizer support for both the ML and Deep learning tasks for the SystemML. Relevant jira link: https://issues.apache.org/jira/browse/SYSTEMML-979 The following is a simple outline of how we are going implement it. Please feel free to make any

Re: about performance statistics of PCA.dml

ver-memory 10g > SystemML.jar -f genRandData4PCA.dml -nvargs R=1 C=1000 F=binary > OUT=pcaData.mtx > ~/spark-2.1.0-bin-hadoop2.7/bin/spark-submit --driver-memory 10g > SystemML.jar -f PCA.dml -stats 30 -nvargs INPUT=pcaData.mtx > OUTPUT=pca-1000x1000-model PROJDATA=1 CENTER=1 SCALE=1 >

about performance statistics of PCA.dml

Hi Mike, I'd like to know how much expensive this critical code is C = (t(A) %*% A)/(N-1) - (N/(N-1))*t(mu) %*% mu; (at https://github.com/apache/systemml/blob/master/scripts/algorithms/PCA.dml#L81) in the SPARK setting given 1. 60Kx700 input for A 2. For a datasize of 28 MB with 100 con

svd( ) implementation

Hi Nakul, Can you help me understand how gpu implementations help scale it. Whether the user always have GPUs in use when using this fn or is it an optional feature. The problem with implementing the scalable svd() in dml is due to the spark backend issues, otherwise there is no problem scaling w

Re: Apache SystemML 1.0 Release build

The verification of SVM, GLM, Logistic Regression and Survival Analysis scripts with the ML Context is pending. SYSTEMML-1648 > 1648, 1649, 1651, 1653 - Janardhan On Sun, Jul 16, 2017 at 6:04 AM, Mike Dusenberry wrote: > The deep learning work (nn library, Caffe2DML, GPUs, and general engine >

Re: Error while Executing code inSystemML

. > > > But thanks for guiding us that it's spark issue, rather than systemML > issue. > > > Thank you! Regards, > > Arijit > > > From: Janardhan Pulivarthi > Sent: Sunday, July 16, 2017 10:15:12 AM > To: dev@systemml.a

Re: Error while Executing code inSystemML

Hi Arijit, Can you please send the exact code (the .dml file), you have used and the dataset details and sizes?. This is problem has something to do with the Apache Spark. Thanks, Janardhan On Sat, Jul 15, 2017 at 3:46 PM, arijit chakraborty wrote: > Hi, > > > I'm suddenly getting this error w

Re: On the need for Parameter Server. ( A Model Parallel Construct )

10:16 PM, Janardhan Pulivarthi < > janardhan.pulivar...@gmail.com> wrote: > > Dear committers, > > Implementation/Integration of existing parameter server for the execution > of algorithms in a distributed fashion both for the machine learning and > deep learning. >

On the need for Parameter Server. ( A Model Parallel Construct )

Dear committers, Implementation/Integration of existing parameter server for the execution of algorithms in a distributed fashion both for the machine learning and deep learning. The following document covers a bit about whether we need one or not ?. My name is Janardhan, currently working on [S

Re: List of SystemML research papers?

Hello @deroneriksson & other developers, My name's Janardhan, (@p-janardhan) currently working on SystemML-1437. So far, I found the following research papers & patents (of 413 google scholar search results.). This list includes research by IBM teams and the before work by other researchers. Per