[jira] [Closed] (SYSTEMML-2525) Add RESTful prediction service to deploy SystemML models

2019-03-29 Thread Niketan Pansare (JIRA)


 [ 
https://issues.apache.org/jira/browse/SYSTEMML-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare closed SYSTEMML-2525.
-
   Resolution: Fixed
Fix Version/s: SystemML 1.3

Closed by the commit 
[https://github.com/apache/systemml/commit/863c9d5cb1752b0e50140f5c6673968b57c2f9d0]

> Add RESTful prediction service to deploy SystemML models
> 
>
> Key: SYSTEMML-2525
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2525
> Project: SystemML
>  Issue Type: Project
>Reporter: Niketan Pansare
>Priority: Major
> Fix For: SystemML 1.3
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2525) Add RESTful prediction service to deploy SystemML models

2019-03-27 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2525:
-

 Summary: Add RESTful prediction service to deploy SystemML models
 Key: SYSTEMML-2525
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2525
 Project: SystemML
  Issue Type: Project
Reporter: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SYSTEMML-2514) Hadoop gives java.lang.UnsatisfiedLinkError

2019-03-20 Thread Niketan Pansare (JIRA)


[ 
https://issues.apache.org/jira/browse/SYSTEMML-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797676#comment-16797676
 ] 

Niketan Pansare commented on SYSTEMML-2514:
---

I usually use SystemML with Spark. To verify whether Spark is correctly 
installed, please follow the below steps:

 

1. Java 
===
The Java version should be > 1.8.

> java -version

Set JAVA_HOME environment variable and include %JAVA_HOME%\bin in the 
environment variable PATH

> ls "%JAVA_HOME%"

2. Spark

Download and extract Spark from https://spark.apache.org/downloads.html,

> tar -xzf spark-2.1.0-bin-hadoop2.7.tgz

and set environment variable SPARK_HOME to point to the extracted directory.
 
Next step, install winutils:

- Download winutils.exe from 
http://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
 
- Place it in c:\winutils\bin
- Set environment variable HADOOP_HOME to point to c:\winutils
- Add c:\winutils\bin to the environment variable PATH.
- Finally, modify permission of hive directory that will be used by spark

> winutils.exe chmod 777 /tmp/hive

Finally, check if Spark is correctly installed:

> %SPARK_HOME%\bin\spark-shell
 > %SPARK_HOME%\bin\pyspark

 

> Hadoop gives java.lang.UnsatisfiedLinkError
> ---
>
> Key: SYSTEMML-2514
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2514
> Project: SystemML
>  Issue Type: Sub-task
>  Components: Build
>Reporter: Janardhan
>Priority: Minor
> Attachments: hadoop_on_win.pdf
>
>
>  
> {code:java}
> java.lang.UnsatisfiedLinkError: 
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native 
> Method)
> at 
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
> at 
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:398)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1458)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1514)
> at 
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:530)
> at 
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:587)
> at 
> org.apache.sysml.test.integration.functions.blocks.VariableTest.testVariablePassing1(VariableTest.java:80)
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2523) Update SystemML to Support Spark 2.3.0

2019-03-19 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2523:
-

 Summary: Update SystemML to Support Spark 2.3.0
 Key: SYSTEMML-2523
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2523
 Project: SystemML
  Issue Type: Project
Reporter: Niketan Pansare


Spark 2.3 (released on February 28, 2018) has updated Antlr version, which 
throws warning every time we invoke SystemML. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (SYSTEMML-2505) Support generation of DML script using Keras2DML and Caffe2DML

2018-12-08 Thread Niketan Pansare (JIRA)


 [ 
https://issues.apache.org/jira/browse/SYSTEMML-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-2505.
---
Resolution: Fixed

Fixed by 
https://github.com/apache/systemml/commit/7019f3bc805aaae67ef32e281cf99e26cbd26b29

> Support generation of DML script using Keras2DML and Caffe2DML
> --
>
> Key: SYSTEMML-2505
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2505
> Project: SystemML
>  Issue Type: Test
>  Components: APIs
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>Priority: Major
> Fix For: SystemML 1.3
>
>
>  
> {code}
> from keras.applications.vgg16 import VGG16
> keras_model = VGG16(weights="imagenet", pooling="max") 
> from systemml.mllearn import Keras2DML
> sysml_model = Keras2DML(spark, keras_model, input_shape=(3,224,224), 
> weights='weights_dir')
> sysml_model.set(test_algo='batch', train_algo='minibatch')
> print(sysml_model.get_training_script())
> print(sysml_model.get_prediction_script())
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2505) Support generation of DML script using Keras2DML and Caffe2DML

2018-12-08 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2505:
-

 Summary: Support generation of DML script using Keras2DML and 
Caffe2DML
 Key: SYSTEMML-2505
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2505
 Project: SystemML
  Issue Type: Test
  Components: APIs
Reporter: Niketan Pansare
Assignee: Niketan Pansare
 Fix For: SystemML 1.3


 
{code}
from keras.applications.vgg16 import VGG16

keras_model = VGG16(weights="imagenet", pooling="max") 

from systemml.mllearn import Keras2DML

sysml_model = Keras2DML(spark, keras_model, input_shape=(3,224,224), 
weights='weights_dir')

sysml_model.set(test_algo='batch', train_algo='minibatch')

print(sysml_model.get_training_script())

print(sysml_model.get_prediction_script())
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (SYSTEMML-2496) Skip MapReduce JUnit tests

2018-10-14 Thread Niketan Pansare (JIRA)


 [ 
https://issues.apache.org/jira/browse/SYSTEMML-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-2496.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.3

Fixed by the commit 
https://github.com/apache/systemml/commit/95bf8cfe6c0ae4f45b460a03c397203b7b354fc5

> Skip MapReduce JUnit tests 
> ---
>
> Key: SYSTEMML-2496
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2496
> Project: SystemML
>  Issue Type: Test
>  Components: Test
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>Priority: Minor
> Fix For: SystemML 1.3
>
>
> Since MR backend is in maintenance mode, skip MR tests to allow the JUnit 
> test to complete in less time. This will also enable us to turn on the larger 
> number of Spark tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2496) Skip MapReduce JUnit tests

2018-10-11 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2496:
-

 Summary: Skip MapReduce JUnit tests 
 Key: SYSTEMML-2496
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2496
 Project: SystemML
  Issue Type: Test
  Components: Test
Reporter: Niketan Pansare
Assignee: Niketan Pansare


Since MR backend is in maintenance mode, skip MR tests to allow the JUnit test 
to complete in less time. This will also enable us to turn on the larger number 
of Spark tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (SYSTEMML-880) Push-down loop structures in Python DSL

2018-07-10 Thread Niketan Pansare (JIRA)


[ 
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539267#comment-16539267
 ] 

Niketan Pansare edited comment on SYSTEMML-880 at 7/10/18 9:54 PM:
---

Pushdown of loop avoids invocation overhead and also enables additional 
optimization. Here is a simple pyspark script that demonstrates the overhead:

 
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)

# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' +  str(time.time()-t0))
# Total time with loop:2.50334095955

# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
    t0 = time.time()
    pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
    totalTime = totalTime + time.time()-t0

print('Total time without loop:' +  str(totalTime))
# Total time without loop:1008.73590732
{code}
 

One way to go about doing this is to define the boundaries using a decorator 
(for example: parallelize) and try by first supporting simple expression and a 
loop structure.

 

Few related links:

[https://greentreesnakes.readthedocs.io/en/latest/nodes.html#control-flow]

[https://eli.thegreenplace.net/2009/11/28/python-internals-working-with-python-asts]

 


was (Author: niketanpansare):
Pushdown of loop avoids invocation overhead and also enables additional 
optimization. Here is a simple pyspark script that demonstrates the overhead:

 
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)

# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' +  str(time.time()-t0))
# Total time with loop:2.50334095955

# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
    t0 = time.time()
    pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
    totalTime = totalTime + time.time()-t0

print('Total time without loop:' +  str(totalTime))
# Total time without loop:1008.73590732
{code}
 

One way to go about doing this is to define the boundaries using a decorator 
(for example: parallelize) and try by first supporting simple expression and a 
loop structure.

 

> Push-down loop structures in Python DSL
> ---
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SYSTEMML-880) Push-down loop structures in Python DSL

2018-07-10 Thread Niketan Pansare (JIRA)


[ 
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539267#comment-16539267
 ] 

Niketan Pansare commented on SYSTEMML-880:
--

Pushdown of loop avoids invocation overhead and also enables additional 
optimization. Here is a simple pyspark script that demonstrates the overhead:

 
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)

# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' +  str(time.time()-t0))
# Total time with loop:2.50334095955

# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
    t0 = time.time()
    pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
    totalTime = totalTime + time.time()-t0

print('Total time without loop:' +  str(totalTime))
# Total time without loop:1008.73590732
{code}
 

One way to go about doing this is to define the boundaries using a decorator 
(for example: parallelize) and try by first supporting simple expression and a 
loop structure.

 

> Push-down loop structures in Python DSL
> ---
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SYSTEMML-880) Push-down loop structures in Python DSL

2018-06-19 Thread Niketan Pansare (JIRA)


[ 
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16517551#comment-16517551
 ] 

Niketan Pansare commented on SYSTEMML-880:
--

Since the below steps are not necessarily subtask, I am listing them:
 # Read the documentation and try the code: 
[http://apache.github.io/systemml/python-reference#matrix-class]
 # Understand the core problem of pushing down the loops: 
[http://apache.github.io/systemml/python-reference#dealing-with-the-loops]
 # Then, read the discussion in [PR 
270|https://github.com/apache/systemml/pull/270] and understand why a [simple 
solution|https://github.com/niketanpansare/systemml/blob/6e4c0a15a597034010ce80439aaee87960e836e1/src/main/python/systemml/pushdown.py#L43-L47]
 is not preferred.
 # The solution that is preferred is to walk the [Python 
AST|https://docs.python.org/2/library/ast.html] of the given function and 
replace loops with PyDML script.

> Push-down loop structures in Python DSL
> ---
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2313) Add custom kernel for sparse matrix dense matrix multiplication

2018-05-10 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2313:
-

 Summary: Add custom kernel for sparse matrix dense matrix 
multiplication
 Key: SYSTEMML-2313
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2313
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2312) Add custom kernel for sparse matrix dense vector multiplication

2018-05-10 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2312:
-

 Summary: Add custom kernel for sparse matrix dense vector 
multiplication
 Key: SYSTEMML-2312
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2312
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (SYSTEMML-1092) Integrate python test cases into junit test case

2018-04-04 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425815#comment-16425815
 ] 

Niketan Pansare edited comment on SYSTEMML-1092 at 4/4/18 4:43 PM:
---

Subtasks for this JIRAs are:
 # Investigate whether it is possible to package pyspark dependencies into our 
maven test suite without spark installation.
 # If not, then come-up with simplest spark installation:
 ## Download spark and set SPARK_HOME
 ## Add pyspark as pip dependency
 # Enable tests 
[https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41]

For reference, I believe PySpark test suite is kicked using the shell script: 
[https://github.com/apache/spark/blob/master/dev/run-tests-jenkins]


was (Author: niketanpansare):
Before closing this issue, one needs to investigate whether it is possible to 
package pyspark dependencies into our maven test suite. If not, the following 
solution needs to be discussed in the mailing list before adding SPARK_HOME 
requirement

- The Python tests are currently disabled as it requires the environment 
variable SPARK_HOME to be set: 
https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41
- For reference, I believe PySpark test suite is kicked using the shell script: 
https://github.com/apache/spark/blob/master/dev/run-tests-jenkins

> Integrate python test cases into junit test case
> 
>
> Key: SYSTEMML-1092
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1092
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SYSTEMML-1092) Integrate python test cases into junit test case

2018-04-04 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16425815#comment-16425815
 ] 

Niketan Pansare commented on SYSTEMML-1092:
---

Before closing this issue, one needs to investigate whether it is possible to 
package pyspark dependencies into our maven test suite. If not, the following 
solution needs to be discussed in the mailing list before adding SPARK_HOME 
requirement

- The Python tests are currently disabled as it requires the environment 
variable SPARK_HOME to be set: 
https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41
- For reference, I believe PySpark test suite is kicked using the shell script: 
https://github.com/apache/spark/blob/master/dev/run-tests-jenkins

> Integrate python test cases into junit test case
> 
>
> Key: SYSTEMML-1092
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1092
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (SYSTEMML-2165) Support ARC eviction policy for the GPU backend

2018-03-01 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-2165:
-

 Summary: Support ARC eviction policy for the GPU backend
 Key: SYSTEMML-2165
 URL: https://issues.apache.org/jira/browse/SYSTEMML-2165
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare


Link to the paper: 
[https://dbs.uni-leipzig.de/file/ARC.pdf|https://dbs.uni-leipzig.de/file/ARC.pdf]

There might be patent related issues with ARC algorithm. If so, we cannot 
support it in SystemML: 
[https://news.ycombinator.com/item?id=9898430|https://news.ycombinator.com/item?id=9898430]





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (SYSTEMML-1975) Fix GPU tests for single precision

2017-10-26 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221207#comment-16221207
 ] 

Niketan Pansare commented on SYSTEMML-1975:
---

Following tests are failing for double precision:
{code}
Failed tests:
MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:267
 Relative error(1.00) is more than threshold (0.10). Expected = 
-0.002456, Actual = 0.00, differed at [0, 0]
  
MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
 expected:<0.0> but was:<1.929108490094312>
{code}

My guess is there is likely a bug caught by these two tests and other tests are 
likely related to precision.

> Fix GPU tests for single precision
> --
>
> Key: SYSTEMML-1975
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1975
> Project: SystemML
>  Issue Type: Bug
>Reporter: Niketan Pansare
>
> The current GPU tests for single precision compares the results of double 
> precision operation on CPU with single precision operation on GPU. So, it is 
> difficult to compare the results. When we support single precision on CPU, we 
> can rework the GPU tests to fix the following tests:
> {code}
> Failed tests:
>   
> NeuralNetworkOpTests.testConv2dBackwardFilter:324->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(0.12) is more than threshold (0.10). Expected = 
> 25311.00, Actual = 25310.687500, differed at [0, 50]
>   
> MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(1.00) is more than threshold (0.10). Expected = 
> -0.002456, Actual = 0.00, differed at [0, 0]
>   
> MatrixMultiplicationOpTest.matrixMatrixTest4:209->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(0.24) is more than threshold (0.10). Expected = 
> 0.573809, Actual = 0.573822, differed at [5, 127]
>   
> MatrixMultiplicationOpTest.commonCaseDLMatrixMatrixTest2:137->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(0.13) is more than threshold (0.10). Expected = 
> -0.922950, Actual = -0.922939, differed at [0, 13]
>   
> MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
>  expected:<0.0> but was:<1.9291085004806519>
>   
> ScalarMatrixElementwiseOpTests.testModulusLeftScalar:167->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(1.00) is more than threshold (0.10). Expected = 
> 0.078125, Actual = 0.00, differed at [1500, 22]
>   
> ScalarMatrixElementwiseOpTests.testIntegerDivideLeftScalar:157->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
>  Relative error(0.015873) is more than threshold (0.10). Expected = 
> 63.00, Actual = 64.00, differed at [1500, 22]
> {code}
> [~nakul02]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Closed] (SYSTEMML-1969) Support single-precision on GPU

2017-10-26 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare closed SYSTEMML-1969.
-
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed in the commit 
https://github.com/apache/systemml/commit/abbffc55ef8f47f10b6e59b0ae5e1f311f4a8f3e

> Support single-precision on GPU
> ---
>
> Key: SYSTEMML-1969
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1969
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1975) Fix GPU tests for single precision

2017-10-26 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1975:
-

 Summary: Fix GPU tests for single precision
 Key: SYSTEMML-1975
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1975
 Project: SystemML
  Issue Type: Bug
Reporter: Niketan Pansare


The current GPU tests for single precision compares the results of double 
precision operation on CPU with single precision operation on GPU. So, it is 
difficult to compare the results. When we support single precision on CPU, we 
can rework the GPU tests to fix the following tests:

{code}
Failed tests:
  
NeuralNetworkOpTests.testConv2dBackwardFilter:324->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(0.12) is more than threshold (0.10). Expected = 
25311.00, Actual = 25310.687500, differed at [0, 50]
  
MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(1.00) is more than threshold (0.10). Expected = 
-0.002456, Actual = 0.00, differed at [0, 0]
  
MatrixMultiplicationOpTest.matrixMatrixTest4:209->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(0.24) is more than threshold (0.10). Expected = 
0.573809, Actual = 0.573822, differed at [5, 127]
  
MatrixMultiplicationOpTest.commonCaseDLMatrixMatrixTest2:137->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(0.13) is more than threshold (0.10). Expected = 
-0.922950, Actual = -0.922939, differed at [0, 13]
  
MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
 expected:<0.0> but was:<1.9291085004806519>
  
ScalarMatrixElementwiseOpTests.testModulusLeftScalar:167->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(1.00) is more than threshold (0.10). Expected = 
0.078125, Actual = 0.00, differed at [1500, 22]
  
ScalarMatrixElementwiseOpTests.testIntegerDivideLeftScalar:157->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
 Relative error(0.015873) is more than threshold (0.10). Expected = 
63.00, Actual = 64.00, differed at [1500, 22]

{code}

[~nakul02]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1969) Support single-precision on GPU

2017-10-21 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1969:
-

 Summary: Support single-precision on GPU
 Key: SYSTEMML-1969
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1969
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1965) Refactor nn layers to move the computation in forward/backward function known at compile time to init function

2017-10-17 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208004#comment-16208004
 ] 

Niketan Pansare commented on SYSTEMML-1965:
---

My two cents: I think both engine improvement as well as good DML practice go 
hand in hand. If `init` is not the right place, then we should add 
`compute_height` and `compute_width` functions. Having a loop-invariant code 
(as part of reusable function) in the for loop is not advisable even if we fix 
the constant propagation portion of the engine. 

> Refactor nn layers to move the computation in forward/backward function known 
> at compile time to init function
> --
>
> Key: SYSTEMML-1965
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1965
> Project: SystemML
>  Issue Type: Bug
>Reporter: Niketan Pansare
>
> Ideally, we should move the computation known at compile time to init layer, 
> rather than keep in the forward function. This reduces recompilation time and 
> also potentially unnecessary instructions. Here is an example snippet from 
> our conv2d layer:
> {code}
> Hout = as.integer(floor((Hin + 2*padh - Hf)/strideh + 1))
> Wout = as.integer(floor((Win + 2*padw - Wf)/stridew + 1))
> {code}
> [~prithvi_r_s] [~dusenberrymw] [~reinwald] do you have any comments or 
> concerns ?
> [~dusenberrymw] do you have free cycles to take this over ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1965) Refactor nn layers to move the computation in forward/backward function known at compile time to init function

2017-10-16 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1965:
-

 Summary: Refactor nn layers to move the computation in 
forward/backward function known at compile time to init function
 Key: SYSTEMML-1965
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1965
 Project: SystemML
  Issue Type: Bug
Reporter: Niketan Pansare


Ideally, we should move the computation known at compile time to init layer, 
rather than keep in the forward function. This reduces recompilation time and 
also potentially unnecessary instructions. Here is an example snippet from our 
conv2d layer:

{code}
Hout = as.integer(floor((Hin + 2*padh - Hf)/strideh + 1))
Wout = as.integer(floor((Win + 2*padw - Wf)/stridew + 1))
{code}

[~prithvi_r_s] [~dusenberrymw] [~reinwald] do you have any comments or concerns 
?

[~dusenberrymw] do you have free cycles to take this over ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-16 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206706#comment-16206706
 ] 

Niketan Pansare commented on SYSTEMML-1962:
---

Interesting paper somewhat related to this PR: https://arxiv.org/abs/1601.00024

> Support model-selection via mllearn APIs
> 
>
> Key: SYSTEMML-1962
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>Assignee: Janardhan
>
> The end goal of this JIRA is to support model selection facility similar to 
> [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
> Currently, we support model selection using MLPipeline's cross-validator. For 
> example: please replace `from pyspark.ml.classification import 
> LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
> the example 
> http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
>  
> However, this invokes k-seperate and independent mlcontext calls. This PR 
> proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
> bayesian optimization which like mllearn has methods `fit` and `predict`. 
> These methods internally generate a script that wraps the external script 
> with a `parfor` when the fit method is called. For example:
> {code}
> from sklearn import datasets
> from systemml.mllearn import GridSearchCV, SVM
> iris = datasets.load_iris()
> parameters = {'C':[1, 10]}
> svm = SVM()
> clf = GridSearchClassifierCV(svm, parameters)
> clf.fit(iris.data, iris.target)
> {code}
> would execute the script:
> {code}
> CVals = matrix("1; 10", rows=2, cols=1)
> parfor(i in seq(1, nrow(CVals))) {
>C = CVals[i, 1]
>reg = 1 / C
> # SVM script
> }
> {code}
> This will require:
> 1. Functionization of the script (for example: L2SVM)
> {code}
> svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
> double reg, double maxiter) returns (matrix[double] w) {
>if(nrow(X) < 2)
>   stop("Stopping due to invalid inputs: Not possible to learn a binary 
> class classifier without at least 2 rows")
>check_min = min(Y)
>
>w = t(cbind(t(w), t(extra_model_params)))
> }
> {code}
>  2. Adding two new java classes in the package `org.apache.sysml.api.ml` 
> called `GridSearchClassifierCV` which extends 
> `Estimator[GridSearchClassifierCVModel]` and `GridSearchClassifierCVModel` 
> which `extends Model[GridSearchClassifierCVModel] with 
> BaseSystemMLClassifierModel`. Then you will have to implement the abstract 
> methods: fit and transform respectively.
> 3. Add a python class GridSearchClassifierCV that invokes the above java 
> classes.
> For more details on step 2 and step 3, please read the design documentation 
> of mllearn API: 
> https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42
> [~dusenberrymw] may be, this can be part of 
> https://issues.apache.org/jira/browse/SYSTEMML-1159



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: 
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
   reg = 1 / C
# SVM script
}
{code}

This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
double reg, double maxiter) returns (matrix[double] w) {
   if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary 
class classifier without at least 2 rows")
   check_min = min(Y)
   

   w = t(cbind(t(w), t(extra_model_params)))
}
{code}

 2. Adding two new java classes in the package `org.apache.sysml.api.ml` called 
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]` 
and `GridSearchClassifierCVModel` which `extends 
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you 
will have to implement the abstract methods: fit and transform respectively.

3. Add a python class GridSearchClassifierCV that invokes the above java 
classes.

For more details on step 2 and step 3, please read the design documentation of 
mllearn API: 
https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42

[~dusenberrymw] may be, this can be part of 
https://issues.apache.org/jira/browse/SYSTEMML-1159

  was:
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}

This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
double reg, double maxiter) returns (matrix[double] w) {
   if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary 
class classifier without at least 2 rows")
   check_min = min(Y)
   

   w = t(cbind(t(w), t(extra_model_params)))
}
{code}

 2. Adding two new java classes in the package `org.apache.sysml.api.ml` called 
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]` 
and `GridSearchClassifierCVModel` which `extends 
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you 
will have to implement the abstract methods: fit and transform respectively.

3. Add a python class GridSearchClassifierCV that invokes the above java 
classes.

For more details on step 2 and step 3, please read the design documentation of 
mllearn API: 
https://github.com/apache/systemml/blob/master/src

[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: 
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}

This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
double reg, double maxiter) returns (matrix[double] w) {
   if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary 
class classifier without at least 2 rows")
   check_min = min(Y)
   

   w = t(cbind(t(w), t(extra_model_params)))
}
{code}

 2. Adding two new java classes in the package `org.apache.sysml.api.ml` called 
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]` 
and `GridSearchClassifierCVModel` which `extends 
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you 
will have to implement the abstract methods: fit and transform respectively.

3. Add a python class GridSearchClassifierCV that invokes the above java 
classes.

For more details on step 2 and step 3, please read the design documentation of 
mllearn API: 
https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42

[~dusenberrymw] may be, this can be part of 
https://issues.apache.org/jira/browse/SYSTEMML-1159

  was:
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}

This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
double reg, double maxiter) returns (matrix[double] w) {
   if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary 
class classifier without at least 2 rows")
   check_min = min(Y)
   

   w = t(cbind(t(w), t(extra_model_params)))
}
{code}

 2. Adding two new java classes in the package `org.apache.sysml.api.ml` called 
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]` 
and `GridSearchClassifierCVModel` which `extends 
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you 
will have to implement the abstract methods: fit and transform respectively.

3. Add a python class GridSearchClassifierCV that invokes the above java 
classes.

For more details on step 2 and step 3, please read the design documentation of 
mllearn API: 
https://github.com/apache/systemml/blob/master/src/main/scala/org

[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: 
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}

This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol, 
double reg, double maxiter) returns (matrix[double] w) {
   if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary 
class classifier without at least 2 rows")
   check_min = min(Y)
   

   w = t(cbind(t(w), t(extra_model_params)))
}
{code}

 2. Adding two new java classes in the package `org.apache.sysml.api.ml` called 
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]` 
and `GridSearchClassifierCVModel` which `extends 
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you 
will have to implement the abstract methods: fit and transform respectively.

3. Add a python class GridSearchClassifierCV that invokes the above java 
classes.

For more details on step 2 and step 3, please read the design documentation of 
mllearn API: 
https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42

[~dusenberrymw] 

  was:
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}


> Support model-selection via mllearn APIs
> 
>
> Key: SYSTEMML-1962
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>
> The end goal of this JIRA is to support model selection facility similar to 
> [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
> Currently, we support model selection using MLPipeline's cross-validator. For 
> example: please replace `from pyspark.ml.classification import 
> LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
> the example 
> http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
>  
> However, this invokes k-seperate and independent mlcontext calls. This PR 
> proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
> bayesian optimization which like mllearn has methods `fit` and `predict`. 
> These methods internally generate a script that wraps the external script 
> with a `

[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: 
The end goal of this JIRA is to support model selection facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].

Currently, we support model selection using MLPipeline's cross-validator. For 
example: please replace `from pyspark.ml.classification import 
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
the example 
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
 

However, this invokes k-seperate and independent mlcontext calls. This PR 
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
bayesian optimization which like mllearn has methods `fit` and `predict`. These 
methods internally generate a script that wraps the external script with a 
`parfor` when the fit method is called. For example:

{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}

would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
   C = CVals[i, 1]
# SVM script
}
{code}

  was:The end goal of this JIRA is to support model selection facility similar 
to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].


> Support model-selection via mllearn APIs
> 
>
> Key: SYSTEMML-1962
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>
> The end goal of this JIRA is to support model selection facility similar to 
> [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
> Currently, we support model selection using MLPipeline's cross-validator. For 
> example: please replace `from pyspark.ml.classification import 
> LogisticRegression` with `from systemml.mllearn import LogisticRegression` in 
> the example 
> http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
>  
> However, this invokes k-seperate and independent mlcontext calls. This PR 
> proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly 
> bayesian optimization which like mllearn has methods `fit` and `predict`. 
> These methods internally generate a script that wraps the external script 
> with a `parfor` when the fit method is called. For example:
> {code}
> from sklearn import datasets
> from systemml.mllearn import GridSearchCV, SVM
> iris = datasets.load_iris()
> parameters = {'C':[1, 10]}
> svm = SVM()
> clf = GridSearchCV(svm, parameters)
> clf.fit(iris.data, iris.target)
> {code}
> would execute the script:
> {code}
> CVals = matrix("1; 10", rows=2, cols=1)
> parfor(i in seq(1, nrow(CVals))) {
>C = CVals[i, 1]
> # SVM script
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: The end goal of this JIRA is to support model selection 
facility similar to 
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
  (was: The end goal of this JIRA is to support model selection facility 
similar to 
[scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection).)

> Support model-selection via mllearn APIs
> 
>
> Key: SYSTEMML-1962
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>
> The end goal of this JIRA is to support model selection facility similar to 
> [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1962:
--
Description: The end goal of this JIRA is to support model selection 
facility similar to 
[scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection).

> Support model-selection via mllearn APIs
> 
>
> Key: SYSTEMML-1962
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>
> The end goal of this JIRA is to support model selection facility similar to 
> [scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1962) Support model-selection via mllearn APIs

2017-10-14 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1962:
-

 Summary: Support model-selection via mllearn APIs
 Key: SYSTEMML-1962
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1962
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1940) Modify the default directory of performance test to avoid maven compilation issue

2017-10-05 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1940.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed in the commit 
https://github.com/apache/systemml/commit/355373990c478293090c90557383ab24c2c01c6b

> Modify the default directory of performance test to avoid maven compilation 
> issue
> -
>
> Key: SYSTEMML-1940
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1940
> Project: SystemML
>  Issue Type: Test
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>
> Currently, the performance test generates it data in scripts folder. This 
> causes maven to take too long for compilation as it tries to create jar with 
> all the data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (SYSTEMML-1940) Modify the default directory of performance test to avoid maven compilation issue

2017-10-05 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare reassigned SYSTEMML-1940:
-

Assignee: Niketan Pansare

> Modify the default directory of performance test to avoid maven compilation 
> issue
> -
>
> Key: SYSTEMML-1940
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1940
> Project: SystemML
>  Issue Type: Test
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>
> Currently, the performance test generates it data in scripts folder. This 
> causes maven to take too long for compilation as it tries to create jar with 
> all the data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1940) Modify the default directory of performance test to avoid maven compilation issue

2017-09-28 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1940:
-

 Summary: Modify the default directory of performance test to avoid 
maven compilation issue
 Key: SYSTEMML-1940
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1940
 Project: SystemML
  Issue Type: Test
Reporter: Niketan Pansare


Currently, the performance test generates it data in scripts folder. This 
causes maven to take too long for compilation as it tries to create jar with 
all the data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (SYSTEMML-1500) Add missing loss layers to Caffe2DML

2017-09-28 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare reassigned SYSTEMML-1500:
-

Assignee: Niketan Pansare

> Add missing loss layers to Caffe2DML
> 
>
> Key: SYSTEMML-1500
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1500
> Project: SystemML
>  Issue Type: Sub-task
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>
> Multinomial Logistic Loss
> Infogain Loss - a generalization of MultinomialLogisticLossLayer.
> Softmax with Loss - computes the multinomial logistic loss of the softmax of 
> its inputs. It’s conceptually identical to a softmax layer followed by a 
> multinomial logistic loss layer, but provides a more numerically stable 
> gradient.
> Sum-of-Squares / Euclidean - computes the sum of squares of differences of 
> its two inputs, 12N∑Ni=1∥x1i−x2i∥2212N∑i=1N‖xi1−xi2‖22.
> Hinge / Margin - The hinge loss layer computes a one-vs-all hinge (L1) or 
> squared hinge loss (L2).
> Sigmoid Cross-Entropy Loss - computes the cross-entropy (logistic) loss, 
> often used for predicting targets interpreted as probabilities.
> Accuracy / Top-k layer - scores the output as an accuracy with respect to 
> target – it is not actually a loss and has no backward step.
> Contrastive Loss



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1500) Add missing loss layers to Caffe2DML

2017-09-28 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184560#comment-16184560
 ] 

Niketan Pansare commented on SYSTEMML-1500:
---

- Euclidean loss added in the commit: 
https://github.com/apache/systemml/commit/61dcc85e48a390c1bb63ee4c42aad9a3fade7d06

> Add missing loss layers to Caffe2DML
> 
>
> Key: SYSTEMML-1500
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1500
> Project: SystemML
>  Issue Type: Sub-task
>Reporter: Niketan Pansare
>
> Multinomial Logistic Loss
> Infogain Loss - a generalization of MultinomialLogisticLossLayer.
> Softmax with Loss - computes the multinomial logistic loss of the softmax of 
> its inputs. It’s conceptually identical to a softmax layer followed by a 
> multinomial logistic loss layer, but provides a more numerically stable 
> gradient.
> Sum-of-Squares / Euclidean - computes the sum of squares of differences of 
> its two inputs, 12N∑Ni=1∥x1i−x2i∥2212N∑i=1N‖xi1−xi2‖22.
> Hinge / Margin - The hinge loss layer computes a one-vs-all hinge (L1) or 
> squared hinge loss (L2).
> Sigmoid Cross-Entropy Loss - computes the cross-entropy (logistic) loss, 
> often used for predicting targets interpreted as probabilities.
> Accuracy / Top-k layer - scores the output as an accuracy with respect to 
> target – it is not actually a loss and has no backward step.
> Contrastive Loss



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1493) Add Tanh layer in nn library and Caffe2DML

2017-09-28 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1493.
---
   Resolution: Fixed
 Assignee: Niketan Pansare
Fix Version/s: SystemML 1.0

Fixed in the commit 
https://github.com/apache/systemml/commit/61dcc85e48a390c1bb63ee4c42aad9a3fade7d06

> Add Tanh layer in nn library and Caffe2DML
> --
>
> Key: SYSTEMML-1493
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1493
> Project: SystemML
>  Issue Type: Sub-task
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>
> http://caffe.berkeleyvision.org/tutorial/layers/tanh.html



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Updated] (SYSTEMML-1930) Fix protoc compilation error for TensorFlow proto files and restore Tensorboard functionality

2017-09-22 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare updated SYSTEMML-1930:
--
Description: 
To use Tensorboard in Caffe2DML, we were compiling 
src/main/proto/tensorflow/event.proto and 
src/main/proto/tensorflow/summary.proto files using protoc 3.0 mvn plugin 
(please see 
[https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369]).
 These proto files match the ones that are available in TensorFlow repository.  

To compile SystemML on Power, we have to switch to a newer version of protoc 
mvn plugin which now only supports 3.4 version (not 3.0). The problem is that 
the above mentioned proto files are giving compilation error on protoc 3.4 
version. So, for time being, we are removed tensorboard integration from 
SystemML in commit: 
https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369

This task involves debugging compilation error on protoc 3.4 version and 
restoring the above commit.

[~freiss] [~nakul02] [~reinwald]

  was:
To use Tensorboard in Caffe2DML, we were compiling 
src/main/proto/tensorflow/event.proto and 
src/main/proto/tensorflow/summary.proto files using protoc 3.0 mvn plugin 
(please see 
[https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369]).
 These proto files match the ones that are available in TensorFlow repository.  

To compile SystemML on Power, we have to switch to a newer version of protoc 
mvn plugin which now only supports 3.4 version (not 3.0). The problem is that 
the above mentioned proto files are giving compilation error. So, for time 
being, we are removed tensorboard integration from SystemML in commit: 
https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369

This task involves debugging compilation error on protoc 3.4 version and 
restoring the above commit.

[~freiss] [~nakul02] [~reinwald]


> Fix protoc compilation error for TensorFlow proto files and restore 
> Tensorboard functionality
> -
>
> Key: SYSTEMML-1930
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1930
> Project: SystemML
>  Issue Type: Bug
>Reporter: Niketan Pansare
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> To use Tensorboard in Caffe2DML, we were compiling 
> src/main/proto/tensorflow/event.proto and 
> src/main/proto/tensorflow/summary.proto files using protoc 3.0 mvn plugin 
> (please see 
> [https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369]).
>  These proto files match the ones that are available in TensorFlow 
> repository.  
> To compile SystemML on Power, we have to switch to a newer version of protoc 
> mvn plugin which now only supports 3.4 version (not 3.0). The problem is that 
> the above mentioned proto files are giving compilation error on protoc 3.4 
> version. So, for time being, we are removed tensorboard integration from 
> SystemML in commit: 
> https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369
> This task involves debugging compilation error on protoc 3.4 version and 
> restoring the above commit.
> [~freiss] [~nakul02] [~reinwald]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1928) Remove tensorboard visualization code

2017-09-22 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176784#comment-16176784
 ] 

Niketan Pansare commented on SYSTEMML-1928:
---

I have created a new JIRA for this feature: 
https://issues.apache.org/jira/browse/SYSTEMML-1930

> Remove tensorboard visualization code
> -
>
> Key: SYSTEMML-1928
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1928
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>
> This avoids protoc compilation error on power.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1930) Fix protoc compilation error for TensorFlow proto files and restore Tensorboard functionality

2017-09-22 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1930:
-

 Summary: Fix protoc compilation error for TensorFlow proto files 
and restore Tensorboard functionality
 Key: SYSTEMML-1930
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1930
 Project: SystemML
  Issue Type: Bug
Reporter: Niketan Pansare


To use Tensorboard in Caffe2DML, we were compiling 
src/main/proto/tensorflow/event.proto and 
src/main/proto/tensorflow/summary.proto files using protoc 3.0 mvn plugin 
(please see 
[https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369]).
 These proto files match the ones that are available in TensorFlow repository.  

To compile SystemML on Power, we have to switch to a newer version of protoc 
mvn plugin which now only supports 3.4 version (not 3.0). The problem is that 
the above mentioned proto files are giving compilation error. So, for time 
being, we are removed tensorboard integration from SystemML in commit: 
https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369

This task involves debugging compilation error on protoc 3.4 version and 
restoring the above commit.

[~freiss] [~nakul02] [~reinwald]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1928) Remove tensorboard visualization code

2017-09-21 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1928.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed by the commit 
https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491

> Remove tensorboard visualization code
> -
>
> Key: SYSTEMML-1928
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1928
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>
> This avoids protoc compilation error on power.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1923) Support sinh, cosh and tanh as built-in functions

2017-09-21 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1923.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed in the commit 
https://github.com/apache/systemml/commit/50a895f86167d5766b249a3db89f9c2b06f7a0d0

> Support sinh, cosh and tanh as built-in functions
> -
>
> Key: SYSTEMML-1923
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1923
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1928) Remove tensorboard visualization code

2017-09-21 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1928:
-

 Summary: Remove tensorboard visualization code
 Key: SYSTEMML-1928
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1928
 Project: SystemML
  Issue Type: Task
Reporter: Niketan Pansare
Assignee: Niketan Pansare


This avoids protoc compilation error on power.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1923) Support sinh, cosh and tanh as built-in functions

2017-09-20 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1923:
-

 Summary: Support sinh, cosh and tanh as built-in functions
 Key: SYSTEMML-1923
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1923
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz

2017-09-14 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16166772#comment-16166772
 ] 

Niketan Pansare commented on SYSTEMML-1907:
---

thanks [~gweidner] :)

> Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
> 
>
> Key: SYSTEMML-1907
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1907
> Project: SystemML
>  Issue Type: Improvement
>  Components: Build
>Reporter: Niketan Pansare
>Assignee: Glenn Weidner
>
> I encountered this issue because pypi has migrated to a new process: 
> https://packaging.python.org/guides/migrating-to-pypi-org/#uploading
> As noted in the above document, the recommended way to upload python packages 
> to pypi is now via `twine`. However, if we use `twine` with our current 
> package naming scheme (i.e. tgz), then it complains `ValueError: Unknown 
> distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend 
> to use suffix `tar.gz` in our subsequent releases. This way we also are 
> compatible with default package naming convention of pypi: `tar.gz`.
> [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any 
> takers ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz

2017-09-14 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1907:
-

 Summary: Rename python package from systemml-*-python.tgz to 
systemml-*-python.tar.gz
 Key: SYSTEMML-1907
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1907
 Project: SystemML
  Issue Type: Improvement
  Components: Build
Reporter: Niketan Pansare


I encountered this issue because pypi has migrated to a new process: 
https://packaging.python.org/guides/migrating-to-pypi-org/#uploading

As noted in the above document, the recommended way to upload python packages 
to pypi is now via `twine`. However, if we use `twine` with our current package 
naming scheme (i.e. tgz), then it complains `ValueError: Unknown distribution 
format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend to use suffix 
`tar.gz` in our subsequent releases. This way we also are compatible with 
default package naming convention of pypi: `tar.gz`.

[~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any 
takers ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-976) Add explain and stats option to Python DSL

2017-09-08 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158911#comment-16158911
 ] 

Niketan Pansare commented on SYSTEMML-976:
--

Yes, absolutely. Please let me know if you need any help. Another interesting 
JIRA is https://issues.apache.org/jira/browse/SYSTEMML-880 :)

> Add explain and stats option to Python DSL
> --
>
> Key: SYSTEMML-976
> URL: https://issues.apache.org/jira/browse/SYSTEMML-976
> Project: SystemML
>  Issue Type: Improvement
>  Components: APIs
>Reporter: Niketan Pansare
>  Labels: Hacktoberfest, starter
>
> This would allow users to see the plans and statistics for a given evaluation 
> of Python DSL. Since this is a starter task, I would give some hints:
> 1. Extend "def eval(outputs, outputDF=False, execute=True):" method to accept 
> explain and statistics options and pass them before "results = 
> matrix.ml.execute(matrix.script)".
> Here are the relevant links:
> https://github.com/apache/incubator-systemml/blob/master/src/main/python/systemml/defmatrix.py#L290
> https://github.com/apache/incubator-systemml/blob/master/src/main/python/systemml/mlcontext.py#L223
> 2. Expose these options through user facing APIs: eval() and methods that 
> invoke eval (such as toPandas, toNumPyArray, toDataFrame, etc)
> 3. Update the documentation: 
> https://apache.github.io/incubator-systemml/beginners-guide-python



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-984) Add mllearn and scala wrappers for PCA

2017-08-28 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144306#comment-16144306
 ] 

Niketan Pansare commented on SYSTEMML-984:
--

If we implement scalable PCA and do reasonable accuracy/performance testing, 
then I would recommend that we add scala/python wrapper :)

> Add mllearn and scala wrappers for PCA
> --
>
> Key: SYSTEMML-984
> URL: https://issues.apache.org/jira/browse/SYSTEMML-984
> Project: SystemML
>  Issue Type: Task
>  Components: APIs
>Reporter: Niketan Pansare
>  Labels: Hacktoberfest, starter
>
> See 
> https://apache.github.io/incubator-systemml/algorithms-matrix-factorization.html#principle-component-analysis
>  for usage



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1870) Include avro dependency in the bin and standalone-jar artifacts

2017-08-24 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1870:
-

 Summary: Include avro dependency in the bin and standalone-jar 
artifacts
 Key: SYSTEMML-1870
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1870
 Project: SystemML
  Issue Type: Bug
Reporter: Niketan Pansare
Assignee: Deron Eriksson


[~prithvi_r_s]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1808) Visualize Hop DAG for explaining the optimizer

2017-07-26 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1808:
-

 Summary: Visualize Hop DAG for explaining the optimizer
 Key: SYSTEMML-1808
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1808
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1798) provide getScriptExecutionString and getScriptString for Python Script object

2017-07-24 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1798.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed in the commit 
https://github.com/apache/systemml/commit/8f412ac5cd2a06453a55efbe4afe60e838cd33b1

> provide getScriptExecutionString and getScriptString for Python Script object
> -
>
> Key: SYSTEMML-1798
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1798
> Project: SystemML
>  Issue Type: Sub-task
>  Components: APIs
>Reporter: Berthold Reinwald
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1802) Fix build failure for distribution profile on Windows

2017-07-24 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1802.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed by the commit 
https://github.com/apache/systemml/commit/11b689d49b78a73f44b5944cfe0d14bb7c05c3a7

> Fix build failure for distribution profile on Windows
> -
>
> Key: SYSTEMML-1802
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1802
> Project: SystemML
>  Issue Type: Bug
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>
> To test please run `mvn package -P distribution` on Windows machine



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1802) Fix build failure for distribution profile on Windows

2017-07-24 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1802:
-

 Summary: Fix build failure for distribution profile on Windows
 Key: SYSTEMML-1802
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1802
 Project: SystemML
  Issue Type: Bug
Reporter: Niketan Pansare
Assignee: Niketan Pansare


To test please run `mvn package -P distribution` on Windows machine



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Assigned] (SYSTEMML-1798) provide getScriptExecutionString and getScriptString for Python Script object

2017-07-21 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare reassigned SYSTEMML-1798:
-

Assignee: Niketan Pansare

> provide getScriptExecutionString and getScriptString for Python Script object
> -
>
> Key: SYSTEMML-1798
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1798
> Project: SystemML
>  Issue Type: Sub-task
>  Components: APIs
>Reporter: Berthold Reinwald
>Assignee: Niketan Pansare
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Resolved] (SYSTEMML-1796) Provide fine-grained statistics for SystemML

2017-07-21 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1796.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed by the commit 
https://github.com/apache/systemml/commit/648eb21d66f9cd8727090cdf950986765a7e6ee8

> Provide fine-grained statistics for SystemML
> 
>
> Key: SYSTEMML-1796
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1796
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1796) Provide fine-grained statistics for SystemML

2017-07-21 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1796:
-

 Summary: Provide fine-grained statistics for SystemML
 Key: SYSTEMML-1796
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1796
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1793) Support left and right indexing on GPU

2017-07-20 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1793:
-

 Summary: Support left and right indexing on GPU
 Key: SYSTEMML-1793
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1793
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1774) Improve Parfor parallelism for deep learning

2017-07-18 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092019#comment-16092019
 ] 

Niketan Pansare commented on SYSTEMML-1774:
---

Thanks [~mboehm7] I will look into it :)

> Improve Parfor parallelism for deep learning
> 
>
> Key: SYSTEMML-1774
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1774
> Project: SystemML
>  Issue Type: Improvement
>  Components: Algorithms, Compiler, ParFor
>Affects Versions: SystemML 1.0
>Reporter: Fei Hu
>  Labels: deeplearning
> Attachments: Explain_For_HYBRID_SPARK_Mode_With_ErrorInfo.txt, 
> Explain_For_Spark_Mode.txt, MNIST_Distrib_Sgd.scala, 
> mnist_lenet_distrib_sgd.dml
>
>
> When running the  [distributed MNIST LeNet example | 
> https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml],
>  each mini-batch could ideally run in parallel without interaction. We try to 
> force {{parfor (j in 1:parallel_batches)}} at line 137 of 
> {{nn/examples/mnist_lenet_distrib_sgd.dml}} to be {{parfor (j in 
> 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use 
> {{REMOTE_SPARK}} mode, but got some errors about 
> {{org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions 
> of type other than CP instructions}} using the mode {{SPARK}}, and the error 
> {{java.lang.NullPointerException}} using the mode {{HYBRID_SPARK}}. More log 
> information can be found at the following comments. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server

2017-07-14 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088255#comment-16088255
 ] 

Niketan Pansare commented on SYSTEMML-1695:
---

Personally, I would go through 
https://github.com/apache/systemml/blob/master/scripts/nn/examples/Example%20-%20MNIST%20LeNet.ipynb
 and think how can we modify 
https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet.dml#L190-L198
 to enable HogWild-style training.

> Create design/investigation document for parameter server
> -
>
> Key: SYSTEMML-1695
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1695
> Project: SystemML
>  Issue Type: Sub-task
>Reporter: Niketan Pansare
>Assignee: Janardhan
>
> Here are relevant links:
> - https://issues.apache.org/jira/browse/SPARK-4590
> - https://issues.apache.org/jira/browse/SPARK-6932
> - 
> https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit
> This document need not cover how to integrate a parameter server in SystemML 
> (which comes later), but should discuss following questions:
> 1. Should we integrate existing parameter server implementation or build one 
> from scratch ? What are the pros and cons of each approaches ? For example: 
> https://github.com/rjagerman/glint
> 2. If we plan to integrate existing parameter server implementation, what is 
> the interface ? (for exampe: update(matrix, new matrix) or update(matrix, 
> delta)).
> 3. Can parameter server help in ML algorithms ? Some experiments might be 
> helpful.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-979) Add support for bayesian optimization

2017-07-10 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081284#comment-16081284
 ] 

Niketan Pansare commented on SYSTEMML-979:
--

Also, notified Sasha regarding this JIRA :)

> Add support for bayesian optimization
> -
>
> Key: SYSTEMML-979
> URL: https://issues.apache.org/jira/browse/SYSTEMML-979
> Project: SystemML
>  Issue Type: New Feature
>  Components: Algorithms
>Reporter: Niketan Pansare
>Assignee: Janardhan
>
> Main paper: 
> https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf
> supplements: 
> *  selecting the next point to evaluate 
> [Jasper|http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf]
> *  sobol sequence generator [P Bratley, BL 
> Fox|http://dl.acm.org/citation.cfm?id=214372]
> *  Handling Sparsity via the Horseshoe [Carlos M. 
> Carvalho|http://ftp.isds.duke.edu/WorkingPapers/09-03.pdf]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server

2017-06-15 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050871#comment-16050871
 ] 

Niketan Pansare commented on SYSTEMML-1695:
---

This is a good start. Once you are ready with your draft, you may want to send 
it over the mailing list for feedback from other committers :)

> Create design/investigation document for parameter server
> -
>
> Key: SYSTEMML-1695
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1695
> Project: SystemML
>  Issue Type: Sub-task
>Reporter: Niketan Pansare
>
> Here are relevant links:
> - https://issues.apache.org/jira/browse/SPARK-4590
> - https://issues.apache.org/jira/browse/SPARK-6932
> - 
> https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit
> This document need not cover how to integrate a parameter server in SystemML 
> (which comes later), but should discuss following questions:
> 1. Should we integrate existing parameter server implementation or build one 
> from scratch ? What are the pros and cons of each approaches ? For example: 
> https://github.com/rjagerman/glint
> 2. If we plan to integrate existing parameter server implementation, what is 
> the interface ? (for exampe: update(matrix, new matrix) or update(matrix, 
> delta)).
> 3. Can parameter server help in ML algorithms ? Some experiments might be 
> helpful.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Created] (SYSTEMML-1695) Create design/investigation document for parameter server

2017-06-14 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1695:
-

 Summary: Create design/investigation document for parameter server
 Key: SYSTEMML-1695
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1695
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare


Here are relevant links:
- https://issues.apache.org/jira/browse/SPARK-4590
- https://issues.apache.org/jira/browse/SPARK-6932
- 
https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit

This document need not cover how to integrate a parameter server in SystemML 
(which comes later), but should discuss following questions:
1. Should we integrate existing parameter server implementation or build one 
from scratch ? What are the pros and cons of each approaches ? For example: 
https://github.com/rjagerman/glint
2. If we plan to integrate existing parameter server implementation, what is 
the interface ? (for exampe: update(matrix, new matrix) or update(matrix, 
delta)).
3. Can parameter server help in ML algorithms ? Some experiments might be 
helpful.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)


[jira] [Commented] (SYSTEMML-739) Explore model-parallel constructs in DML

2017-06-08 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043214#comment-16043214
 ] 

Niketan Pansare commented on SYSTEMML-739:
--

Hi Janardhan,

Welcome to SystemML community !!

Here is a light-weight parameter server implementation that you can look at: 
https://github.com/dmlc/ps-lite and the corresponding paper: 
https://www.cs.cmu.edu/~muli/file/parameter_server_osdi14.pdf

To integrate a parameter server with SystemML, you might want to look into how 
SystemML's bufferpool work and how one can extend DML to support model 
parallelism. You can begin by reading the papers 
http://www.vldb.org/pvldb/vol9/p1425-boehm.pdf and 
https://arxiv.org/pdf/1503.06384.pdf. 

Thanks,

Niketan.

> Explore model-parallel constructs in DML
> 
>
> Key: SYSTEMML-739
> URL: https://issues.apache.org/jira/browse/SYSTEMML-739
> Project: SystemML
>  Issue Type: Task
>Reporter: Niketan Pansare
>
> An example of such construct is providing access to the parameter server.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (SYSTEMML-1661) Builtin functions bias_add and bias_multiply not documented

2017-06-03 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16036174#comment-16036174
 ] 

Niketan Pansare commented on SYSTEMML-1661:
---

You are welcome :)

> Builtin functions bias_add and bias_multiply not documented
> ---
>
> Key: SYSTEMML-1661
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1661
> Project: SystemML
>  Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Niketan Pansare
>Priority: Minor
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (SYSTEMML-1661) Builtin functions bias_add and bias_multiply not documented

2017-06-03 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1661.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Resolved by the commit 
https://github.com/apache/incubator-systemml/commit/88f4a468f48081031d926d917ebc4f3e9014fc7f

> Builtin functions bias_add and bias_multiply not documented
> ---
>
> Key: SYSTEMML-1661
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1661
> Project: SystemML
>  Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Niketan Pansare
>Priority: Minor
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (SYSTEMML-1643) Extra jar does not include license information

2017-05-30 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030631#comment-16030631
 ] 

Niketan Pansare commented on SYSTEMML-1643:
---

Thanks Glenn for looking into this. You may want to take a look at: 
https://github.com/apache/incubator-systemml/blob/master/pom.xml#L296

> Extra jar does not include license information
> --
>
> Key: SYSTEMML-1643
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1643
> Project: SystemML
>  Issue Type: Bug
>Reporter: Arvind Surve
>Assignee: Glenn Weidner
>
> File systemml-x.y.z-incubating-SNAPSHOT-extra.jar has default LICENSE file 
> but it does not contain license information for files such as generated 
> classes from Caffe and TensorFlow. 
> It will be better to have separate LICENSE file in src/assembly/extra-jar 
> location of repository and add it in 
> systemml-x.y.z-incubating-SNAPSHOT-extra.jar file.
> Besides that, this jar file contains some extra files which should not be 
> included e.g. pom.xml etc.
> [~niketanpansare] Can you please look at this?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (SYSTEMML-1650) GPU cudnn produces worrisome amount of numerical instability

2017-05-30 Thread Niketan Pansare (JIRA)

[ 
https://issues.apache.org/jira/browse/SYSTEMML-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030627#comment-16030627
 ] 

Niketan Pansare commented on SYSTEMML-1650:
---

Let's consider conv2d builtin function. Can you please try following two things:
1. First, let's validate whether the results are due to bugs in SystemML GPU 
backend. Eg: incorrect parameters passed to CuDNN, incorrect memory transfer, 
etc. A simple way to validate this is to edit the following lines: 
https://github.com/apache/incubator-systemml/blob/master/scripts/nn/test/grad_check.dml#L447
 and replace them with
{code}
# Try modifying N, C, ... for further testing if you want
X = matrix(seq(1, N*C*Hin*Win), rows=N, cols=C*Hin*Win)
y = matrix(seq(N*F*Hin*Win, 1), rows=N, cols=F*Hin*Win)
{code} 

2. Second, assuming that the above test succeeds, we need to quantify how much 
precision we lose when using CuDNN as opposed to implementing 
numerically-stable conv2d using custom kernel.

> GPU cudnn produces worrisome amount of numerical instability
> 
>
> Key: SYSTEMML-1650
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1650
> Project: SystemML
>  Issue Type: Bug
>  Components: Runtime
>Affects Versions: SystemML 0.14
>Reporter: Nakul Jindal
> Fix For: SystemML 1.0
>
>
> When running GPU tests (mike's run_tests.dml in the nn directory)
> {code}
> 17/05/30 17:24:19 INFO api.DMLScript: BEGIN DML run 05/30/2017 17:24:19
> 17/05/30 17:24:20 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> 17/05/30 17:24:21 INFO context.GPUContext: Initializing CUDA
> Starting grad checks.
> ---
> 17/05/30 17:24:22 INFO context.GPUContext:  GPU memory - Total: 2096.300032 
> MB, Available: 1295.97438 MB on GPUContext{deviceNum=0}
> 17/05/30 17:24:22 INFO context.GPUContext: Total number of GPUs on the 
> machine: 1
> Grad checking the cross-entropy loss function.
> Grad checking the L1 loss function.
> Grad checking the L1 regularization function.
> Grad checking the L2 loss function.
> Grad checking the L2 regularization function.
> Grad checking the log loss function.
> Grad checking the affine layer with L2 loss.
>  - Grad checking X.
>  - Grad checking W.
>  - Grad checking b.
> Grad checking the 1D batch normalization layer with L2 loss.
>  - Grad checking the 'train' mode.
>- Grad checking X.
>- Grad checking gamma.
>- Grad checking beta.
>  - Grad checking the 'test' mode.
>- Grad checking X.
>- Grad checking gamma.
>- Grad checking beta.
> Grad checking the 2D (spatial) batch normalization layer with L2 loss.
>  - Grad checking the 'train' mode.
>- Grad checking X.
>- Grad checking gamma.
>- Grad checking beta.
>  - Grad checking the 'test' mode.
>- Grad checking X.
>- Grad checking gamma.
>- Grad checking beta.
> Grad checking the `im2col` 2D convolutional layer with L2 loss.
> 17/05/30 17:24:28 INFO Configuration.deprecation: session.id is deprecated. 
> Instead, use dfs.metrics.session-id
> 17/05/30 17:24:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
> processName=JobTracker, sessionId=
>  - Grad checking X.
>  - Grad checking W.
>  - Grad checking b.
> Grad checking the built-in 2D convolutional layer with L2 loss.
>  - Grad checking X.
>  - Grad checking W.
> WARNING: Relative error 3.063931109511093E-4 > 1.0E-4 & <= 0.01 with 
> -11.682479557456533 analytical vs -11.689640614065409 numerical, with lossph 
> 40.510115394324195 and lossmh 40.51034918713648
> WARNING: Relative error 6.785572589631694E-4 > 1.0E-4 & <= 0.01 with 
> -14.363880156229683 analytical vs -14.383386822913733 numerical, with lossph 
> 40.510088543924184 and lossmh 40.51037621166064
> WARNING: Relative error 8.117464157218959E-4 > 1.0E-4 & <= 0.01 with 
> -13.400658690617757 analytical vs -13.378920463225084 numerical, with lossph 
> 40.51009898805432 and lossmh 40.51036656646358
> WARNING: Relative error 6.785567321010216E-4 > 1.0E-4 & <= 0.01 with 
> -14.37300870216048 analytical vs -14.39252775057298 numerical, with lossph 
> 40.510088452456074 and lossmh 40.510376303011085
> WARNING: Relative error 0.0023065358169588085 > 1.0E-4 & <= 0.01 with 
> -15.081214796672182 analytical vs -15.011804170583785 numerical, with lossph 
> 40.510081360786614 and lossmh 40.510381596870026
> WARNING: Relative error 1.2020843619724922E-4 > 1.0E-4 & <= 0.01 with 
> -14.602099111310885 analytical vs -14.60561012436301 numerical, with lossph 
> 40.51008637609418 and lossmh 40.51037848829
> WARNING: Relative error 3.063921242335014E-4 > 1.0E-4 & <= 0.01 with 
> -11.654549775926586 analytical vs -11.66169368929104 numerical, with lossph 
> 40.51011567395115 and lossmh 40.510348907824934
>  - Grad

[jira] [Resolved] (SYSTEMML-1630) Remove confusing warning for disabled native BLAS

2017-05-26 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1630.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed by the commit 
https://github.com/apache/incubator-systemml/commit/fd41f0253810a8d0ea62e4fb21854229dfe526ef

> Remove confusing warning for disabled native BLAS
> -
>
> Key: SYSTEMML-1630
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1630
> Project: SystemML
>  Issue Type: Bug
>Affects Versions: SystemML 1.0
>Reporter: Matthias Boehm
>Assignee: Niketan Pansare
>Priority: Trivial
> Fix For: SystemML 1.0
>
>
> We should not raise BLAS-related warnings if it's explicitly or by default 
> disabled. Let's use this warning (after rewording) only if BLAS is enabled 
> but cannot be loaded.
> {code}
> WARN utils.NativeHelper: Using internal Java BLAS as native BLAS support the 
> configuration 'native.blas'=none.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Resolved] (SYSTEMML-1632) Support loading and saving models via mllearn

2017-05-25 Thread Niketan Pansare (JIRA)

 [ 
https://issues.apache.org/jira/browse/SYSTEMML-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Niketan Pansare resolved SYSTEMML-1632.
---
   Resolution: Fixed
Fix Version/s: SystemML 1.0

Fixed by the commit 
https://github.com/apache/incubator-systemml/commit/d69f3441c8243ddd13dd3da6aab9c2d5701c6e50

> Support loading and saving models via mllearn
> -
>
> Key: SYSTEMML-1632
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1632
> Project: SystemML
>  Issue Type: New Feature
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
> Fix For: SystemML 1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (SYSTEMML-1632) Support loading and saving models via mllearn

2017-05-25 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1632:
-

 Summary: Support loading and saving models via mllearn
 Key: SYSTEMML-1632
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1632
 Project: SystemML
  Issue Type: New Feature
Reporter: Niketan Pansare
Assignee: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (SYSTEMML-1611) Support fused weight update operators (similar to codegen on CP)

2017-05-18 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1611:
-

 Summary: Support fused weight update operators (similar to codegen 
on CP)
 Key: SYSTEMML-1611
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1611
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Created] (SYSTEMML-1610) Support alternative algorithms for CuDNN operators such as convolution

2017-05-18 Thread Niketan Pansare (JIRA)
Niketan Pansare created SYSTEMML-1610:
-

 Summary: Support alternative algorithms for CuDNN operators such 
as convolution
 Key: SYSTEMML-1610
 URL: https://issues.apache.org/jira/browse/SYSTEMML-1610
 Project: SystemML
  Issue Type: Sub-task
Reporter: Niketan Pansare
Assignee: Niketan Pansare


https://devblogs.nvidia.com/parallelforall/cudnn-v2-higher-performance-deep-learning-gpus/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)