[jira] [Closed] (SYSTEMML-2525) Add RESTful prediction service to deploy SystemML models
[ https://issues.apache.org/jira/browse/SYSTEMML-2525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare closed SYSTEMML-2525. - Resolution: Fixed Fix Version/s: SystemML 1.3 Closed by the commit [https://github.com/apache/systemml/commit/863c9d5cb1752b0e50140f5c6673968b57c2f9d0] > Add RESTful prediction service to deploy SystemML models > > > Key: SYSTEMML-2525 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2525 > Project: SystemML > Issue Type: Project >Reporter: Niketan Pansare >Priority: Major > Fix For: SystemML 1.3 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2525) Add RESTful prediction service to deploy SystemML models
Niketan Pansare created SYSTEMML-2525: - Summary: Add RESTful prediction service to deploy SystemML models Key: SYSTEMML-2525 URL: https://issues.apache.org/jira/browse/SYSTEMML-2525 Project: SystemML Issue Type: Project Reporter: Niketan Pansare -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-2514) Hadoop gives java.lang.UnsatisfiedLinkError
[
https://issues.apache.org/jira/browse/SYSTEMML-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797676#comment-16797676
]
Niketan Pansare commented on SYSTEMML-2514:
---
I usually use SystemML with Spark. To verify whether Spark is correctly
installed, please follow the below steps:
1. Java
===
The Java version should be > 1.8.
> java -version
Set JAVA_HOME environment variable and include %JAVA_HOME%\bin in the
environment variable PATH
> ls "%JAVA_HOME%"
2. Spark
Download and extract Spark from https://spark.apache.org/downloads.html,
> tar -xzf spark-2.1.0-bin-hadoop2.7.tgz
and set environment variable SPARK_HOME to point to the extracted directory.
Next step, install winutils:
- Download winutils.exe from
http://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe
- Place it in c:\winutils\bin
- Set environment variable HADOOP_HOME to point to c:\winutils
- Add c:\winutils\bin to the environment variable PATH.
- Finally, modify permission of hive directory that will be used by spark
> winutils.exe chmod 777 /tmp/hive
Finally, check if Spark is correctly installed:
> %SPARK_HOME%\bin\spark-shell
> %SPARK_HOME%\bin\pyspark
> Hadoop gives java.lang.UnsatisfiedLinkError
> ---
>
> Key: SYSTEMML-2514
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2514
> Project: SystemML
> Issue Type: Sub-task
> Components: Build
>Reporter: Janardhan
>Priority: Minor
> Attachments: hadoop_on_win.pdf
>
>
>
> {code:java}
> java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native
> Method)
> at
> org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:219)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.(RawLocalFileSystem.java:209)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.(ChecksumFileSystem.java:398)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1458)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1514)
> at
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:530)
> at
> org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:587)
> at
> org.apache.sysml.test.integration.functions.blocks.VariableTest.testVariablePassing1(VariableTest.java:80)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SYSTEMML-2523) Update SystemML to Support Spark 2.3.0
Niketan Pansare created SYSTEMML-2523: - Summary: Update SystemML to Support Spark 2.3.0 Key: SYSTEMML-2523 URL: https://issues.apache.org/jira/browse/SYSTEMML-2523 Project: SystemML Issue Type: Project Reporter: Niketan Pansare Spark 2.3 (released on February 28, 2018) has updated Antlr version, which throws warning every time we invoke SystemML. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SYSTEMML-2505) Support generation of DML script using Keras2DML and Caffe2DML
[
https://issues.apache.org/jira/browse/SYSTEMML-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niketan Pansare resolved SYSTEMML-2505.
---
Resolution: Fixed
Fixed by
https://github.com/apache/systemml/commit/7019f3bc805aaae67ef32e281cf99e26cbd26b29
> Support generation of DML script using Keras2DML and Caffe2DML
> --
>
> Key: SYSTEMML-2505
> URL: https://issues.apache.org/jira/browse/SYSTEMML-2505
> Project: SystemML
> Issue Type: Test
> Components: APIs
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>Priority: Major
> Fix For: SystemML 1.3
>
>
>
> {code}
> from keras.applications.vgg16 import VGG16
> keras_model = VGG16(weights="imagenet", pooling="max")
> from systemml.mllearn import Keras2DML
> sysml_model = Keras2DML(spark, keras_model, input_shape=(3,224,224),
> weights='weights_dir')
> sysml_model.set(test_algo='batch', train_algo='minibatch')
> print(sysml_model.get_training_script())
> print(sysml_model.get_prediction_script())
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SYSTEMML-2496) Skip MapReduce JUnit tests
[ https://issues.apache.org/jira/browse/SYSTEMML-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-2496. --- Resolution: Fixed Fix Version/s: SystemML 1.3 Fixed by the commit https://github.com/apache/systemml/commit/95bf8cfe6c0ae4f45b460a03c397203b7b354fc5 > Skip MapReduce JUnit tests > --- > > Key: SYSTEMML-2496 > URL: https://issues.apache.org/jira/browse/SYSTEMML-2496 > Project: SystemML > Issue Type: Test > Components: Test >Reporter: Niketan Pansare >Assignee: Niketan Pansare >Priority: Minor > Fix For: SystemML 1.3 > > > Since MR backend is in maintenance mode, skip MR tests to allow the JUnit > test to complete in less time. This will also enable us to turn on the larger > number of Spark tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2496) Skip MapReduce JUnit tests
Niketan Pansare created SYSTEMML-2496: - Summary: Skip MapReduce JUnit tests Key: SYSTEMML-2496 URL: https://issues.apache.org/jira/browse/SYSTEMML-2496 Project: SystemML Issue Type: Test Components: Test Reporter: Niketan Pansare Assignee: Niketan Pansare Since MR backend is in maintenance mode, skip MR tests to allow the JUnit test to complete in less time. This will also enable us to turn on the larger number of Spark tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SYSTEMML-880) Push-down loop structures in Python DSL
[
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539267#comment-16539267
]
Niketan Pansare edited comment on SYSTEMML-880 at 7/10/18 9:54 PM:
---
Pushdown of loop avoids invocation overhead and also enables additional
optimization. Here is a simple pyspark script that demonstrates the overhead:
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)
# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' + str(time.time()-t0))
# Total time with loop:2.50334095955
# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
t0 = time.time()
pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
totalTime = totalTime + time.time()-t0
print('Total time without loop:' + str(totalTime))
# Total time without loop:1008.73590732
{code}
One way to go about doing this is to define the boundaries using a decorator
(for example: parallelize) and try by first supporting simple expression and a
loop structure.
Few related links:
[https://greentreesnakes.readthedocs.io/en/latest/nodes.html#control-flow]
[https://eli.thegreenplace.net/2009/11/28/python-internals-working-with-python-asts]
was (Author: niketanpansare):
Pushdown of loop avoids invocation overhead and also enables additional
optimization. Here is a simple pyspark script that demonstrates the overhead:
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)
# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' + str(time.time()-t0))
# Total time with loop:2.50334095955
# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
t0 = time.time()
pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
totalTime = totalTime + time.time()-t0
print('Total time without loop:' + str(totalTime))
# Total time without loop:1008.73590732
{code}
One way to go about doing this is to define the boundaries using a decorator
(for example: parallelize) and try by first supporting simple expression and a
loop structure.
> Push-down loop structures in Python DSL
> ---
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
> Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SYSTEMML-880) Push-down loop structures in Python DSL
[
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16539267#comment-16539267
]
Niketan Pansare commented on SYSTEMML-880:
--
Pushdown of loop avoids invocation overhead and also enables additional
optimization. Here is a simple pyspark script that demonstrates the overhead:
{code:java}
from systemml import MLContext, dml
import numpy as np
import time
numpyX = np.ones((1,100))
ml = MLContext(sc)
# Execute with pushdown of loop
script_with_loop = dml('s = 0; for(i in 1:1000) { s = s + sum(X); } ')
t0 = time.time()
ml.execute(script_with_loop.input(X=numpyX).output('s')).get('s')
print('Total time with loop:' + str(time.time()-t0))
# Total time with loop:2.50334095955
# Execute without pushdown of loop
pythonS = 0
totalTime = 0
script_without_loop = dml('s = s + sum(X)').input(X=numpyX).output('s')
for i in range(1000):
t0 = time.time()
pythonS = ml.execute(script_without_loop.input(s=pythonS)).get('s')
totalTime = totalTime + time.time()-t0
print('Total time without loop:' + str(totalTime))
# Total time without loop:1008.73590732
{code}
One way to go about doing this is to define the boundaries using a decorator
(for example: parallelize) and try by first supporting simple expression and a
loop structure.
> Push-down loop structures in Python DSL
> ---
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
> Issue Type: Task
>Reporter: Niketan Pansare
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SYSTEMML-880) Push-down loop structures in Python DSL
[ https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517551#comment-16517551 ] Niketan Pansare commented on SYSTEMML-880: -- Since the below steps are not necessarily subtask, I am listing them: # Read the documentation and try the code: [http://apache.github.io/systemml/python-reference#matrix-class] # Understand the core problem of pushing down the loops: [http://apache.github.io/systemml/python-reference#dealing-with-the-loops] # Then, read the discussion in [PR 270|https://github.com/apache/systemml/pull/270] and understand why a [simple solution|https://github.com/niketanpansare/systemml/blob/6e4c0a15a597034010ce80439aaee87960e836e1/src/main/python/systemml/pushdown.py#L43-L47] is not preferred. # The solution that is preferred is to walk the [Python AST|https://docs.python.org/2/library/ast.html] of the given function and replace loops with PyDML script. > Push-down loop structures in Python DSL > --- > > Key: SYSTEMML-880 > URL: https://issues.apache.org/jira/browse/SYSTEMML-880 > Project: SystemML > Issue Type: Task >Reporter: Niketan Pansare >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2313) Add custom kernel for sparse matrix dense matrix multiplication
Niketan Pansare created SYSTEMML-2313: - Summary: Add custom kernel for sparse matrix dense matrix multiplication Key: SYSTEMML-2313 URL: https://issues.apache.org/jira/browse/SYSTEMML-2313 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2312) Add custom kernel for sparse matrix dense vector multiplication
Niketan Pansare created SYSTEMML-2312: - Summary: Add custom kernel for sparse matrix dense vector multiplication Key: SYSTEMML-2312 URL: https://issues.apache.org/jira/browse/SYSTEMML-2312 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SYSTEMML-1092) Integrate python test cases into junit test case
[ https://issues.apache.org/jira/browse/SYSTEMML-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425815#comment-16425815 ] Niketan Pansare edited comment on SYSTEMML-1092 at 4/4/18 4:43 PM: --- Subtasks for this JIRAs are: # Investigate whether it is possible to package pyspark dependencies into our maven test suite without spark installation. # If not, then come-up with simplest spark installation: ## Download spark and set SPARK_HOME ## Add pyspark as pip dependency # Enable tests [https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41] For reference, I believe PySpark test suite is kicked using the shell script: [https://github.com/apache/spark/blob/master/dev/run-tests-jenkins] was (Author: niketanpansare): Before closing this issue, one needs to investigate whether it is possible to package pyspark dependencies into our maven test suite. If not, the following solution needs to be discussed in the mailing list before adding SPARK_HOME requirement - The Python tests are currently disabled as it requires the environment variable SPARK_HOME to be set: https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41 - For reference, I believe PySpark test suite is kicked using the shell script: https://github.com/apache/spark/blob/master/dev/run-tests-jenkins > Integrate python test cases into junit test case > > > Key: SYSTEMML-1092 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1092 > Project: SystemML > Issue Type: Task >Reporter: Niketan Pansare >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-1092) Integrate python test cases into junit test case
[ https://issues.apache.org/jira/browse/SYSTEMML-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425815#comment-16425815 ] Niketan Pansare commented on SYSTEMML-1092: --- Before closing this issue, one needs to investigate whether it is possible to package pyspark dependencies into our maven test suite. If not, the following solution needs to be discussed in the mailing list before adding SPARK_HOME requirement - The Python tests are currently disabled as it requires the environment variable SPARK_HOME to be set: https://github.com/apache/systemml/blob/master/src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java#L41 - For reference, I believe PySpark test suite is kicked using the shell script: https://github.com/apache/spark/blob/master/dev/run-tests-jenkins > Integrate python test cases into junit test case > > > Key: SYSTEMML-1092 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1092 > Project: SystemML > Issue Type: Task >Reporter: Niketan Pansare >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SYSTEMML-2165) Support ARC eviction policy for the GPU backend
Niketan Pansare created SYSTEMML-2165: - Summary: Support ARC eviction policy for the GPU backend Key: SYSTEMML-2165 URL: https://issues.apache.org/jira/browse/SYSTEMML-2165 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare Link to the paper: [https://dbs.uni-leipzig.de/file/ARC.pdf|https://dbs.uni-leipzig.de/file/ARC.pdf] There might be patent related issues with ARC algorithm. If so, we cannot support it in SystemML: [https://news.ycombinator.com/item?id=9898430|https://news.ycombinator.com/item?id=9898430] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SYSTEMML-1975) Fix GPU tests for single precision
[
https://issues.apache.org/jira/browse/SYSTEMML-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221207#comment-16221207
]
Niketan Pansare commented on SYSTEMML-1975:
---
Following tests are failing for double precision:
{code}
Failed tests:
MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:267
Relative error(1.00) is more than threshold (0.10). Expected =
-0.002456, Actual = 0.00, differed at [0, 0]
MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
expected:<0.0> but was:<1.929108490094312>
{code}
My guess is there is likely a bug caught by these two tests and other tests are
likely related to precision.
> Fix GPU tests for single precision
> --
>
> Key: SYSTEMML-1975
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1975
> Project: SystemML
> Issue Type: Bug
>Reporter: Niketan Pansare
>
> The current GPU tests for single precision compares the results of double
> precision operation on CPU with single precision operation on GPU. So, it is
> difficult to compare the results. When we support single precision on CPU, we
> can rework the GPU tests to fix the following tests:
> {code}
> Failed tests:
>
> NeuralNetworkOpTests.testConv2dBackwardFilter:324->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(0.12) is more than threshold (0.10). Expected =
> 25311.00, Actual = 25310.687500, differed at [0, 50]
>
> MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(1.00) is more than threshold (0.10). Expected =
> -0.002456, Actual = 0.00, differed at [0, 0]
>
> MatrixMultiplicationOpTest.matrixMatrixTest4:209->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(0.24) is more than threshold (0.10). Expected =
> 0.573809, Actual = 0.573822, differed at [5, 127]
>
> MatrixMultiplicationOpTest.commonCaseDLMatrixMatrixTest2:137->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(0.13) is more than threshold (0.10). Expected =
> -0.922950, Actual = -0.922939, differed at [0, 13]
>
> MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
> expected:<0.0> but was:<1.9291085004806519>
>
> ScalarMatrixElementwiseOpTests.testModulusLeftScalar:167->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(1.00) is more than threshold (0.10). Expected =
> 0.078125, Actual = 0.00, differed at [1500, 22]
>
> ScalarMatrixElementwiseOpTests.testIntegerDivideLeftScalar:157->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
> Relative error(0.015873) is more than threshold (0.10). Expected =
> 63.00, Actual = 64.00, differed at [1500, 22]
> {code}
> [~nakul02]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1969) Support single-precision on GPU
[ https://issues.apache.org/jira/browse/SYSTEMML-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare closed SYSTEMML-1969. - Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed in the commit https://github.com/apache/systemml/commit/abbffc55ef8f47f10b6e59b0ae5e1f311f4a8f3e > Support single-precision on GPU > --- > > Key: SYSTEMML-1969 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1969 > Project: SystemML > Issue Type: New Feature >Reporter: Niketan Pansare >Assignee: Niketan Pansare > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1975) Fix GPU tests for single precision
Niketan Pansare created SYSTEMML-1975:
-
Summary: Fix GPU tests for single precision
Key: SYSTEMML-1975
URL: https://issues.apache.org/jira/browse/SYSTEMML-1975
Project: SystemML
Issue Type: Bug
Reporter: Niketan Pansare
The current GPU tests for single precision compares the results of double
precision operation on CPU with single precision operation on GPU. So, it is
difficult to compare the results. When we support single precision on CPU, we
can rework the GPU tests to fix the following tests:
{code}
Failed tests:
NeuralNetworkOpTests.testConv2dBackwardFilter:324->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(0.12) is more than threshold (0.10). Expected =
25311.00, Actual = 25310.687500, differed at [0, 50]
MatrixMultiplicationOpTest.matrixMatrixTest2:161->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(1.00) is more than threshold (0.10). Expected =
-0.002456, Actual = 0.00, differed at [0, 0]
MatrixMultiplicationOpTest.matrixMatrixTest4:209->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(0.24) is more than threshold (0.10). Expected =
0.573809, Actual = 0.573822, differed at [5, 127]
MatrixMultiplicationOpTest.commonCaseDLMatrixMatrixTest2:137->assertMatrixMultiplication:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(0.13) is more than threshold (0.10). Expected =
-0.922950, Actual = -0.922939, differed at [0, 13]
MatrixMultiplicationOpTest.transposeSelfMatrixMultiply:235->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:272
expected:<0.0> but was:<1.9291085004806519>
ScalarMatrixElementwiseOpTests.testModulusLeftScalar:167->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(1.00) is more than threshold (0.10). Expected =
0.078125, Actual = 0.00, differed at [1500, 22]
ScalarMatrixElementwiseOpTests.testIntegerDivideLeftScalar:157->runScalarMatrixElementWiseTests:261->GPUTests.assertEqualObjects:367->GPUTests.assertEqualMatrices:269
Relative error(0.015873) is more than threshold (0.10). Expected =
63.00, Actual = 64.00, differed at [1500, 22]
{code}
[~nakul02]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SYSTEMML-1969) Support single-precision on GPU
Niketan Pansare created SYSTEMML-1969: - Summary: Support single-precision on GPU Key: SYSTEMML-1969 URL: https://issues.apache.org/jira/browse/SYSTEMML-1969 Project: SystemML Issue Type: New Feature Reporter: Niketan Pansare Assignee: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1965) Refactor nn layers to move the computation in forward/backward function known at compile time to init function
[
https://issues.apache.org/jira/browse/SYSTEMML-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208004#comment-16208004
]
Niketan Pansare commented on SYSTEMML-1965:
---
My two cents: I think both engine improvement as well as good DML practice go
hand in hand. If `init` is not the right place, then we should add
`compute_height` and `compute_width` functions. Having a loop-invariant code
(as part of reusable function) in the for loop is not advisable even if we fix
the constant propagation portion of the engine.
> Refactor nn layers to move the computation in forward/backward function known
> at compile time to init function
> --
>
> Key: SYSTEMML-1965
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1965
> Project: SystemML
> Issue Type: Bug
>Reporter: Niketan Pansare
>
> Ideally, we should move the computation known at compile time to init layer,
> rather than keep in the forward function. This reduces recompilation time and
> also potentially unnecessary instructions. Here is an example snippet from
> our conv2d layer:
> {code}
> Hout = as.integer(floor((Hin + 2*padh - Hf)/strideh + 1))
> Wout = as.integer(floor((Win + 2*padw - Wf)/stridew + 1))
> {code}
> [~prithvi_r_s] [~dusenberrymw] [~reinwald] do you have any comments or
> concerns ?
> [~dusenberrymw] do you have free cycles to take this over ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (SYSTEMML-1965) Refactor nn layers to move the computation in forward/backward function known at compile time to init function
Niketan Pansare created SYSTEMML-1965:
-
Summary: Refactor nn layers to move the computation in
forward/backward function known at compile time to init function
Key: SYSTEMML-1965
URL: https://issues.apache.org/jira/browse/SYSTEMML-1965
Project: SystemML
Issue Type: Bug
Reporter: Niketan Pansare
Ideally, we should move the computation known at compile time to init layer,
rather than keep in the forward function. This reduces recompilation time and
also potentially unnecessary instructions. Here is an example snippet from our
conv2d layer:
{code}
Hout = as.integer(floor((Hin + 2*padh - Hf)/strideh + 1))
Wout = as.integer(floor((Win + 2*padw - Wf)/stridew + 1))
{code}
[~prithvi_r_s] [~dusenberrymw] [~reinwald] do you have any comments or concerns
?
[~dusenberrymw] do you have free cycles to take this over ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs
[
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niketan Pansare updated SYSTEMML-1962:
--
Description:
The end goal of this JIRA is to support model selection facility similar to
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
Currently, we support model selection using MLPipeline's cross-validator. For
example: please replace `from pyspark.ml.classification import
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in
the example
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
However, this invokes k-seperate and independent mlcontext calls. This PR
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly
bayesian optimization which like mllearn has methods `fit` and `predict`. These
methods internally generate a script that wraps the external script with a
`parfor` when the fit method is called. For example:
{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}
would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
C = CVals[i, 1]
reg = 1 / C
# SVM script
}
{code}
This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol,
double reg, double maxiter) returns (matrix[double] w) {
if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary
class classifier without at least 2 rows")
check_min = min(Y)
w = t(cbind(t(w), t(extra_model_params)))
}
{code}
2. Adding two new java classes in the package `org.apache.sysml.api.ml` called
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]`
and `GridSearchClassifierCVModel` which `extends
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you
will have to implement the abstract methods: fit and transform respectively.
3. Add a python class GridSearchClassifierCV that invokes the above java
classes.
For more details on step 2 and step 3, please read the design documentation of
mllearn API:
https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42
[~dusenberrymw] may be, this can be part of
https://issues.apache.org/jira/browse/SYSTEMML-1159
was:
The end goal of this JIRA is to support model selection facility similar to
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
Currently, we support model selection using MLPipeline's cross-validator. For
example: please replace `from pyspark.ml.classification import
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in
the example
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
However, this invokes k-seperate and independent mlcontext calls. This PR
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly
bayesian optimization which like mllearn has methods `fit` and `predict`. These
methods internally generate a script that wraps the external script with a
`parfor` when the fit method is called. For example:
{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}
would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
C = CVals[i, 1]
# SVM script
}
{code}
This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol,
double reg, double maxiter) returns (matrix[double] w) {
if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary
class classifier without at least 2 rows")
check_min = min(Y)
w = t(cbind(t(w), t(extra_model_params)))
}
{code}
2. Adding two new java classes in the package `org.apache.sysml.api.ml` called
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]`
and `GridSearchClassifierCVModel` which `extends
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you
will have to implement the abstract methods: fit and transform respectively.
3. Add a python class GridSearchClassifierCV that invokes the above java
classes.
For more details on step 2 and step 3, please read the design documentation of
mllearn API:
[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs
[
https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niketan Pansare updated SYSTEMML-1962:
--
Description:
The end goal of this JIRA is to support model selection facility similar to
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
Currently, we support model selection using MLPipeline's cross-validator. For
example: please replace `from pyspark.ml.classification import
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in
the example
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
However, this invokes k-seperate and independent mlcontext calls. This PR
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly
bayesian optimization which like mllearn has methods `fit` and `predict`. These
methods internally generate a script that wraps the external script with a
`parfor` when the fit method is called. For example:
{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}
would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
C = CVals[i, 1]
# SVM script
}
{code}
This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol,
double reg, double maxiter) returns (matrix[double] w) {
if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary
class classifier without at least 2 rows")
check_min = min(Y)
w = t(cbind(t(w), t(extra_model_params)))
}
{code}
2. Adding two new java classes in the package `org.apache.sysml.api.ml` called
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]`
and `GridSearchClassifierCVModel` which `extends
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you
will have to implement the abstract methods: fit and transform respectively.
3. Add a python class GridSearchClassifierCV that invokes the above java
classes.
For more details on step 2 and step 3, please read the design documentation of
mllearn API:
https://github.com/apache/systemml/blob/master/src/main/scala/org/apache/sysml/api/ml/BaseSystemMLClassifier.scala#L42
[~dusenberrymw] may be, this can be part of
https://issues.apache.org/jira/browse/SYSTEMML-1159
was:
The end goal of this JIRA is to support model selection facility similar to
[http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection].
Currently, we support model selection using MLPipeline's cross-validator. For
example: please replace `from pyspark.ml.classification import
LogisticRegression` with `from systemml.mllearn import LogisticRegression` in
the example
http://spark.apache.org/docs/2.1.1/ml-tuning.html#example-model-selection-via-cross-validation.
However, this invokes k-seperate and independent mlcontext calls. This PR
proposes to add a new class `GridSearchCV`, `RandomizedSearchCV` and possibly
bayesian optimization which like mllearn has methods `fit` and `predict`. These
methods internally generate a script that wraps the external script with a
`parfor` when the fit method is called. For example:
{code}
from sklearn import datasets
from systemml.mllearn import GridSearchCV, SVM
iris = datasets.load_iris()
parameters = {'C':[1, 10]}
svm = SVM()
clf = GridSearchClassifierCV(svm, parameters)
clf.fit(iris.data, iris.target)
{code}
would execute the script:
{code}
CVals = matrix("1; 10", rows=2, cols=1)
parfor(i in seq(1, nrow(CVals))) {
C = CVals[i, 1]
# SVM script
}
{code}
This will require:
1. Functionization of the script (for example: L2SVM)
{code}
svm = function(matrix[double] X, matrix[double] Y, double icpt, double tol,
double reg, double maxiter) returns (matrix[double] w) {
if(nrow(X) < 2)
stop("Stopping due to invalid inputs: Not possible to learn a binary
class classifier without at least 2 rows")
check_min = min(Y)
w = t(cbind(t(w), t(extra_model_params)))
}
{code}
2. Adding two new java classes in the package `org.apache.sysml.api.ml` called
`GridSearchClassifierCV` which extends `Estimator[GridSearchClassifierCVModel]`
and `GridSearchClassifierCVModel` which `extends
Model[GridSearchClassifierCVModel] with BaseSystemMLClassifierModel`. Then you
will have to implement the abstract methods: fit and transform respectively.
3. Add a python class GridSearchClassifierCV that invokes the above java
classes.
For more details on step 2 and step 3, please read the design documentation of
mllearn API:
[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs
[ https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare updated SYSTEMML-1962: -- Description: The end goal of this JIRA is to support model selection facility similar to [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection]. (was: The end goal of this JIRA is to support model selection facility similar to [scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection).) > Support model-selection via mllearn APIs > > > Key: SYSTEMML-1962 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1962 > Project: SystemML > Issue Type: New Feature >Reporter: Niketan Pansare > > The end goal of this JIRA is to support model selection facility similar to > [http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1962) Support model-selection via mllearn APIs
[ https://issues.apache.org/jira/browse/SYSTEMML-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare updated SYSTEMML-1962: -- Description: The end goal of this JIRA is to support model selection facility similar to [scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection). > Support model-selection via mllearn APIs > > > Key: SYSTEMML-1962 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1962 > Project: SystemML > Issue Type: New Feature >Reporter: Niketan Pansare > > The end goal of this JIRA is to support model selection facility similar to > [scikit-learn](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.model_selection). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1962) Support model-selection via mllearn APIs
Niketan Pansare created SYSTEMML-1962: - Summary: Support model-selection via mllearn APIs Key: SYSTEMML-1962 URL: https://issues.apache.org/jira/browse/SYSTEMML-1962 Project: SystemML Issue Type: New Feature Reporter: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1940) Modify the default directory of performance test to avoid maven compilation issue
[ https://issues.apache.org/jira/browse/SYSTEMML-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-1940. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed in the commit https://github.com/apache/systemml/commit/355373990c478293090c90557383ab24c2c01c6b > Modify the default directory of performance test to avoid maven compilation > issue > - > > Key: SYSTEMML-1940 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1940 > Project: SystemML > Issue Type: Test >Reporter: Niketan Pansare >Assignee: Niketan Pansare > Fix For: SystemML 1.0 > > > Currently, the performance test generates it data in scripts folder. This > causes maven to take too long for compilation as it tries to create jar with > all the data. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SYSTEMML-1940) Modify the default directory of performance test to avoid maven compilation issue
[ https://issues.apache.org/jira/browse/SYSTEMML-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare reassigned SYSTEMML-1940: - Assignee: Niketan Pansare > Modify the default directory of performance test to avoid maven compilation > issue > - > > Key: SYSTEMML-1940 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1940 > Project: SystemML > Issue Type: Test >Reporter: Niketan Pansare >Assignee: Niketan Pansare > > Currently, the performance test generates it data in scripts folder. This > causes maven to take too long for compilation as it tries to create jar with > all the data. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SYSTEMML-1500) Add missing loss layers to Caffe2DML
[ https://issues.apache.org/jira/browse/SYSTEMML-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare reassigned SYSTEMML-1500: - Assignee: Niketan Pansare > Add missing loss layers to Caffe2DML > > > Key: SYSTEMML-1500 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1500 > Project: SystemML > Issue Type: Sub-task >Reporter: Niketan Pansare >Assignee: Niketan Pansare > > Multinomial Logistic Loss > Infogain Loss - a generalization of MultinomialLogisticLossLayer. > Softmax with Loss - computes the multinomial logistic loss of the softmax of > its inputs. It’s conceptually identical to a softmax layer followed by a > multinomial logistic loss layer, but provides a more numerically stable > gradient. > Sum-of-Squares / Euclidean - computes the sum of squares of differences of > its two inputs, 12N∑Ni=1∥x1i−x2i∥2212N∑i=1N‖xi1−xi2‖22. > Hinge / Margin - The hinge loss layer computes a one-vs-all hinge (L1) or > squared hinge loss (L2). > Sigmoid Cross-Entropy Loss - computes the cross-entropy (logistic) loss, > often used for predicting targets interpreted as probabilities. > Accuracy / Top-k layer - scores the output as an accuracy with respect to > target – it is not actually a loss and has no backward step. > Contrastive Loss -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1500) Add missing loss layers to Caffe2DML
[ https://issues.apache.org/jira/browse/SYSTEMML-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184560#comment-16184560 ] Niketan Pansare commented on SYSTEMML-1500: --- - Euclidean loss added in the commit: https://github.com/apache/systemml/commit/61dcc85e48a390c1bb63ee4c42aad9a3fade7d06 > Add missing loss layers to Caffe2DML > > > Key: SYSTEMML-1500 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1500 > Project: SystemML > Issue Type: Sub-task >Reporter: Niketan Pansare > > Multinomial Logistic Loss > Infogain Loss - a generalization of MultinomialLogisticLossLayer. > Softmax with Loss - computes the multinomial logistic loss of the softmax of > its inputs. It’s conceptually identical to a softmax layer followed by a > multinomial logistic loss layer, but provides a more numerically stable > gradient. > Sum-of-Squares / Euclidean - computes the sum of squares of differences of > its two inputs, 12N∑Ni=1∥x1i−x2i∥2212N∑i=1N‖xi1−xi2‖22. > Hinge / Margin - The hinge loss layer computes a one-vs-all hinge (L1) or > squared hinge loss (L2). > Sigmoid Cross-Entropy Loss - computes the cross-entropy (logistic) loss, > often used for predicting targets interpreted as probabilities. > Accuracy / Top-k layer - scores the output as an accuracy with respect to > target – it is not actually a loss and has no backward step. > Contrastive Loss -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1930) Fix protoc compilation error for TensorFlow proto files and restore Tensorboard functionality
Niketan Pansare created SYSTEMML-1930: - Summary: Fix protoc compilation error for TensorFlow proto files and restore Tensorboard functionality Key: SYSTEMML-1930 URL: https://issues.apache.org/jira/browse/SYSTEMML-1930 Project: SystemML Issue Type: Bug Reporter: Niketan Pansare To use Tensorboard in Caffe2DML, we were compiling src/main/proto/tensorflow/event.proto and src/main/proto/tensorflow/summary.proto files using protoc 3.0 mvn plugin (please see [https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369]). These proto files match the ones that are available in TensorFlow repository. To compile SystemML on Power, we have to switch to a newer version of protoc mvn plugin which now only supports 3.4 version (not 3.0). The problem is that the above mentioned proto files are giving compilation error. So, for time being, we are removed tensorboard integration from SystemML in commit: https://github.com/apache/systemml/commit/a4962f39a065bce89dd1c45635e318b9440e5491#diff-600376dffeb79835ede4a0b285078036L369 This task involves debugging compilation error on protoc 3.4 version and restoring the above commit. [~freiss] [~nakul02] [~reinwald] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1923) Support sinh, cosh and tanh as built-in functions
Niketan Pansare created SYSTEMML-1923: - Summary: Support sinh, cosh and tanh as built-in functions Key: SYSTEMML-1923 URL: https://issues.apache.org/jira/browse/SYSTEMML-1923 Project: SystemML Issue Type: New Feature Reporter: Niketan Pansare Assignee: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166772#comment-16166772 ] Niketan Pansare commented on SYSTEMML-1907: --- thanks [~gweidner] :) > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-976) Add explain and stats option to Python DSL
[ https://issues.apache.org/jira/browse/SYSTEMML-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158911#comment-16158911 ] Niketan Pansare commented on SYSTEMML-976: -- Yes, absolutely. Please let me know if you need any help. Another interesting JIRA is https://issues.apache.org/jira/browse/SYSTEMML-880 :) > Add explain and stats option to Python DSL > -- > > Key: SYSTEMML-976 > URL: https://issues.apache.org/jira/browse/SYSTEMML-976 > Project: SystemML > Issue Type: Improvement > Components: APIs >Reporter: Niketan Pansare > Labels: Hacktoberfest, starter > > This would allow users to see the plans and statistics for a given evaluation > of Python DSL. Since this is a starter task, I would give some hints: > 1. Extend "def eval(outputs, outputDF=False, execute=True):" method to accept > explain and statistics options and pass them before "results = > matrix.ml.execute(matrix.script)". > Here are the relevant links: > https://github.com/apache/incubator-systemml/blob/master/src/main/python/systemml/defmatrix.py#L290 > https://github.com/apache/incubator-systemml/blob/master/src/main/python/systemml/mlcontext.py#L223 > 2. Expose these options through user facing APIs: eval() and methods that > invoke eval (such as toPandas, toNumPyArray, toDataFrame, etc) > 3. Update the documentation: > https://apache.github.io/incubator-systemml/beginners-guide-python -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-984) Add mllearn and scala wrappers for PCA
[ https://issues.apache.org/jira/browse/SYSTEMML-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144306#comment-16144306 ] Niketan Pansare commented on SYSTEMML-984: -- If we implement scalable PCA and do reasonable accuracy/performance testing, then I would recommend that we add scala/python wrapper :) > Add mllearn and scala wrappers for PCA > -- > > Key: SYSTEMML-984 > URL: https://issues.apache.org/jira/browse/SYSTEMML-984 > Project: SystemML > Issue Type: Task > Components: APIs >Reporter: Niketan Pansare > Labels: Hacktoberfest, starter > > See > https://apache.github.io/incubator-systemml/algorithms-matrix-factorization.html#principle-component-analysis > for usage -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1870) Include avro dependency in the bin and standalone-jar artifacts
Niketan Pansare created SYSTEMML-1870: - Summary: Include avro dependency in the bin and standalone-jar artifacts Key: SYSTEMML-1870 URL: https://issues.apache.org/jira/browse/SYSTEMML-1870 Project: SystemML Issue Type: Bug Reporter: Niketan Pansare Assignee: Deron Eriksson [~prithvi_r_s] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1808) Visualize Hop DAG for explaining the optimizer
Niketan Pansare created SYSTEMML-1808: - Summary: Visualize Hop DAG for explaining the optimizer Key: SYSTEMML-1808 URL: https://issues.apache.org/jira/browse/SYSTEMML-1808 Project: SystemML Issue Type: New Feature Reporter: Niketan Pansare Assignee: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1798) provide getScriptExecutionString and getScriptString for Python Script object
[ https://issues.apache.org/jira/browse/SYSTEMML-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-1798. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed in the commit https://github.com/apache/systemml/commit/8f412ac5cd2a06453a55efbe4afe60e838cd33b1 > provide getScriptExecutionString and getScriptString for Python Script object > - > > Key: SYSTEMML-1798 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1798 > Project: SystemML > Issue Type: Sub-task > Components: APIs >Reporter: Berthold Reinwald >Assignee: Niketan Pansare > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1802) Fix build failure for distribution profile on Windows
[ https://issues.apache.org/jira/browse/SYSTEMML-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-1802. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed by the commit https://github.com/apache/systemml/commit/11b689d49b78a73f44b5944cfe0d14bb7c05c3a7 > Fix build failure for distribution profile on Windows > - > > Key: SYSTEMML-1802 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1802 > Project: SystemML > Issue Type: Bug >Reporter: Niketan Pansare >Assignee: Niketan Pansare > Fix For: SystemML 1.0 > > > To test please run `mvn package -P distribution` on Windows machine -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1802) Fix build failure for distribution profile on Windows
Niketan Pansare created SYSTEMML-1802: - Summary: Fix build failure for distribution profile on Windows Key: SYSTEMML-1802 URL: https://issues.apache.org/jira/browse/SYSTEMML-1802 Project: SystemML Issue Type: Bug Reporter: Niketan Pansare Assignee: Niketan Pansare To test please run `mvn package -P distribution` on Windows machine -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (SYSTEMML-1798) provide getScriptExecutionString and getScriptString for Python Script object
[ https://issues.apache.org/jira/browse/SYSTEMML-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare reassigned SYSTEMML-1798: - Assignee: Niketan Pansare > provide getScriptExecutionString and getScriptString for Python Script object > - > > Key: SYSTEMML-1798 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1798 > Project: SystemML > Issue Type: Sub-task > Components: APIs >Reporter: Berthold Reinwald >Assignee: Niketan Pansare > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1796) Provide fine-grained statistics for SystemML
[ https://issues.apache.org/jira/browse/SYSTEMML-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-1796. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed by the commit https://github.com/apache/systemml/commit/648eb21d66f9cd8727090cdf950986765a7e6ee8 > Provide fine-grained statistics for SystemML > > > Key: SYSTEMML-1796 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1796 > Project: SystemML > Issue Type: New Feature >Reporter: Niketan Pansare >Assignee: Niketan Pansare > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1796) Provide fine-grained statistics for SystemML
Niketan Pansare created SYSTEMML-1796: - Summary: Provide fine-grained statistics for SystemML Key: SYSTEMML-1796 URL: https://issues.apache.org/jira/browse/SYSTEMML-1796 Project: SystemML Issue Type: New Feature Reporter: Niketan Pansare Assignee: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1793) Support left and right indexing on GPU
Niketan Pansare created SYSTEMML-1793: - Summary: Support left and right indexing on GPU Key: SYSTEMML-1793 URL: https://issues.apache.org/jira/browse/SYSTEMML-1793 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare Assignee: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server
[ https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088255#comment-16088255 ] Niketan Pansare commented on SYSTEMML-1695: --- Personally, I would go through https://github.com/apache/systemml/blob/master/scripts/nn/examples/Example%20-%20MNIST%20LeNet.ipynb and think how can we modify https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet.dml#L190-L198 to enable HogWild-style training. > Create design/investigation document for parameter server > - > > Key: SYSTEMML-1695 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1695 > Project: SystemML > Issue Type: Sub-task >Reporter: Niketan Pansare >Assignee: Janardhan > > Here are relevant links: > - https://issues.apache.org/jira/browse/SPARK-4590 > - https://issues.apache.org/jira/browse/SPARK-6932 > - > https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit > This document need not cover how to integrate a parameter server in SystemML > (which comes later), but should discuss following questions: > 1. Should we integrate existing parameter server implementation or build one > from scratch ? What are the pros and cons of each approaches ? For example: > https://github.com/rjagerman/glint > 2. If we plan to integrate existing parameter server implementation, what is > the interface ? (for exampe: update(matrix, new matrix) or update(matrix, > delta)). > 3. Can parameter server help in ML algorithms ? Some experiments might be > helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-979) Add support for bayesian optimization
[ https://issues.apache.org/jira/browse/SYSTEMML-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081284#comment-16081284 ] Niketan Pansare commented on SYSTEMML-979: -- Also, notified Sasha regarding this JIRA :) > Add support for bayesian optimization > - > > Key: SYSTEMML-979 > URL: https://issues.apache.org/jira/browse/SYSTEMML-979 > Project: SystemML > Issue Type: New Feature > Components: Algorithms >Reporter: Niketan Pansare >Assignee: Janardhan > > Main paper: > https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf > supplements: > * selecting the next point to evaluate > [Jasper|http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf] > * sobol sequence generator [P Bratley, BL > Fox|http://dl.acm.org/citation.cfm?id=214372] > * Handling Sparsity via the Horseshoe [Carlos M. > Carvalho|http://ftp.isds.duke.edu/WorkingPapers/09-03.pdf] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1695) Create design/investigation document for parameter server
[ https://issues.apache.org/jira/browse/SYSTEMML-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050871#comment-16050871 ] Niketan Pansare commented on SYSTEMML-1695: --- This is a good start. Once you are ready with your draft, you may want to send it over the mailing list for feedback from other committers :) > Create design/investigation document for parameter server > - > > Key: SYSTEMML-1695 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1695 > Project: SystemML > Issue Type: Sub-task >Reporter: Niketan Pansare > > Here are relevant links: > - https://issues.apache.org/jira/browse/SPARK-4590 > - https://issues.apache.org/jira/browse/SPARK-6932 > - > https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit > This document need not cover how to integrate a parameter server in SystemML > (which comes later), but should discuss following questions: > 1. Should we integrate existing parameter server implementation or build one > from scratch ? What are the pros and cons of each approaches ? For example: > https://github.com/rjagerman/glint > 2. If we plan to integrate existing parameter server implementation, what is > the interface ? (for exampe: update(matrix, new matrix) or update(matrix, > delta)). > 3. Can parameter server help in ML algorithms ? Some experiments might be > helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1695) Create design/investigation document for parameter server
Niketan Pansare created SYSTEMML-1695: - Summary: Create design/investigation document for parameter server Key: SYSTEMML-1695 URL: https://issues.apache.org/jira/browse/SYSTEMML-1695 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare Here are relevant links: - https://issues.apache.org/jira/browse/SPARK-4590 - https://issues.apache.org/jira/browse/SPARK-6932 - https://docs.google.com/document/d/1SX3nkmF41wFXAAIr9BgqvrHSS5mW362fJ7roBXJm06o/edit This document need not cover how to integrate a parameter server in SystemML (which comes later), but should discuss following questions: 1. Should we integrate existing parameter server implementation or build one from scratch ? What are the pros and cons of each approaches ? For example: https://github.com/rjagerman/glint 2. If we plan to integrate existing parameter server implementation, what is the interface ? (for exampe: update(matrix, new matrix) or update(matrix, delta)). 3. Can parameter server help in ML algorithms ? Some experiments might be helpful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-739) Explore model-parallel constructs in DML
[ https://issues.apache.org/jira/browse/SYSTEMML-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043214#comment-16043214 ] Niketan Pansare commented on SYSTEMML-739: -- Hi Janardhan, Welcome to SystemML community !! Here is a light-weight parameter server implementation that you can look at: https://github.com/dmlc/ps-lite and the corresponding paper: https://www.cs.cmu.edu/~muli/file/parameter_server_osdi14.pdf To integrate a parameter server with SystemML, you might want to look into how SystemML's bufferpool work and how one can extend DML to support model parallelism. You can begin by reading the papers http://www.vldb.org/pvldb/vol9/p1425-boehm.pdf and https://arxiv.org/pdf/1503.06384.pdf. Thanks, Niketan. > Explore model-parallel constructs in DML > > > Key: SYSTEMML-739 > URL: https://issues.apache.org/jira/browse/SYSTEMML-739 > Project: SystemML > Issue Type: Task >Reporter: Niketan Pansare > > An example of such construct is providing access to the parameter server. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1661) Builtin functions bias_add and bias_multiply not documented
[ https://issues.apache.org/jira/browse/SYSTEMML-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niketan Pansare resolved SYSTEMML-1661. --- Resolution: Fixed Fix Version/s: SystemML 1.0 Resolved by the commit https://github.com/apache/incubator-systemml/commit/88f4a468f48081031d926d917ebc4f3e9014fc7f > Builtin functions bias_add and bias_multiply not documented > --- > > Key: SYSTEMML-1661 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1661 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Niketan Pansare >Priority: Minor > Fix For: SystemML 1.0 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (SYSTEMML-1630) Remove confusing warning for disabled native BLAS
[
https://issues.apache.org/jira/browse/SYSTEMML-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Niketan Pansare resolved SYSTEMML-1630.
---
Resolution: Fixed
Fix Version/s: SystemML 1.0
Fixed by the commit
https://github.com/apache/incubator-systemml/commit/fd41f0253810a8d0ea62e4fb21854229dfe526ef
> Remove confusing warning for disabled native BLAS
> -
>
> Key: SYSTEMML-1630
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1630
> Project: SystemML
> Issue Type: Bug
>Affects Versions: SystemML 1.0
>Reporter: Matthias Boehm
>Assignee: Niketan Pansare
>Priority: Trivial
> Fix For: SystemML 1.0
>
>
> We should not raise BLAS-related warnings if it's explicitly or by default
> disabled. Let's use this warning (after rewording) only if BLAS is enabled
> but cannot be loaded.
> {code}
> WARN utils.NativeHelper: Using internal Java BLAS as native BLAS support the
> configuration 'native.blas'=none.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (SYSTEMML-1611) Support fused weight update operators (similar to codegen on CP)
Niketan Pansare created SYSTEMML-1611: - Summary: Support fused weight update operators (similar to codegen on CP) Key: SYSTEMML-1611 URL: https://issues.apache.org/jira/browse/SYSTEMML-1611 Project: SystemML Issue Type: Sub-task Reporter: Niketan Pansare -- This message was sent by Atlassian JIRA (v6.3.15#6346)
