[jira] [Closed] (SYSTEMML-1217) Perftest 0.12 release and related improvements
[ https://issues.apache.org/jira/browse/SYSTEMML-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1217. --- > Perftest 0.12 release and related improvements > -- > > Key: SYSTEMML-1217 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1217 > Project: SystemML > Issue Type: Umbrella >Affects Versions: SystemML 0.12 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > Attachments: runMultiLogReg.sh, > runMultiLogReg_10M_1k_sparse_k150_v11.out, > runMultiLogReg_10M_1k_sparse_k150_v12.out, times_All_8g_rc2.txt, > times_Binomial_80g_rc1.txt, times_Binomial_80g_rc2.txt, > times_Binomial_80g_v11.txt, times_Binomial_8g_rc1.txt, > times_Binomial_8g_v11.txt, times_Clustering_80g_rc1.txt, > times_Clustering_80g_v11.txt, times_Clustering_8g_rc1.txt, > times_Clustering_8g_v11.txt, times_Multinomial_80g_bayes_sparse_rc1.txt, > times_Multinomial_80g_mlreg_dense_rc1.txt, > times_Multinomial_80g_mlreg_sparse_rc1.txt, > times_Multinomial_80g_rc1_subset.txt, > times_Multinomial_80g_v11_mlreg_dense.txt, > times_Multinomial_80g_v11_mlreg_dense_run2.txt, > times_Multinomial_80g_v11_mlreg_dense_run3.txt, > times_Multinomial_80g_v11_mlreg_sparse.txt, > times_Multinomial_8g_bayes_predict.txt, > times_Multinomial_8g_bayes_predict_v11.txt, times_Multinomial_8g_rc1.txt, > times_Multinomial_8g_v11.txt, times_Regression_80g_rc1.txt, > times_Regression_80g_rc2.txt, times_Regression_80g_v11.txt, > times_Regression_8g_rc1.txt, times_Regression_8g_v11.txt, > times_Stats_80g_rc1.txt, times_Stats_80g_v11.txt > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1217) Perftest 0.12 release and related improvements
[ https://issues.apache.org/jira/browse/SYSTEMML-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1217. - Resolution: Not A Problem Fix Version/s: Not Applicable The time variations were due to script parameters (e.g., k for number of classes) which has been addressed in commits [687e19c|https://github.com/apache/systemml/commit/687e19c5529b371f937a9ecb60075373f36fd4d6] and [918e579|https://github.com/apache/systemml/commit/918e57937dc6476ae27744a37f400e0a5e0997e6]. > Perftest 0.12 release and related improvements > -- > > Key: SYSTEMML-1217 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1217 > Project: SystemML > Issue Type: Umbrella >Affects Versions: SystemML 0.12 >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > Attachments: runMultiLogReg.sh, > runMultiLogReg_10M_1k_sparse_k150_v11.out, > runMultiLogReg_10M_1k_sparse_k150_v12.out, times_All_8g_rc2.txt, > times_Binomial_80g_rc1.txt, times_Binomial_80g_rc2.txt, > times_Binomial_80g_v11.txt, times_Binomial_8g_rc1.txt, > times_Binomial_8g_v11.txt, times_Clustering_80g_rc1.txt, > times_Clustering_80g_v11.txt, times_Clustering_8g_rc1.txt, > times_Clustering_8g_v11.txt, times_Multinomial_80g_bayes_sparse_rc1.txt, > times_Multinomial_80g_mlreg_dense_rc1.txt, > times_Multinomial_80g_mlreg_sparse_rc1.txt, > times_Multinomial_80g_rc1_subset.txt, > times_Multinomial_80g_v11_mlreg_dense.txt, > times_Multinomial_80g_v11_mlreg_dense_run2.txt, > times_Multinomial_80g_v11_mlreg_dense_run3.txt, > times_Multinomial_80g_v11_mlreg_sparse.txt, > times_Multinomial_8g_bayes_predict.txt, > times_Multinomial_8g_bayes_predict_v11.txt, times_Multinomial_8g_rc1.txt, > times_Multinomial_8g_v11.txt, times_Regression_80g_rc1.txt, > times_Regression_80g_rc2.txt, times_Regression_80g_v11.txt, > times_Regression_8g_rc1.txt, times_Regression_8g_v11.txt, > times_Stats_80g_rc1.txt, times_Stats_80g_v11.txt > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1219) Improve instructions and tips for running performance tests
[ https://issues.apache.org/jira/browse/SYSTEMML-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1219. --- > Improve instructions and tips for running performance tests > --- > > Key: SYSTEMML-1219 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1219 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner >Priority: Minor > Fix For: Not Applicable > > > There is a readme file at > https://github.com/apache/incubator-systemml/blob/master/scripts/perftest/README.TXT > which provides good description of scripts used for running performance > tests. Add supplemental information for additional configuration tips as > part of release process document at > http://apache.github.io/incubator-systemml/release-process.html#performance-suite > or separate best practices. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-286) Create design document for changes required to reduce time of unit test harness
[ https://issues.apache.org/jira/browse/SYSTEMML-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-286. -- > Create design document for changes required to reduce time of unit test > harness > --- > > Key: SYSTEMML-286 > URL: https://issues.apache.org/jira/browse/SYSTEMML-286 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 0.9 > > Attachments: UnitTestHarnessUpdates.doc > > > The unit tests need to be refactored to reduce the time it takes to run the > tests. This involves making the tests more granular, avoiding redundant R > script execution, and enabling separate folders for storing test data and > artifacts to increase utilization/parallelism. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-286) Create design document for changes required to reduce time of unit test harness
[ https://issues.apache.org/jira/browse/SYSTEMML-286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-286. Resolution: Fixed Fix Version/s: SystemML 0.9 > Create design document for changes required to reduce time of unit test > harness > --- > > Key: SYSTEMML-286 > URL: https://issues.apache.org/jira/browse/SYSTEMML-286 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 0.9 > > Attachments: UnitTestHarnessUpdates.doc > > > The unit tests need to be refactored to reduce the time it takes to run the > tests. This involves making the tests more granular, avoiding redundant R > script execution, and enabling separate folders for storing test data and > artifacts to increase utilization/parallelism. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1710) Document additional usage scenarios of SystemML
[ https://issues.apache.org/jira/browse/SYSTEMML-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1710. --- > Document additional usage scenarios of SystemML > --- > > Key: SYSTEMML-1710 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1710 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1710) Document additional usage scenarios of SystemML
[ https://issues.apache.org/jira/browse/SYSTEMML-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1710. - Resolution: Fixed Fix Version/s: Not Applicable > Document additional usage scenarios of SystemML > --- > > Key: SYSTEMML-1710 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1710 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1712) Reformat specific notebooks as tutorial or reference
[ https://issues.apache.org/jira/browse/SYSTEMML-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1712. --- > Reformat specific notebooks as tutorial or reference > > > Key: SYSTEMML-1712 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1712 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1712) Reformat specific notebooks as tutorial or reference
[ https://issues.apache.org/jira/browse/SYSTEMML-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1712: Fix Version/s: Not Applicable > Reformat specific notebooks as tutorial or reference > > > Key: SYSTEMML-1712 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1712 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1929) Update deploy-mode in sparkDML.sh and docs
[ https://issues.apache.org/jira/browse/SYSTEMML-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1929. - Resolution: Fixed Fix Version/s: SystemML 1.0 Resolved with commit [0505fd3|https://github.com/apache/systemml/commit/0505fd38c3191551a14c9b21314b0c3432b47e2f] and website commit [6fc5bad|https://github.com/gweidner/systemml-website/commit/6fc5bada5c31aa7f602fb9266713ddbdd38d840e]. > Update deploy-mode in sparkDML.sh and docs > -- > > Key: SYSTEMML-1929 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1929 > Project: SystemML > Issue Type: Improvement >Reporter: Glenn Weidner >Assignee: Glenn Weidner >Priority: Minor > Fix For: SystemML 1.0 > > > Update sparkDML.sh to use --deploy-mode instead of deprecated parameters. > Also update references in documentation (e.g., spark-batch-mode). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-312) Umbrella: Refactor unit tests
[ https://issues.apache.org/jira/browse/SYSTEMML-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-312. Resolution: Fixed Fix Version/s: SystemML 0.9 > Umbrella: Refactor unit tests > - > > Key: SYSTEMML-312 > URL: https://issues.apache.org/jira/browse/SYSTEMML-312 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 0.9 > > > This JIRA is an umbrella for subtasks involving work to refactor unit tests > to increase parallelism and reduce the time to run the tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-312) Umbrella: Refactor unit tests
[ https://issues.apache.org/jira/browse/SYSTEMML-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-312. -- > Umbrella: Refactor unit tests > - > > Key: SYSTEMML-312 > URL: https://issues.apache.org/jira/browse/SYSTEMML-312 > Project: SystemML > Issue Type: Task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 0.9 > > > This JIRA is an umbrella for subtasks involving work to refactor unit tests > to increase parallelism and reduce the time to run the tests. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-369) Combine tests under unary.matrix
[ https://issues.apache.org/jira/browse/SYSTEMML-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-369. -- > Combine tests under unary.matrix > > > Key: SYSTEMML-369 > URL: https://issues.apache.org/jira/browse/SYSTEMML-369 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > > Incorporate code review from https://github.com/SparkTC/systemml/pull/64: > Combine individual tests (such as ACosTest, ASinTest, ATanTest, etc.) under > functions.unary.matrix package into a single file -- this should include all > the following builtins: abs, sin, cos, tan, asin, acos, atan, sqrt, log, exp, > round, ceil, floor, inverse, cast_as_scalar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-369) Combine tests under unary.matrix
[ https://issues.apache.org/jira/browse/SYSTEMML-369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-369. Resolution: Won't Fix Fix Version/s: Not Applicable Multiple tests in unary.matrix pacakge were updated for enabling additional Spark tests in [SYSTEMML-1942]. > Combine tests under unary.matrix > > > Key: SYSTEMML-369 > URL: https://issues.apache.org/jira/browse/SYSTEMML-369 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: Not Applicable > > > Incorporate code review from https://github.com/SparkTC/systemml/pull/64: > Combine individual tests (such as ACosTest, ASinTest, ATanTest, etc.) under > functions.unary.matrix package into a single file -- this should include all > the following builtins: abs, sin, cos, tan, asin, acos, atan, sqrt, log, exp, > round, ceil, floor, inverse, cast_as_scalar. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1942) Enable conditional spark tests
[ https://issues.apache.org/jira/browse/SYSTEMML-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1942. - Resolution: Fixed Fix Version/s: SystemML 1.0 Fixed with [PR 677|https://github.com/apache/systemml/pull/677]. > Enable conditional spark tests > -- > > Key: SYSTEMML-1942 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1942 > Project: SystemML > Issue Type: Improvement > Components: Test >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 1.0 > > > Several tests suites in test.integration.functions.unary.matrix package have > conditional spark tests that can be enabled for improved test coverage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1945) Update python performance test scripts to use current Spark parameters
Glenn Weidner created SYSTEMML-1945: --- Summary: Update python performance test scripts to use current Spark parameters Key: SYSTEMML-1945 URL: https://issues.apache.org/jira/browse/SYSTEMML-1945 Project: SystemML Issue Type: Improvement Reporter: Glenn Weidner Priority: Minor The python performance test scripts use deprecated Spark parameters. Ideally, these should be updated to prevent Spark warning when launched. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1942) Enable conditional spark tests
Glenn Weidner created SYSTEMML-1942: --- Summary: Enable conditional spark tests Key: SYSTEMML-1942 URL: https://issues.apache.org/jira/browse/SYSTEMML-1942 Project: SystemML Issue Type: Improvement Components: Test Reporter: Glenn Weidner Assignee: Glenn Weidner Several tests suites in test.integration.functions.unary.matrix package have conditional spark tests that can be enabled for improved test coverage. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1941) Array index out of range exception observed for QuantilePickSPInstruction
[
https://issues.apache.org/jira/browse/SYSTEMML-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187222#comment-16187222
]
Glenn Weidner commented on SYSTEMML-1941:
-
Additional console output for the individual test shown below.
{code:java}
17/09/30 14:54:25 INFO scheduler.DAGScheduler: Job 5 finished: lookup at
QuantilePickSPInstruction.java:170, took 0.025935 s
SystemML Statistics:
Total execution time: 1.600 sec.
Number of executed Spark inst: 3.
17/09/30 14:54:25 INFO spark.MapOutputTrackerMasterEndpoint:
MapOutputTrackerMasterEndpoint stopped!
17/09/30 14:54:25 INFO memory.MemoryStore: MemoryStore cleared
17/09/30 14:54:25 INFO storage.BlockManager: BlockManager stopped
17/09/30 14:54:25 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
17/09/30 14:54:25 INFO
scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
17/09/30 14:54:25 INFO spark.SparkContext: Successfully stopped SparkContext
17/09/30 14:54:25 INFO api.DMLScript: END DML run 09/30/2017 14:54:25
17/09/30 14:54:25 ERROR api.DMLScript: Failed to execute DML script.
org.apache.sysml.runtime.DMLRuntimeException:
org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program
block generated from statement block between lines 23 and 0 -- Error evaluating
instruction: SPARK°qpick°_mVar9·MATRIX·DOUBLE°_Var11·SCALAR·DOUBLE°IQM°true
at
org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
at
org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:99)
at org.apache.sysml.api.DMLScript.execute(DMLScript.java:746)
at org.apache.sysml.api.DMLScript.executeScript(DMLScript.java:510)
at org.apache.sysml.api.DMLScript.main(DMLScript.java:237)
at
org.apache.sysml.test.integration.AutomatedTestBase.runTest(AutomatedTestBase.java:1222)
at
org.apache.sysml.test.integration.functions.unary.matrix.IQMTest.runTest(IQMTest.java:273)
at
org.apache.sysml.test.integration.functions.unary.matrix.IQMTest.testIQM1wt_SP(IQMTest.java:206)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:507)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error
in program block generated from statement block between lines 23 and 0 -- Error
evaluating instruction:
SPARK°qpick°_mVar9·MATRIX·DOUBLE°_Var11·SCALAR·DOUBLE°IQM°true
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:296)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:220)
at
org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:165)
at
[jira] [Created] (SYSTEMML-1941) Array index out of range exception observed for QuantilePickSPInstruction
Glenn Weidner created SYSTEMML-1941: --- Summary: Array index out of range exception observed for QuantilePickSPInstruction Key: SYSTEMML-1941 URL: https://issues.apache.org/jira/browse/SYSTEMML-1941 Project: SystemML Issue Type: Bug Reporter: Glenn Weidner Attachments: IQMTest.java After adding spark setup (i.e., set DMLScript.USE_LOCAL_SPARK_CONFIG = true) and enabling conditional spark tests in functions.unary.matrix.IQMTest (e.g., testIQM1wt_SP) as shown in attachment, observed the following error in local test development environment: Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 23 and 0 -- Error evaluating instruction: SPARK°qpick°_mVar9·MATRIX·DOUBLE°_Var11·SCALAR·DOUBLE°IQM°true at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:296) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:220) at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:165) at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123) ... 32 more Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 28 at org.apache.sysml.runtime.matrix.data.MatrixBlock.quickGetValue(MatrixBlock.java:583) at org.apache.sysml.runtime.instructions.spark.QuantilePickSPInstruction.lookupKey(QuantilePickSPInstruction.java:171) at org.apache.sysml.runtime.instructions.spark.QuantilePickSPInstruction.processInstruction(QuantilePickSPInstruction.java:146) at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:266) ... 35 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1426) Support both ceil and ceiling built-in functions
[ https://issues.apache.org/jira/browse/SYSTEMML-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1426. - Resolution: Fixed Fixed with [PR 674|https://github.com/apache/systemml/pull/674]. > Support both ceil and ceiling built-in functions > > > Key: SYSTEMML-1426 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1426 > Project: SystemML > Issue Type: Task > Components: APIs, Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Glenn Weidner > Labels: beginner > Fix For: SystemML 1.0 > > > The builtin function ceil unnecessarily differs from R's ceiling, which might > cause confusion. Hence, this task aims to rename ceil to ceiling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1426) Support both ceil and ceiling built-in functions
[ https://issues.apache.org/jira/browse/SYSTEMML-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1426. --- Closed with commit [id 6a75104|https://github.com/apache/systemml/commit/6a75104e5e870b3dd69bcad493bd4fe9fadd7c98]. > Support both ceil and ceiling built-in functions > > > Key: SYSTEMML-1426 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1426 > Project: SystemML > Issue Type: Task > Components: APIs, Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Glenn Weidner > Labels: beginner > Fix For: SystemML 1.0 > > > The builtin function ceil unnecessarily differs from R's ceiling, which might > cause confusion. Hence, this task aims to rename ceil to ceiling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1426) Support both ceil and ceiling built-in functions
[ https://issues.apache.org/jira/browse/SYSTEMML-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1426: Summary: Support both ceil and ceiling built-in functions (was: Rename builtin function ceil to ceiling) > Support both ceil and ceiling built-in functions > > > Key: SYSTEMML-1426 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1426 > Project: SystemML > Issue Type: Task > Components: APIs, Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Glenn Weidner > Labels: beginner > Fix For: SystemML 1.0 > > > The builtin function ceil unnecessarily differs from R's ceiling, which might > cause confusion. Hence, this task aims to rename ceil to ceiling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1426) Rename builtin function ceil to ceiling
[ https://issues.apache.org/jira/browse/SYSTEMML-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1426: Sprint: Sprint 7 > Rename builtin function ceil to ceiling > --- > > Key: SYSTEMML-1426 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1426 > Project: SystemML > Issue Type: Task > Components: APIs, Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Glenn Weidner > Labels: beginner > Fix For: SystemML 1.0 > > > The builtin function ceil unnecessarily differs from R's ceiling, which might > cause confusion. Hence, this task aims to rename ceil to ceiling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1426) Rename builtin function ceil to ceiling
[ https://issues.apache.org/jira/browse/SYSTEMML-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1426: Issue Type: Task (was: Sub-task) Parent: (was: SYSTEMML-1299) > Rename builtin function ceil to ceiling > --- > > Key: SYSTEMML-1426 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1426 > Project: SystemML > Issue Type: Task > Components: APIs, Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Glenn Weidner > Labels: beginner > Fix For: SystemML 1.0 > > > The builtin function ceil unnecessarily differs from R's ceiling, which might > cause confusion. Hence, this task aims to rename ceil to ceiling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1929) Update deploy-mode in sparkDML.sh and docs
[ https://issues.apache.org/jira/browse/SYSTEMML-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181492#comment-16181492 ] Glenn Weidner commented on SYSTEMML-1929: - Submitted [PR 670|https://github.com/apache/systemml/pull/670]. > Update deploy-mode in sparkDML.sh and docs > -- > > Key: SYSTEMML-1929 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1929 > Project: SystemML > Issue Type: Improvement >Reporter: Glenn Weidner >Assignee: Glenn Weidner >Priority: Minor > > Update sparkDML.sh to use --deploy-mode instead of deprecated parameters. > Also update references in documentation (e.g., spark-batch-mode). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (SYSTEMML-1929) Update deploy-mode in sparkDML.sh and docs
Glenn Weidner created SYSTEMML-1929: --- Summary: Update deploy-mode in sparkDML.sh and docs Key: SYSTEMML-1929 URL: https://issues.apache.org/jira/browse/SYSTEMML-1929 Project: SystemML Issue Type: Improvement Reporter: Glenn Weidner Assignee: Glenn Weidner Priority: Minor Update sparkDML.sh to use --deploy-mode instead of deprecated parameters. Also update references in documentation (e.g., spark-batch-mode). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1189) Investigate intermittent test failure in indexing.RightIndexingMatrixTest
[ https://issues.apache.org/jira/browse/SYSTEMML-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1189. - Resolution: Duplicate Fix Version/s: SystemML 1.0 Resolving this as duplicate with fix addressed in [SYSTEMML-1924]. > Investigate intermittent test failure in indexing.RightIndexingMatrixTest > - > > Key: SYSTEMML-1189 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1189 > Project: SystemML > Issue Type: Sub-task >Reporter: Glenn Weidner > Fix For: SystemML 1.0 > > Attachments: failsafe-reports-build-760.zip, > failsafe-reports-build-761.zip, > org.apache.sysml.test.integration.functions.indexing.RightIndexingMatrixTest-build-760.txt, > > org.apache.sysml.test.integration.functions.indexing.RightIndexingMatrixTest-build-761.txt, > > org.apache.sysml.test.integration.functions.indexing.RightIndexingMatrixTest-output-761.txt, > > org.apache.sysml.test.integration.functions.indexing.RightIndexingMatrixTest-output-build-760.txt > > > Test failure reported in daily test builds 760 and 761. Attached > failsafe-reports. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner closed SYSTEMML-1907. --- > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > Fix For: SystemML 1.0 > > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169343#comment-16169343 ] Glenn Weidner edited comment on SYSTEMML-1907 at 9/18/17 6:41 PM: -- Fixed with [PR 666|https://github.com/apache/systemml/pull/666] and website commit [ada6746|https://github.com/apache/systemml-website/commit/ada674608be074f4c5f64a7fdd8ac39731544a95]. Keeping JIRA open since renamed python artifact not available at [sparktc latest|https://sparktc.ibmcloud.com/repo/latest/]. . was (Author: gweidner): Fixed with [PR 666|https://github.com/apache/systemml/pull/666] and website commit [ada6746|https://github.com/apache/systemml-website/commit/ada674608be074f4c5f64a7fdd8ac39731544a95]. Keeping JIRA open since python artifact not available at [sparktc latest|https://sparktc.ibmcloud.com/repo/latest/]. . > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169343#comment-16169343 ] Glenn Weidner commented on SYSTEMML-1907: - Fixed with [PR 666|https://github.com/apache/systemml/pull/666] and website commit [ada6746|https://github.com/apache/systemml-website/commit/ada674608be074f4c5f64a7fdd8ac39731544a95]. Keeping JIRA open since python artifact not available at [sparktc latest|https://sparktc.ibmcloud.com/repo/latest/]. . > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (SYSTEMML-1896) Update python tests to use getOrCreate
[ https://issues.apache.org/jira/browse/SYSTEMML-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner resolved SYSTEMML-1896. - Resolution: Fixed Fix Version/s: SystemML 1.0 Resolved with [PR 657|https://github.com/apache/systemml/pull/657]. > Update python tests to use getOrCreate > -- > > Key: SYSTEMML-1896 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1896 > Project: SystemML > Issue Type: Bug > Components: Test > Environment: RedHat Linux 7.3 with Spark 2.1.0 and Hadoop 2.7.3. >Reporter: Glenn Weidner >Assignee: Glenn Weidner > Fix For: SystemML 1.0 > > > Use SparkContext.getOrCreate or SparkSession.builder.getOrCreate in python > tests to avoid "Cannot run multiple SparkContexts at once" error observed in > some environments running scripts under src/main/python/tests with > spark-submit. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1907: Sprint: Sprint 6 > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (SYSTEMML-1907) Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz
[ https://issues.apache.org/jira/browse/SYSTEMML-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1612#comment-1612 ] Glenn Weidner commented on SYSTEMML-1907: - Thanks for catching this - I'll take this one. > Rename python package from systemml-*-python.tgz to systemml-*-python.tar.gz > > > Key: SYSTEMML-1907 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1907 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Niketan Pansare >Assignee: Glenn Weidner > > I encountered this issue because pypi has migrated to a new process: > https://packaging.python.org/guides/migrating-to-pypi-org/#uploading > As noted in the above document, the recommended way to upload python packages > to pypi is now via `twine`. However, if we use `twine` with our current > package naming scheme (i.e. tgz), then it complains `ValueError: Unknown > distribution format: 'systemml-0.15.0-python.tgz'`. Hence, I would recommend > to use suffix `tar.gz` in our subsequent releases. This way we also are > compatible with default package naming convention of pypi: `tar.gz`. > [~acs_s] [~gweidner] [~deron] [~dusenberrymw] [~reinwald] Suggestions ? Any > takers ? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1724) Remove Guava from compile-time dependencies
[ https://issues.apache.org/jira/browse/SYSTEMML-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1724: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Remove Guava from compile-time dependencies > --- > > Key: SYSTEMML-1724 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1724 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Dylan Hutchison >Assignee: Dylan Hutchison >Priority: Minor > Fix For: SystemML 0.15 > > > SYSTEMML-1663 reintroduced Guava as a compile-time dependency into SystemML > during [PR 540|https://github.com/apache/systemml/pull/540]. Let's remove it > to reduce the compile-time memory footprint, as per SYSTEMML-698. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1083) Fix spelling of "Demonstration" in VLDB award
[ https://issues.apache.org/jira/browse/SYSTEMML-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1083: Fix Version/s: (was: SystemML 1.0) SystemML 0.11 > Fix spelling of "Demonstration" in VLDB award > - > > Key: SYSTEMML-1083 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1083 > Project: SystemML > Issue Type: Sub-task > Components: Website >Reporter: Jeremy Anderson >Assignee: Jason Azares >Priority: Minor > Fix For: SystemML 0.11 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1549) Cox.dml - return S & T in usable format
[ https://issues.apache.org/jira/browse/SYSTEMML-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1549: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Cox.dml - return S & T in usable format > --- > > Key: SYSTEMML-1549 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1549 > Project: SystemML > Issue Type: Improvement > Components: Algorithms >Reporter: Brendan Dwyer >Assignee: Brendan Dwyer >Priority: Minor > Fix For: SystemML 0.15 > > > Variables S & T are returned as strings. They should also be returned as a > matrix like R4ML > [does|https://github.com/SparkTC/r4ml/blob/master/R4ML/inst/sysml/scripts/algorithms/Cox.dml]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1600) Display version in MLContext welcome message
[ https://issues.apache.org/jira/browse/SYSTEMML-1600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1600: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Display version in MLContext welcome message > > > Key: SYSTEMML-1600 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1600 > Project: SystemML > Issue Type: Improvement > Components: APIs >Reporter: Deron Eriksson >Assignee: Krishna Kalyan >Priority: Minor > Fix For: SystemML 0.15 > > > Append SystemML version number to MLContext welcome message. It is available > via the MLContext version() method. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1668) Wrong worst-case estimates for rbind in BinaryOp
[
https://issues.apache.org/jira/browse/SYSTEMML-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1668:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Wrong worst-case estimates for rbind in BinaryOp
>
>
> Key: SYSTEMML-1668
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1668
> Project: SystemML
> Issue Type: Bug
> Components: Compiler
>Affects Versions: SystemML 0.14
>Reporter: Dylan Hutchison
>Assignee: Dylan Hutchison
>Priority: Minor
> Fix For: SystemML 0.15
>
>
> In {{BinaryOp.inferOutputCharacteristics}}, RBIND is not checked and CBIND is
> checked twice. The second case should refer to RBIND.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1737) BufferedReader should be closed in ParameterizedBuiltinCPFileInstruction#createCellResultFile()
[
https://issues.apache.org/jira/browse/SYSTEMML-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1737:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> BufferedReader should be closed in
> ParameterizedBuiltinCPFileInstruction#createCellResultFile()
> ---
>
> Key: SYSTEMML-1737
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1737
> Project: SystemML
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: SystemML 0.15
>
>
> {code}
> BufferedReader fkeyMap = StagingFileUtils.openKeyMap(metaOut);
> {code}
> BufferedReader should be closed upon exit from method.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1455) Change the term PLAIN_R2 to R2 in all algorithms
[ https://issues.apache.org/jira/browse/SYSTEMML-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1455: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Change the term PLAIN_R2 to R2 in all algorithms > > > Key: SYSTEMML-1455 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1455 > Project: SystemML > Issue Type: Improvement > Components: Algorithms >Reporter: Imran Younus >Assignee: Krishna Kalyan >Priority: Minor > Fix For: SystemML 0.15 > > > In some of the regression algorithms, we return several metrics. One of these > is R2. But we call if PLAIN_R2. This is unconventional. We should just call > it R2. I've never see the term PLAIN_R2 in any book or paper or software etc. > There is R2 and Adjusted R2. I think it would be better to use the > conventional terminology as mush as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1380) Kmeans isY and verb parameters can be boolean
[ https://issues.apache.org/jira/browse/SYSTEMML-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1380: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Kmeans isY and verb parameters can be boolean > - > > Key: SYSTEMML-1380 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1380 > Project: SystemML > Issue Type: Improvement > Components: Algorithms >Reporter: Deron Eriksson >Assignee: Krishna Kalyan >Priority: Minor > Fix For: SystemML 0.15 > > > In the Kmeans.dml script, the 'isY' and 'verb' input parameters are integers. > However, they are basically on/off switches, so replacing the integer types > with boolean types could potentially be a little clearer to users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1774) Improve Parfor parallelism for deep learning
[
https://issues.apache.org/jira/browse/SYSTEMML-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1774:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Improve Parfor parallelism for deep learning
>
>
> Key: SYSTEMML-1774
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1774
> Project: SystemML
> Issue Type: Improvement
> Components: Algorithms, Compiler, ParFor
>Affects Versions: SystemML 1.0
>Reporter: Fei Hu
>Assignee: Fei Hu
> Labels: deeplearning
> Fix For: SystemML 0.15
>
> Attachments: Explain_For_HYBRID_SPARK_Mode_With_ErrorInfo.txt,
> Explain_For_Spark_Mode.txt, MNIST_Distrib_Sgd.scala,
> mnist_lenet_distrib_sgd.dml
>
>
> When running the [distributed MNIST LeNet example |
> https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml],
> each mini-batch could ideally run in parallel without interaction. We try to
> force {{parfor (j in 1:parallel_batches)}} at line 137 of
> {{nn/examples/mnist_lenet_distrib_sgd.dml}} to be {{parfor (j in
> 1:parallel_batches, mode=REMOTE_SPARK, opt=CONSTRAINED)}} use
> {{REMOTE_SPARK}} mode, but got some errors about
> {{org.apache.sysml.runtime.DMLRuntimeException: Not supported: Instructions
> of type other than CP instructions}} using the mode {{SPARK}}, and the error
> {{java.lang.NullPointerException}} using the mode {{HYBRID_SPARK}}. More log
> information can be found at the following comments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1736) Add new 2D top_k utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1736:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Add new 2D top_k utility function
> -
>
> Key: SYSTEMML-1736
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1736
> Project: SystemML
> Issue Type: New Feature
>Reporter: Mike Dusenberry
>Assignee: Fei Hu
> Fix For: SystemML 0.15
>
>
> We should add a new {{top_k2d}} utility function (in {{nn/util.dml}}) that
> accepts a matrix {{X}} and return matrices {{values}} and {{indices}} with
> the top {{k}} values (i.e. probabilities) and associated indices (i.e.
> classes) along a certain dimension. This will be modeled after the
> [{{top_k}} function in TensorFlow |
> https://www.tensorflow.org/api_docs/python/tf/nn/top_k]. For the 2D case,
> {{top_k}} will operate on the channels dimension. A typical use case here is
> that in which {{X}} is the output of a {{softmax2d}} layer (so each channel
> contains a set of normalized class probabilities), and {{values}} and
> {{indices}} will contain the top {{k}} probabilities and indices along the
> channel axis. This scenario would be common in an image segmentation
> problem, in which every pixel of the output image will have a set of class
> probabilities along the channel axis.
> Having these {{top-k}} functions will allow us to extract either predict a
> single class for each item, or the top {{k}} classes, and therefore may be
> more useful that a {{predict_class}} function.
> Although we will use {{values}} and {{indices}} as the names of the returned
> matrices within the functions, in practice, one is likely to name the results
> {{probs}} and {{classes}} in the calling environment.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1677) Add a new 2D cross-entropy layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1677: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add a new 2D cross-entropy layer > > > Key: SYSTEMML-1677 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1677 > Project: SystemML > Issue Type: New Feature > Components: Algorithms >Reporter: Mike Dusenberry >Assignee: Fei Hu > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1676) Add a new 2D softmax layer
[
https://issues.apache.org/jira/browse/SYSTEMML-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1676:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Add a new 2D softmax layer
> --
>
> Key: SYSTEMML-1676
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1676
> Project: SystemML
> Issue Type: New Feature
> Components: Algorithms
>Reporter: Mike Dusenberry
>Assignee: Fei Hu
> Fix For: SystemML 0.15
>
>
> A 2D softmax layer would accept a tensor of shape {{(N,C,H,W)}}, where the
> {{C}} axis contains scores for {{D}} classes, and output a tensor of the same
> shape, with the scores transformed to normalized probabilities. The typical
> use case would be a segmentation problem, in which every pixel has a
> multiclass prediction.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1694) Add snapshot version number to docs header
[ https://issues.apache.org/jira/browse/SYSTEMML-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1694: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add snapshot version number to docs header > -- > > Key: SYSTEMML-1694 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1694 > Project: SystemML > Issue Type: Improvement > Components: Documentation >Reporter: Deron Eriksson >Assignee: Gus Jenkins > Fix For: SystemML 0.15 > > > Currently the latest snapshot documentation > (http://apache.github.io/systemml/) has "Latest" in the header. "Latest" can > be a little confusing since it's hard to tell whether this is the latest > release version (currently 0.14.0-incubating) or the latest snapshot version > (1.0.0-SNAPSHOT). > We should probably change "Latest" to either something like: > Latest (1.0.0-SNAPSHOT) > or > 1.0.0-SNAPSHOT -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1647) Verify whether StepLinearReg script works with MLContext
[ https://issues.apache.org/jira/browse/SYSTEMML-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1647: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Verify whether StepLinearReg script works with MLContext > > > Key: SYSTEMML-1647 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1647 > Project: SystemML > Issue Type: Improvement > Components: Algorithms >Reporter: Imran Younus >Assignee: Imran Younus > Fix For: SystemML 0.15 > > > This jira plans to fix StepLinearReg script in order to make it work with new > MLContext. Currently its not working with new MLContext. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1679) Add a new threshold utility function
[
https://issues.apache.org/jira/browse/SYSTEMML-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1679:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Add a new threshold utility function
>
>
> Key: SYSTEMML-1679
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1679
> Project: SystemML
> Issue Type: New Feature
>Reporter: Mike Dusenberry
>Assignee: Fei Hu
> Fix For: SystemML 0.15
>
>
> We should add a new {{threshold}} utility function (in {{nn/util.dml}}) that
> accepts a matrix {{X}} and a threshold parameter {{thresh}} and returns an
> indicator matrix {{out}} with values in \{0, 1\} depending on whether or not
> the values in {{X}} are above {{thresh}}. We could use this, for example,
> for determining the predicted class in a binary classification problem given
> the output of a sigmoid layer.
> We should also add a test case in {{nn/test/test.dml}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1608) Add ALS notebook example
[ https://issues.apache.org/jira/browse/SYSTEMML-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1608: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add ALS notebook example > > > Key: SYSTEMML-1608 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1608 > Project: SystemML > Issue Type: Task >Reporter: Imran Younus >Assignee: Imran Younus > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1567) Remove conditionals from nn layers
[
https://issues.apache.org/jira/browse/SYSTEMML-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1567:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Remove conditionals from nn layers
> --
>
> Key: SYSTEMML-1567
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1567
> Project: SystemML
> Issue Type: Improvement
> Components: APIs
>Affects Versions: SystemML 1.0
>Reporter: Niketan Pansare
> Fix For: SystemML 0.15
>
>
> Conditionals in nn layers introduce transient read/write variables that
> disables fused operators such as CP relu_maxpooling_backward and hence
> redundant execute sparsity-introducing sel+ operator. This operator causes
> unnecessary dense-to-sparse-to-dense conversion and becomes the heavy hitter
> after native BLAS change. Note: some fused operators such as CP
> relu_maxpooling are still applied because there is no conditional in between
> those layers.
> Without conditionals in dropout layer:
> https://github.com/apache/incubator-systemml/blob/master/scripts/nn/layers/dropout.dml#L49-L53
>
> {code}
> Iter:2000.0, training loss:0.003149394810197065, training accuracy:100.0
> Iter:2000.0, validation loss:191.9888157354513, validation accuracy:96.875
> SystemML Statistics:
> Total elapsed time: 416.609 sec.
> Total compilation time: 0.000 sec.
> Total execution time: 416.609 sec.
> Number of compiled Spark inst: 69.
> Number of executed Spark inst: 2.
> Native mkl calls (LibMatrixMult/LibMatrixDNN): 4270/10553.
> Cache hits (Mem, WB, FS, HDFS): 277973/0/0/0.
> Cache writes (WB, FS, HDFS):143616/0/0.
> Cache times (ACQr/m, RLS, EXP): 0.101/0.080/1.988/0.000 sec.
> HOP DAGs recompiled (PRED, SB): 0/2277.
> HOP DAGs recompile time:6.146 sec.
> Spark ctx create time (lazy): 0.027 sec.
> Spark trans counts (par,bc,col):0/0/0.
> Spark trans times (par,bc,col): 0.000/0.000/0.000 secs.
> Total JIT compile time: 37.746 sec.
> Total JVM GC count: 3949.
> Total JVM GC time: 56.609 sec.
> Heavy hitter instructions (name, time, count):
> -- 1) conv2d_bias_add 48.984 sec 4514
> -- 2) conv2d_backward_filter 47.780 sec 4026
> -- 3) -* 38.246 sec 16104
> -- 4) +* 35.902 sec 8052
> -- 5) + 34.227 sec 30566
> -- 6) ba+*30.643 sec 12566
> -- 7) relu_maxpooling_backward29.678 sec 4026
> -- 8) conv2d_backward_data28.520 sec 2013
> -- 9) * 26.825 sec 35275
> -- 10) relu_backward 24.842 sec 6039
> {code}
> With conditional, we add sel+ to the heavy hitter:
> {code}
> -- 1) sel+55.054 sec 6283
> {code}
> [~mwdus...@us.ibm.com] Since you created the layers, I think you should
> decide how best to restructure the DML. My recommendation would be to create
> two layers in case of conditionals.
> [~mboehm7] [~reinwald]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1585) Include JCuda jars into SystemML's extra.jar
[ https://issues.apache.org/jira/browse/SYSTEMML-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1585: Fix Version/s: (was: SystemML 1.0) Not Applicable > Include JCuda jars into SystemML's extra.jar > > > Key: SYSTEMML-1585 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1585 > Project: SystemML > Issue Type: Improvement >Reporter: Niketan Pansare > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1469) Add a new `conv2d_transpose` layer.
[ https://issues.apache.org/jira/browse/SYSTEMML-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1469: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add a new `conv2d_transpose` layer. > --- > > Key: SYSTEMML-1469 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1469 > Project: SystemML > Issue Type: New Feature >Reporter: Mike Dusenberry >Assignee: Prithviraj Sen > Fix For: SystemML 0.15 > > > A conv2d tranpose layer is the gradient of a conv2d layer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1068) Add Code Highlighting
[ https://issues.apache.org/jira/browse/SYSTEMML-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1068: Fix Version/s: (was: SystemML 1.0) SystemML 0.14 > Add Code Highlighting > - > > Key: SYSTEMML-1068 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1068 > Project: SystemML > Issue Type: Sub-task > Components: Website >Reporter: Mike Dusenberry >Assignee: Dexter Lesaca > Fix For: SystemML 0.14 > > > For our tutorials, it would be nice to have code syntax highlighting to make > it easier to understand the code snippets. Jekyll supports this feature > \[1], as do a number of other libraries. At a minimum, we should have R, > Python, and Scala syntax highlighting. > \[1]: https://jekyllrb.com/docs/posts/#highlighting-code-snippets -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-230) Blockwise data partitioning
[ https://issues.apache.org/jira/browse/SYSTEMML-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-230: --- Fix Version/s: (was: SystemML 1.0) Not Applicable > Blockwise data partitioning > --- > > Key: SYSTEMML-230 > URL: https://issues.apache.org/jira/browse/SYSTEMML-230 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Frederick Reiss > Fix For: Not Applicable > > Original Estimate: 80h > Remaining Estimate: 80h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-515) 'Sparsity' Parameter For `rand` Statement Should Allow An Expression
[
https://issues.apache.org/jira/browse/SYSTEMML-515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-515:
---
Fix Version/s: (was: SystemML 1.0)
SystemML 0.13
> 'Sparsity' Parameter For `rand` Statement Should Allow An Expression
>
>
> Key: SYSTEMML-515
> URL: https://issues.apache.org/jira/browse/SYSTEMML-515
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
> Fix For: SystemML 0.13
>
>
> The {{rand(...)}} function has a {{sparsity}} parameter that allows one to
> specify the desired sparsity of the generated matrix as in
> {code}
> X = rand(rows=10, cols=20, min=0, max=1, pdf=”uniform”, sparsity=0.2)
> {code}.
> Currently, the {{sparsity}} parameter only accepts {{Literal}} inputs, i.e.
> hard-coded double values, or variables that are themselves hard-coded double
> values. It would be better to allow an expression that evaluates to a double
> value, such as the following, simple, contrived example:
> {code}
> s = log(0.2)
> X = rand(rows=10, cols=20, min=0, max=1, pdf=”uniform”, sparsity=s)
> {code}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1415) Rename `nn/layers/max_pool.dml` to `nn/layers/max_pool2d.dml`
[
https://issues.apache.org/jira/browse/SYSTEMML-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1415:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> Rename `nn/layers/max_pool.dml` to `nn/layers/max_pool2d.dml`
> -
>
> Key: SYSTEMML-1415
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1415
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>Priority: Minor
> Fix For: SystemML 0.14
>
>
> Note that this breaks the current API. This is fine though since the {{nn}}
> library is currently in staging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1416) Rename `nn/layers/conv_builtin.dml` to `nn/layers/conv2d_builtin.dml`
[
https://issues.apache.org/jira/browse/SYSTEMML-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1416:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> Rename `nn/layers/conv_builtin.dml` to `nn/layers/conv2d_builtin.dml`
> -
>
> Key: SYSTEMML-1416
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1416
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>Priority: Minor
> Fix For: SystemML 0.14
>
>
> Note that this breaks the current API. This is fine though since the {{nn}}
> library is currently in staging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1417) Rename `nn/layers/max_pool_builtin.dml` to `nn/layers/max_pool2d_builtin.dml`
[
https://issues.apache.org/jira/browse/SYSTEMML-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1417:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> Rename `nn/layers/max_pool_builtin.dml` to `nn/layers/max_pool2d_builtin.dml`
> -
>
> Key: SYSTEMML-1417
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1417
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
>Priority: Minor
> Fix For: SystemML 0.14
>
>
> Note that this breaks the current API. This is fine though since the {{nn}}
> library is currently in staging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1766) Merge experimental breast cancer project code into main repo
[ https://issues.apache.org/jira/browse/SYSTEMML-1766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1766: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Merge experimental breast cancer project code into main repo > > > Key: SYSTEMML-1766 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1766 > Project: SystemML > Issue Type: New Feature > Components: Algorithms >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > > This aims to consolidate and cleanup experimental breast cancer project code, > and move it into the main repo. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1813) Preprocessing simplification and cleanup
[ https://issues.apache.org/jira/browse/SYSTEMML-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1813: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Preprocessing simplification and cleanup > > > Key: SYSTEMML-1813 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1813 > Project: SystemML > Issue Type: Improvement >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > > In anticipation of near-future algorithmic improvements to the preprocessing > to improve model training, this simplifies and cleans up the preprocessing > code as follows. > - Previously, we were processing all slides into one large saved > DataFrame, and then splitting that DataFrame into train and validation > DataFrames. We should simplify this by splitting the slide numbers > into train and validation sets, and then processing those slides > separately. This will effectively skip the creation of the large DataFrame, > and remove the need to split that large DataFrame into train/val ones, > which should provide a large performance benefit. The DataFrame `union` > method can be used to combine two DataFrames row-wise. > - Previously, we maintained a list of "broken" slides that were manually > removed. We should remove that manual list, and instead add a > try/except filtering step to automatically remove problematic slides. > - We should move ad-hoc sampling code into a new `sample` function. > - We should move code to add row indices to a DataFrame into a new > `add_row_indices` function. > The benefit is that near-future algorithmic improvements to the > preprocessing code will be much easier to incorporate. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1674) Add a new 2D depthwise convolution layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1674: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add a new 2D depthwise convolution layer > > > Key: SYSTEMML-1674 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1674 > Project: SystemML > Issue Type: New Feature >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > > A depthwise convolution (1) applies a different set of M filters to each > input channel separately, thus expanding each input channel to M output > channels, and (2) concatenates the results into a single volume with C*M > output channels. This is in contrast to a regular 2D convolution, in which > all of the filters would be applied to all of the input channels at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1575) DataType Change Test Failure
[
https://issues.apache.org/jira/browse/SYSTEMML-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1575:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> DataType Change Test Failure
>
>
> Key: SYSTEMML-1575
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1575
> Project: SystemML
> Issue Type: Bug
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.15
>
>
> While working on SYSTEMML-1554, an additional bug was uncovered.
> Specifically, with the IPA scalar replacement enhancement, the
> {{org.apache.sysml.test.integration.functions.misc.DataTypeChangeTest#testDataTypeChangeValidate4c}}
> test has started to fail. Looking into it, it fails due to trying to cast a
> Matrix to a Scalar object. At a deeper level, it looks like the propagated
> variable map is holding onto the "matrix" `X`, rather than dropping it as it
> should, since X is turned into a scalar by the call `X = foo(X)`.
> Interestingly, the FunctionOp for the `foo` function is marked as having an
> `Unknown` datatype and valuetype. Overall, this seems like a bug that was
> just hidden before, rather than being newly introduced.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1675) Add a new 2D depthwise transpose convolution layer
[ https://issues.apache.org/jira/browse/SYSTEMML-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1675: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add a new 2D depthwise transpose convolution layer > -- > > Key: SYSTEMML-1675 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1675 > Project: SystemML > Issue Type: New Feature >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > > A depthwise transpose convolution (1) applies a different filter to each > unique group of M input channels separately, thus condensing each group of M > input channels to 1 output channel, and (2) concatenates the results into a > single volume with C/M output channels. This is in contrast to a regular 2D > transpose convolution, in which all of the filters would be applied to all of > the input channels at once. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1561) Improve constant folding during compilation
[
https://issues.apache.org/jira/browse/SYSTEMML-1561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1561:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Improve constant folding during compilation
> ---
>
> Key: SYSTEMML-1561
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1561
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.15
>
> Attachments: scenario1_plan.txt, scenario1.py, scenario2_plan.txt,
> scenario2.py
>
>
> In our `nn` library, our convolution and pooling layers have to pass around
> the spatial dimensions (height and width) of the images that are stretched
> out into rows of the input/output matrices. These output dimensions are
> computed within the forward functions of the above layers as small scalar
> equations. From a mathematical standpoint, these sizes can be determined at
> compile time, and it is nice to have these size equations in DML (v.s. hiding
> them inside the engine within built-in functions). However, we do not
> currently evaluate these expressions during compilation, and thus we are left
> with unknown sizes even during recompilation. This naturally leads to max
> memory estimates and thus often leads to unnecessary distributed runtime ops
> rather than simple CP ones.
> I have two related scenarios for which this is a problem. They both involve
> the {{Houtc1}} & {{Woutc1}} values that are returned from a
> `conv2d::forward(...)` function. These represent the spatial dimensions of
> the volume with each of the rows of the output {{outc1}} of the function, and
> the third dimension is {{F1}}. Thus, {{outc1}} has a number of columns equal
> to {{F1*Houtc1*Wouc1}}.
> In the first scenario ({{scenario1.py}}), a random matrix {{doutc1}} is
> created that should have the same dimensions as {{outc1}}. For the columns,
> if I use {{cols=ncol(outc1)}} in this rand statement, the size will be
> propagated and CP ops will be compiled and run. I I instead use
> {{cols=F1*Houtc1*Woutc1}}, the size will forever be unknown, even during
> recompilation, and thus Spark ops will be compiled and run. I have included
> the recompile hops plan ({{scenario1_plan.txt}}).
> In the second scenario ({{scenario2.py}}), a {{max_pool2d::forward(...)}}
> function is inserted after the {{conv2d::forward(...)}} function that
> requires the {{Houtc1}} and {{Woutc1}} variables to be supplied as arguments.
> Since those latter variables are not executed during compilation time, the
> max pooling sizes remain unknown, even during recompilation, and thus Spark
> ops will be compiled and run. I have included the recompile hops plan
> ({{scenario2_plan.txt}}).
> We should either improve or fix our constant folding rewrites so that these
> scenarios are fixed, as they are necessary for performant deep learning
> applications. Note too that this issue will be present in other non-deep
> learning scenarios as well.
> Mailing list thread:
> https://www.mail-archive.com/dev@systemml.incubator.apache.org/msg01657.html
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1563) Add a distributed synchronous SGD MNIST LeNet example
[
https://issues.apache.org/jira/browse/SYSTEMML-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1563:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Add a distributed synchronous SGD MNIST LeNet example
> -
>
> Key: SYSTEMML-1563
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1563
> Project: SystemML
> Issue Type: New Feature
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.15
>
>
> This aims to add a *distributed synchronous SGD* MNIST LeNet example. In
> distributed synchronous SGD, multiple mini-batches are run forward & backward
> simultaneously, and the gradients are aggregated together by addition before
> the model parameters are updated. This is mathematically equivalent to
> simply using a large mini-batch size, i.e. {{new_mini_batch_size =
> mini_batch_size * number_of_parallel_mini_batches}}. The benefit is that
> distributed synchronous SGD can make use of multiple devices, i.e. multiple
> GPUs or multiple CPU machines, and thus can speed up training time. More
> specifically, using an effectively larger mini-batch size can yield a more
> stable gradient in expectation, and a larger number of epochs can be run in
> the same amount of time, both of which lead to faster convergence.
> Alternatives include various forms of distributed _asynchronous_ SGD, such as
> Downpour, Hogwild, etc. However, a recent paper \[1] from Google Brain /
> Open AI has found evidence supporting the claim that distributed synchronous
> SGD can lead to faster convergence, particularly if it is extending with the
> notion of "backup workers" as described in the paper.
> We will first aim for distributed synchronous SGD with no backup workers, and
> then extend this to include backup workers. The MNIST LeNet model will
> simply serve as an example, and this same approach can be extended to more
> recent models, such as ResNets.
> \[1]: https://arxiv.org/abs/1604.00981
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1564) Add a Java test suite wrapper around `nn` DML test suite
[
https://issues.apache.org/jira/browse/SYSTEMML-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1564:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Add a Java test suite wrapper around `nn` DML test suite
>
>
> Key: SYSTEMML-1564
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1564
> Project: SystemML
> Issue Type: New Feature
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.15
>
>
> The {{nn}} library contains it's own DML test suite for gradient checks and
> unit tests. The test suite produces "ERROR..." messages if any of the
> mathematical operations return incorrect results. Note that this has helped
> to find mathematical bugs in the library & engine that do not result in JVM
> exceptions, but are equally important.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1465) Add stain normalization to preprocessing.
[ https://issues.apache.org/jira/browse/SYSTEMML-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1465: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add stain normalization to preprocessing. > - > > Key: SYSTEMML-1465 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1465 > Project: SystemML > Issue Type: New Feature >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1554) IPA Scalar Transient Read Replacement
[ https://issues.apache.org/jira/browse/SYSTEMML-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1554: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > IPA Scalar Transient Read Replacement > - > > Key: SYSTEMML-1554 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1554 > Project: SystemML > Issue Type: Improvement >Reporter: Mike Dusenberry >Assignee: Mike Dusenberry > Fix For: SystemML 0.15 > > Attachments: convnet_distrib_sgd.dml, parfor_oom_convnet_plan.txt, > parfor_oom_convnet.py, parfor_oom_plan.txt, parfor_oom.py > > > Currently, during IPA we collect all variables (scalars & matrices) eligible > for propagation across blocks (i.e. not updated in block), and then propagate > the only the matrix sizes across the blocks. It seems plausible that we > could also replace all eligible scalar transient reads with literals based on > the variables that have already been collected. The benefit is that many ops > will be able to determine their respective output sizes during regular > compilation, instead of having to wait until dynamic recompilation, and thus > we can reduce the pressure on dynamic recompilation. > Are there drawbacks to this approach? The use case is that I was seeing a > large number of memory warnings while training a convolutional net due to the > sizes being unknown during regular compilation, yet the engine only having CP > versions of the ops. Additionally, I was running into actual heap space OOM > errors for situations that should not run out of memory, and thus I started > exploring. > I've attached an example script and the explain plan (hops & runtime) w/ and > w/o the IPA scalar replacement. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1414) Rename `nn/layers/conv.dml` to `nn/layers/conv2d.dml`
[
https://issues.apache.org/jira/browse/SYSTEMML-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1414:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> Rename `nn/layers/conv.dml` to `nn/layers/conv2d.dml`
> -
>
> Key: SYSTEMML-1414
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1414
> Project: SystemML
> Issue Type: Improvement
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.14
>
>
> Note that this breaks the current API. This is fine though since the {{nn}}
> library is currently in staging.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1524) Graduate `nn` library from `scripts/staging/SystemML-NN/nn` to `scripts/nn`
[
https://issues.apache.org/jira/browse/SYSTEMML-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1524:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Graduate `nn` library from `scripts/staging/SystemML-NN/nn` to `scripts/nn`
> ---
>
> Key: SYSTEMML-1524
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1524
> Project: SystemML
> Issue Type: New Feature
>Reporter: Mike Dusenberry
>Assignee: Mike Dusenberry
> Fix For: SystemML 0.15
>
>
> For our upcoming 1.0 release, we should release the {{nn}} deep learning
> library as an official top-level SystemML library. This would coincide with
> our Caffe integration via Caffe2DML that targets the {{nn}} library, as well
> as our native BLAS and GPU runtime targets, which our deep learning use cases
> will benefit from.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1034) Implement solve builtin function using cublas kernels
[ https://issues.apache.org/jira/browse/SYSTEMML-1034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1034: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Implement solve builtin function using cublas kernels > - > > Key: SYSTEMML-1034 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1034 > Project: SystemML > Issue Type: Sub-task > Components: Runtime >Reporter: Niketan Pansare >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > > 1. Extend BinaryOp to enable GPU for solve > 2. Add MatrixMatrixBuiltinGPUInstruction and use JCuBlas2's > cublasDtrsmBatched and cublasDgeqrfBatched (or cublasDgetrfBatched) methods. > For reference implementation, see > https://github.com/apache/incubator-systemml/blob/master/src/main/java/org/apache/sysml/runtime/matrix/data/LibCommonsMath.java#L97 > [~nakul02] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1816) toString can return -0
[
https://issues.apache.org/jira/browse/SYSTEMML-1816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1816:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> toString can return -0
> --
>
> Key: SYSTEMML-1816
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1816
> Project: SystemML
> Issue Type: Bug
> Components: Runtime
>Reporter: Deron Eriksson
>Assignee: Nakul Jindal
> Fix For: SystemML 0.15
>
>
> When display matrix values with toString, -0 can be displayed.
> Example:
> {code}
> m = matrix("50 99 100 200",rows=2,cols=2);
> x = 100;
> m = (m - x) * ((m-x) >= 0)
> print(toString(m))
> {code}
> gives:
> {code}
> -0.000 -0.000
> 0.000 100.000
> {code}
> Using as.scalar on the individual cells returns 0:
> {code}
> for (i in 1:nrow(m)) {
> for (j in 1:ncol(m)) {
> n = m[i,j]
> print('[' + i + ',' + j + ']:' + as.scalar(n))
> }
> }
> {code}
> gives:
> {code}
> [1,1]:0.0
> [1,2]:0.0
> [2,1]:0.0
> [2,2]:100.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-704) Host jcu*.jar libraries on mvn repo
[ https://issues.apache.org/jira/browse/SYSTEMML-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-704: --- Fix Version/s: (was: SystemML 1.0) SystemML 0.14 > Host jcu*.jar libraries on mvn repo > --- > > Key: SYSTEMML-704 > URL: https://issues.apache.org/jira/browse/SYSTEMML-704 > Project: SystemML > Issue Type: Task >Reporter: Niketan Pansare >Assignee: Nakul Jindal > Fix For: SystemML 0.14 > > > The PR https://github.com/apache/incubator-systemml/pull/165/ uses system > scope for jcu*.jar as they are not published on mvn central. Since we are > planning to include them into SystemML, it would be good to host them into a > repo we maintain and have provided scope instead. If for LICENSE or some > other reasons, we are not able to host them, I am fine with rejecting this > issue too. From jcuda's website "JCuda is published under the terms of the > MIT/X11 License". > The current version depends on jcu*-0.7.5b.jar (except jcudnn-0.7.5.jar). The > jars are available for download from > http://www.jcuda.org/downloads/downloads.html. The source is available at > https://github.com/jcuda > [~nakul02] [~deron] [~luciano resende] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1806) setTextValue in DMLConfig is not behaving correctly
[
https://issues.apache.org/jira/browse/SYSTEMML-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1806:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> setTextValue in DMLConfig is not behaving correctly
> ---
>
> Key: SYSTEMML-1806
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1806
> Project: SystemML
> Issue Type: Bug
> Components: Runtime
>Reporter: Nakul Jindal
>Assignee: Nakul Jindal
> Fix For: SystemML 0.15
>
>
> The problem was discovered when trying to set a configuration property from
> MLContext.
> Specifically, it was to try and restrict which GPU to use for the program.
> Currently this is done via a System property.
> This was the script:
> {code}
> unet = Caffe2DML(spark, solver='solver.prototxt', input_shape=img_shape)
> unet.setGPU(True)
> unet.setForceGPU(True)
> unet.setConfigProperty("systemml.stats.extraGPU", "true")
> unet.setConfigProperty("systemml.gpu.availableGPUs", "1")
> {code}
> Here is what I discovered:
> The first time
> [setText|https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysml/conf/DMLConfig.java#L266]
> value is called on an empty DMLConfig (by calling new DMLConfig()) as
> opposed to by parsing a file, a new _xmlRoot is initialized.
> Thereafter, since the _xmlRoot is not null, it tries to call
> getElementsByTagName, even when the tag is different. In the example above,
> the tag is {{systemml.stats.extraGPU"}} the first time around and
> {{systemml.gpu.availableGPUs}} the second time around.
> The bug is in
> [setText|https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysml/conf/DMLConfig.java#L253]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1744) Include jcuda jars in extra assembly jar for easy pip install
[ https://issues.apache.org/jira/browse/SYSTEMML-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1744: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Include jcuda jars in extra assembly jar for easy pip install > - > > Key: SYSTEMML-1744 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1744 > Project: SystemML > Issue Type: Improvement > Components: Build >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1735) Add relational operators for GPU
[ https://issues.apache.org/jira/browse/SYSTEMML-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1735: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add relational operators for GPU > > > Key: SYSTEMML-1735 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1735 > Project: SystemML > Issue Type: Sub-task > Components: Runtime >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1758) Add cbind (and rbind) GPU ops
[ https://issues.apache.org/jira/browse/SYSTEMML-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1758: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add cbind (and rbind) GPU ops > - > > Key: SYSTEMML-1758 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1758 > Project: SystemML > Issue Type: Sub-task > Components: Runtime >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > > Ping [~niketanpansare] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1654) GPU cannot handle nested local parfors
[ https://issues.apache.org/jira/browse/SYSTEMML-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1654: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > GPU cannot handle nested local parfors > -- > > Key: SYSTEMML-1654 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1654 > Project: SystemML > Issue Type: Bug > Components: Runtime >Affects Versions: SystemML 0.14 >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1713) Verify and correct memory estimates for various ops on the GPU
[ https://issues.apache.org/jira/browse/SYSTEMML-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1713: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Verify and correct memory estimates for various ops on the GPU > -- > > Key: SYSTEMML-1713 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1713 > Project: SystemML > Issue Type: Sub-task >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1625) Add (Unit) Tests for GPU functions
[ https://issues.apache.org/jira/browse/SYSTEMML-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1625: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add (Unit) Tests for GPU functions > -- > > Key: SYSTEMML-1625 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1625 > Project: SystemML > Issue Type: Sub-task > Components: Test >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1701) Fix the need to add force to -gpu always
[ https://issues.apache.org/jira/browse/SYSTEMML-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1701: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Fix the need to add force to -gpu always > > > Key: SYSTEMML-1701 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1701 > Project: SystemML > Issue Type: Sub-task > Components: Runtime >Affects Versions: SystemML 0.14 >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1138) Exception thrown when a GPU sparse-sparse matrix multiply is performed
[ https://issues.apache.org/jira/browse/SYSTEMML-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1138: Fix Version/s: (was: SystemML 1.0) SystemML 0.12 > Exception thrown when a GPU sparse-sparse matrix multiply is performed > -- > > Key: SYSTEMML-1138 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1138 > Project: SystemML > Issue Type: Bug >Reporter: Nakul Jindal >Assignee: Nakul Jindal > Fix For: SystemML 0.12 > > > For a simple program like so: > A = rand(rows=5, cols=10, sparsity=0.0003) > B = rand(rows=10, cols=100, sparsity=0.7) > C = A %*% B > print(toString(C)) > This is the exception: > Caused by: jcuda.CudaException: cudaErrorIllegalAddress > at jcuda.runtime.JCuda.checkResult(JCuda.java:460) > at jcuda.runtime.JCuda.cudaDeviceSynchronize(JCuda.java:7361) > at > org.apache.sysml.runtime.instructions.gpu.context.JCudaObject.columnMajorDenseToRowMajorSparse(JCudaObject.java:1130) > at > org.apache.sysml.runtime.matrix.data.LibMatrixCUDA.sparseDenseMatmult(LibMatrixCUDA.java:668) > at > org.apache.sysml.runtime.matrix.data.LibMatrixCUDA.eitherSparseMatmult(LibMatrixCUDA.java:573) > at > org.apache.sysml.runtime.matrix.data.LibMatrixCUDA.matmult(LibMatrixCUDA.java:538) > at > org.apache.sysml.runtime.instructions.gpu.AggregateBinaryGPUInstruction.processInstruction(AggregateBinaryGPUInstruction.java:98) > at > org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1568) NULL condition not check for Spark version in MLContext
[
https://issues.apache.org/jira/browse/SYSTEMML-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1568:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> NULL condition not check for Spark version in MLContext
> ---
>
> Key: SYSTEMML-1568
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1568
> Project: SystemML
> Issue Type: Bug
>Reporter: Niketan Pansare
>Assignee: Niketan Pansare
>Priority: Minor
> Fix For: SystemML 0.15
>
>
> I see following warning after starting pyspark shell:
> {code}
> 17/04/30 14:05:25 WARN MLContext: Apache Spark null or above is recommended
> for SystemML null
> Welcome to Apache SystemML!
> {code}
> To reproduce the warning, please use Spark 2.1:
> {code}
> # checkout current master
> mvn package -P distribution
> pip install target/systemml-1.0.0-incubating-SNAPSHOT-python.tgz
> pyspark
> >> run simple script with python mlcontext
> {code}
> [~deron] Can you please take a look at it ?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1589) conv2d_bias_add fails w/ NPE on lenet with random data
[
https://issues.apache.org/jira/browse/SYSTEMML-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1589:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> conv2d_bias_add fails w/ NPE on lenet with random data
> --
>
> Key: SYSTEMML-1589
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1589
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Niketan Pansare
> Fix For: SystemML 0.15
>
>
> The lenet dml script fails with a null pointer exception for random multi
> class data, generated with
> {code}
> X_full = rand(rows=6,cols=784);
> y_full = round(rand(rows=nrow(X_full), cols=1, min=1, max=10));
> {code}
> The detailed stacktrace is as follows:
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN.getRowInDenseFormat(LibMatrixDNN.java:1355)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN.doIm2colSparse(LibMatrixDNN.java:1382)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN.doIm2col(LibMatrixDNN.java:1421)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN.doLoopedIm2ColConv2d(LibMatrixDNN.java:406)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN.access$400(LibMatrixDNN.java:51)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN$ConvTask.call(LibMatrixDNN.java:1143)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixDNN$ConvTask.call(LibMatrixDNN.java:1076)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1661) Builtin functions bias_add and bias_multiply not documented
[ https://issues.apache.org/jira/browse/SYSTEMML-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1661: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Builtin functions bias_add and bias_multiply not documented > --- > > Key: SYSTEMML-1661 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1661 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Niketan Pansare >Priority: Minor > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-888) Add PNMF algorithm to SystemML
[ https://issues.apache.org/jira/browse/SYSTEMML-888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-888: --- Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Add PNMF algorithm to SystemML > -- > > Key: SYSTEMML-888 > URL: https://issues.apache.org/jira/browse/SYSTEMML-888 > Project: SystemML > Issue Type: Task > Components: Algorithms >Reporter: Deron Eriksson >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > > Add the Poisson Nonnegative Matrix Factorization algorithm to the SystemML > algorithms. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1878) Perftest: Performance issues MSVM 1M x 1K, sparse
[
https://issues.apache.org/jira/browse/SYSTEMML-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1878:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Perftest: Performance issues MSVM 1M x 1K, sparse
> -
>
> Key: SYSTEMML-1878
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1878
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
> Fix For: SystemML 0.15
>
>
> For the MSVM 1M x 1K, sparse performance test, the parfor optimizer currently
> selects a local plan although the data size is just 244MB. This task aims to
> make the necessary fixes to automatically compile this outer parfor loop to
> {{REMOTE_SPARK}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-455) OOM CP transpose in Spark hybrid mode
[
https://issues.apache.org/jira/browse/SYSTEMML-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-455:
---
Fix Version/s: (was: SystemML 1.0)
SystemML 0.14
> OOM CP transpose in Spark hybrid mode
> --
>
> Key: SYSTEMML-455
> URL: https://issues.apache.org/jira/browse/SYSTEMML-455
> Project: SystemML
> Issue Type: Bug
> Components: Compiler
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
> Fix For: SystemML 0.14
>
>
> The following data generation script failed with OOM in hybrid_spark
> execution mode (config: 20GB driver memory), whereas the same script runs
> fine with the same memory budget in hybrid_mr execution mode.
> {code}
> n = 3;
> B = Rand (rows = n, cols = n, min = -1, max = 1, pdf = "uniform", seed =
> 1234);
> v = exp (Rand (rows = n, cols = 1, min = -3, max = 3, pdf = "uniform", seed =
> 5678));
> A = t(B) %*% (B * v);
> write(A, "./tmp/A", format="binary");
> {code}
> The resulting hop explain output is as follows:
> {code}
> # Memory Budget local/remote = 13739MB/184320MB/8602MB
> # Degree of Parallelism (vcores) local/remote = 16/120
> PROGRAM
> --MAIN PROGRAM
> GENERIC (lines 4-12) [recompile=true]
> --(10) dg(rand) [3,3,1000,1000,9] [0,0,6866 -> 6866MB], CP
> --(21) r(t) (10) [3,3,1000,1000,9] [6866,0,6866 ->
> 13733MB], CP
> --(19) dg(rand) [3,1,1000,1000,3] [0,0,0 -> 0MB], CP
> --(20) u(exp) (19) [3,1,1000,1000,-1] [0,0,0 -> 0MB], CP
> --(22) b(*) (10,20) [3,3,1000,1000,-1] [6867,0,6866 -> 13733MB],
> CP
> --(23) ba(+*) (21,22) [3,3,1000,1000,-1] [13733,6866,6866 ->
> 27466MB], SPARK
> --(28) PWrite A (23) [3,3,1000,1000,-1] [6866,0,0 -> 6866MB], CP
> {code}
> The scripts fails at CP transpose with
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:414)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixReorg.transposeDenseToDense(LibMatrixReorg.java:752)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixReorg.transpose(LibMatrixReorg.java:136)
> at
> org.apache.sysml.runtime.matrix.data.LibMatrixReorg.reorg(LibMatrixReorg.java:105)
> at
> org.apache.sysml.runtime.matrix.data.MatrixBlock.reorgOperations(MatrixBlock.java:3458)
> at
> org.apache.sysml.runtime.instructions.cp.ReorgCPInstruction.processInstruction(ReorgCPInstruction.java:129)
> {code}
> It's noteworthy that the failing cp instructions requires 13733MB at a memory
> budget of 13739MB. The current guess is that Spark itself occupies
> substantial memory overhead which eventually leads to the OOM - we should
> adjust our memory budget in Spark execution modes to account for this
> overhead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1262) Keep track of parallelized RDDs and broadcasts
[ https://issues.apache.org/jira/browse/SYSTEMML-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1262: Fix Version/s: (was: SystemML 1.0) SystemML 0.14 > Keep track of parallelized RDDs and broadcasts > -- > > Key: SYSTEMML-1262 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1262 > Project: SystemML > Issue Type: Sub-task > Components: Test >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.14 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1319) Statistical estimates over compressed matrix blocks
[ https://issues.apache.org/jira/browse/SYSTEMML-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1319: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Statistical estimates over compressed matrix blocks > --- > > Key: SYSTEMML-1319 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1319 > Project: SystemML > Issue Type: Sub-task > Components: APIs, Runtime >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > > Statistical estimates like moment, cov, aggregate, table, median, and > quantiles can be efficiently computed over compressed matrix blocks by > mapping distinct items + counts to weighted statistical estimates. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1560) Cache-conscious compressed tsmm operations
[ https://issues.apache.org/jira/browse/SYSTEMML-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1560: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Cache-conscious compressed tsmm operations > -- > > Key: SYSTEMML-1560 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1560 > Project: SystemML > Issue Type: Task >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1682) Missing in-memory spark csv-reblock w/ unknown sizes
[ https://issues.apache.org/jira/browse/SYSTEMML-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1682: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Missing in-memory spark csv-reblock w/ unknown sizes > > > Key: SYSTEMML-1682 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1682 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1534) Multi-aggregates w/ dot products as aggregation roots
[ https://issues.apache.org/jira/browse/SYSTEMML-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1534: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Multi-aggregates w/ dot products as aggregation roots > - > > Key: SYSTEMML-1534 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1534 > Project: SystemML > Issue Type: Sub-task > Components: Compiler, Runtime >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1538) Improved dynamic recompilation (size update after rewrites)
[ https://issues.apache.org/jira/browse/SYSTEMML-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1538: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Improved dynamic recompilation (size update after rewrites) > --- > > Key: SYSTEMML-1538 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1538 > Project: SystemML > Issue Type: Sub-task > Components: Compiler >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > > Dynamic recompilation currently first updates matrix characteristics and > subsequently applied dynamic rewrites and operator selection which depend on > the updates stats. However, there are various scenarios where applied > rewrites simplify the propagation of statistics. Hence, we should > additionally update statistics after rewrites in order to increase the > potential of subsequent operator selection and code generation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1519) Old MLContext API setConfig only take affect on execute
[ https://issues.apache.org/jira/browse/SYSTEMML-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1519: Fix Version/s: (was: SystemML 1.0) Not Applicable > Old MLContext API setConfig only take affect on execute > --- > > Key: SYSTEMML-1519 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1519 > Project: SystemML > Issue Type: Bug >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: Not Applicable > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1881) Tuning parfor degree of parallelism for operations
[
https://issues.apache.org/jira/browse/SYSTEMML-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1881:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Tuning parfor degree of parallelism for operations
> --
>
> Key: SYSTEMML-1881
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1881
> Project: SystemML
> Issue Type: Task
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
> Fix For: SystemML 0.15
>
>
> Currently, we assign remaining parfor parallelism conservatively to
> operations of the parfor body. Consider, for example, a Kmeans or MSVM
> scenario with 10 runs or 10 classes respectively. On a box with 16 HW
> threads, we assign k=10 to the parfor and {{floor(16/10)}} to remaining
> operations. Since it is usually a good idea to slightly over-provision CPU in
> order to get full utilization (due to barriers at the end of each operation),
> we should tune this to {{round(16/10)}} which provides performance
> improvements of about 15% in above examples.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1879) Parfor remote spark w/ reuse of shared inputs
[ https://issues.apache.org/jira/browse/SYSTEMML-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1879: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Parfor remote spark w/ reuse of shared inputs > - > > Key: SYSTEMML-1879 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1879 > Project: SystemML > Issue Type: Sub-task > Components: APIs, Runtime >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > > Currently, we read shared inputs redundantly in each parfor worker. This > causes redundant read and is unnecessarily memory-inefficient. > This task aims to read shared inputs once per process and reuse them across > threads. The most elegant way of handling this is to reuse initially parsed > symbol table entries (instances of matrix objects), except for result > variables. Then the sharing happens automatically (similar to local parfor) > over the shared per-process buffer pool. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1877) Perfttest: Univariate statistics 1M x 1K fails on DPESP
[
https://issues.apache.org/jira/browse/SYSTEMML-1877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1877:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Perfttest: Univariate statistics 1M x 1K fails on DPESP
> ---
>
> Key: SYSTEMML-1877
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1877
> Project: SystemML
> Issue Type: Bug
>Affects Versions: SystemML 0.14
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
> Fix For: SystemML 0.15
>
>
> The univariate statistics script fails in hybrid_spark on 1M x 1K with the
> following exception (this issue has been introduced w/ SYSTEMML-1310)
> {code}
> Caused by: java.lang.RuntimeException: Unsupported partition format:
> COLUMN_WISE
> at
> org.apache.sysml.runtime.controlprogram.parfor.RemoteDPParForSparkWorker.(RemoteDPParForSparkWorker.java:100)
> at
> org.apache.sysml.runtime.controlprogram.parfor.RemoteDPParForSpark.runJob(RemoteDPParForSpark.java:98)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeRemoteSparkParForDP(ParForProgramBlock.java:1104)
> at
> org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:638)
> ... 14 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1871) Rework compiler/runtime predicate handling
[ https://issues.apache.org/jira/browse/SYSTEMML-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Glenn Weidner updated SYSTEMML-1871: Fix Version/s: (was: SystemML 1.0) SystemML 0.15 > Rework compiler/runtime predicate handling > -- > > Key: SYSTEMML-1871 > URL: https://issues.apache.org/jira/browse/SYSTEMML-1871 > Project: SystemML > Issue Type: Sub-task > Components: Compiler >Reporter: Matthias Boehm >Assignee: Matthias Boehm > Fix For: SystemML 0.15 > > > Currently, the handling of if, while, and for predicates exhibits a couple of > shortcomings. First, there are different representations for operations (as > single-root HOP DAGs) and literals (as dedicated constants). Second, the > runtime has to explicitly find intermediate variable names, remove rmvar > instructions, which is brittle and error-prone. Third, the special handling > of operations vs literals renders constant folding during dynamic > recompilation invalid because, we would have to handle the transitioning from > operation DAGs to constants accordingly. > This task aims to resolve all these issues, by properly compiling transient > writes to special predicate variables (e.g., _pred that are guaranteed not to > conflict with external variables). This requires a complete rework of the > entire predicate handling during compilation and runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (SYSTEMML-1862) Perftest: MSVM 800GB fails on buffer pool eviction
[
https://issues.apache.org/jira/browse/SYSTEMML-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Glenn Weidner updated SYSTEMML-1862:
Fix Version/s: (was: SystemML 1.0)
SystemML 0.15
> Perftest: MSVM 800GB fails on buffer pool eviction
> --
>
> Key: SYSTEMML-1862
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1862
> Project: SystemML
> Issue Type: Bug
>Reporter: Matthias Boehm
>Assignee: Matthias Boehm
> Fix For: SystemML 0.15
>
>
> {code}
> Caused by: org.apache.sysml.runtime.controlprogram.caching.CacheException:
> Eviction to local path
> /tmp/systemml/_p196865_1.12.34.56//cache/cache31072.dat (_mVar372) failed.
> at
> org.apache.sysml.runtime.controlprogram.caching.CacheableData.release(CacheableData.java:619)
> at
> org.apache.sysml.runtime.controlprogram.context.ExecutionContext.setMatrixOutput(ExecutionContext.java:426)
> at
> org.apache.sysml.runtime.instructions.cp.ScalarMatrixRelationalCPInstruction.processInstruction(ScalarMatrixRelationalCPInstruction.java:64)
> at
> org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:286)
> ... 6 more
> Caused by: java.util.NoSuchElementException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:721)
> at
> java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
> at
> java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
> at
> org.apache.sysml.runtime.controlprogram.caching.LazyWriteBuffer$EvictionQueue.removeFirst(LazyWriteBuffer.java:273)
> at
> org.apache.sysml.runtime.controlprogram.caching.LazyWriteBuffer.writeBlock(LazyWriteBuffer.java:82)
> at
> org.apache.sysml.runtime.controlprogram.caching.CacheableData.release(CacheableData.java:615)
> ... 9 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
