systemml git commit: [MINOR] DML Tips and Tricks Jupyter notebook

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 1adfc7266 -> 8e06ff3cc


[MINOR] DML Tips and Tricks Jupyter notebook

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/8e06ff3c
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/8e06ff3c
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/8e06ff3c

Branch: refs/heads/master
Commit: 8e06ff3cca487715ae47da25cd896f59a7fcd817
Parents: 1adfc72
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Fri Aug 18 21:49:47 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Fri Aug 18 22:12:32 2017 -0700

--
 ...DML Tips and Tricks (aka Fun With DML).ipynb | 753 +++
 1 file changed, 753 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/8e06ff3c/samples/jupyter-notebooks/DML
 Tips and Tricks (aka Fun With DML).ipynb
--
diff --git a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb
new file mode 100644
index 000..23d975a
--- /dev/null
+++ b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
@@ -0,0 +1,753 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"1. [Value-based join of two Matrices](#JoinMatrices)\n",
+"* [Filter Matrix to include only Frequent Column 
Values](#FilterMatrix)\n",
+"* [Construct (sparse) Matrix from (rowIndex, colIndex, values) 
triplets](#Construct_sparse_Matrix)\n",
+"* [Find and remove duplicates in columns or 
rows](#Find_and_remove_duplicates)\n",
+"* [Set based Indexing](#Set_based_Indexing)\n",
+"* [Group by Aggregate using Linear Algebra](#Multi_column_Sorting)\n",
+"* [Cumulative Summation with Decay Multiplier](#CumSum_Product)\n",
+"* [Invert Lower Triangular Matrix](#Invert_Lower_Triangular_Matrix)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+"collapsed": false,
+"scrolled": true
+   },
+   "outputs": [],
+   "source": [
+"from systemml import MLContext, dml, jvm_stdout\n",
+"ml = MLContext(sc)\n",
+"\n",
+"print (ml.buildTime())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"## Value-based join of two Matrices"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"Given matrix M1 and M2, join M1 on column 2 with M2 on column 2, and 
return matching rows of M1."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 55,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+  "M1 \n",
+  "1.000 1.000\n",
+  "2.000 3.000\n",
+  "3.000 3.000\n",
+  "4.000 4.000\n",
+  "5.000 3.000\n",
+  "6.000 4.000\n",
+  "7.000 1.000\n",
+  "8.000 2.000\n",
+  "9.000 1.000\n",
+  "\n",
+  "M2 \n",
+  "1.000 1.000\n",
+  "2.000 8.000\n",
+  "3.000 3.000\n",
+  "4.000 3.000\n",
+  "5.000 1.000\n",
+  "\n",
+  "M1[,2] joined with M2[,2], and return matching M1 rows\n",
+  "1.000 1.000\n",
+  "2.000 3.000\n",
+  "3.000 3.000\n",
+  "5.000 3.000\n",
+  "7.000 1.000\n",
+  "9.000 1.000\n",
+  "\n",
+  "SystemML Statistics:\n",
+  "Total execution time:\t\t0.001 sec.\n",
+  "Number of executed Spark inst:\t0.\n",
+  "\n",
+  "\n"
+ ]
+}
+   ],
+   "source": [
+"prog = \"\"\"\n",
+"M1 = matrix ('1 1 2 3 3 3 4 4 5 3 6 4 7 1 8 2 9 1', rows = 9, cols = 
2)\n",
+"M2 = matrix ('1 1 2 8 3 3 4 3 5 1', rows = 5, cols = 2)\n",
+"\n",
+"I = rowSums (outer (M1[,2], t(M2[,2]), \"==\"))  # I : 
indicator matrix for M1\n",
+"M12 = removeEmpty (

systemml git commit: [Minor] added cross validation example

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 754548190 -> cddd2a4f6


[Minor] added cross validation example

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/cddd2a4f
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/cddd2a4f
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/cddd2a4f

Branch: refs/heads/master
Commit: cddd2a4f60e22e8b621712135e5ed263b25343c0
Parents: 7545481
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Sep 11 15:30:29 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Sep 11 15:46:52 2017 -0700

--
 ...DML Tips and Tricks (aka Fun With DML).ipynb | 189 ++-
 1 file changed, 186 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/cddd2a4f/samples/jupyter-notebooks/DML
 Tips and Tricks (aka Fun With DML).ipynb
--
diff --git a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb
index 23d975a..c0391ce 100644
--- a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
+++ b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
@@ -4,7 +4,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
-"1. [Value-based join of two Matrices](#JoinMatrices)\n",
+"1. [Cross Validation](#CrossValidation)\n",
+"* [Value-based join of two Matrices](#JoinMatrices)\n",
 "* [Filter Matrix to include only Frequent Column 
Values](#FilterMatrix)\n",
 "* [Construct (sparse) Matrix from (rowIndex, colIndex, values) 
triplets](#Construct_sparse_Matrix)\n",
 "* [Find and remove duplicates in columns or 
rows](#Find_and_remove_duplicates)\n",
@@ -16,12 +17,20 @@
   },
   {
"cell_type": "code",
-   "execution_count": null,
+   "execution_count": 2,
"metadata": {
 "collapsed": false,
 "scrolled": true
},
-   "outputs": [],
+   "outputs": [
+{
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+  "2017-08-18 21:33:18 UTC\n"
+ ]
+}
+   ],
"source": [
 "from systemml import MLContext, dml, jvm_stdout\n",
 "ml = MLContext(sc)\n",
@@ -33,6 +42,180 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+"## Cross Validation"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"Perform kFold cross validation by running in parallel fold creation, 
training algorithm, test algorithm, and evaluation."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+  "Test data Xyi2\n",
+  "10.000 11.000 12.000 4.000\n",
+  "16.000 17.000 18.000 6.000\n",
+  "\n",
+  "Train data Xyni2\n",
+  "1.000 2.000 3.000 1.000\n",
+  "4.000 5.000 6.000 2.000\n",
+  "7.000 8.000 9.000 3.000\n",
+  "13.000 14.000 15.000 5.000\n",
+  "\n",
+  "w2\n",
+  "95.000\n",
+  "106.000\n",
+  "117.000\n",
+  "\n",
+  "stats2\n",
+  "8938.000\n",
+  "\n",
+  "\n",
+  "Test data Xyi3\n",
+  "1.000 2.000 3.000 1.000\n",
+  "7.000 8.000 9.000 3.000\n",
+  "\n",
+  "Train data Xyni3\n",
+  "4.000 5.000 6.000 2.000\n",
+  "10.000 11.000 12.000 4.000\n",
+  "13.000 14.000 15.000 5.000\n",
+  "16.000 17.000 18.000 6.000\n",
+  "\n",
+  "w3\n",
+  "209.000\n",
+  "226.000\n",
+  "243.000\n",
+  "\n",
+  "stats3\n",
+  "6844.000\n",
+  "\n",
+  "\n",
+  "Test data Xyi1\n",
+  "4.000 5.000 6.000 2.000\n",
+  "13.000 14.000 15.000 5.000\n",
+  "\n",
+  &

systemml git commit: [maven-release-plugin] prepare for next development iteration

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 0e4051011 -> db378759e


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/db378759
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/db378759
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/db378759

Branch: refs/heads/master
Commit: db378759e697ccfd66b13dad1ebd7ad4c7209f23
Parents: 0e40510
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Dec 4 17:07:15 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Dec 4 17:07:15 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/db378759/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8f042b6..78b0428 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.0.0
+   1.1.0-SNAPSHOT
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   v1.0.0-rc2
+   HEAD


JIRA

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc2 [created] 63f725caa

systemml git commit: [maven-release-plugin] prepare release v1.0.0-rc2

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 696fe3f6d -> 0e4051011


[maven-release-plugin] prepare release v1.0.0-rc2


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/0e405101
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/0e405101
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/0e405101

Branch: refs/heads/master
Commit: 0e4051011ca99f6c550c13eb2d9b2e0d68f44c84
Parents: 696fe3f
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Dec 4 17:07:01 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Dec 4 17:07:01 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/0e405101/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 78b0428..8f042b6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.1.0-SNAPSHOT
+   1.0.0
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.0.0-rc2


JIRA

systemml git commit: [maven-release-plugin] prepare release v1.0.0-rc1

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 22905278c -> 228662bf5


[maven-release-plugin] prepare release v1.0.0-rc1


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/228662bf
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/228662bf
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/228662bf

Branch: refs/heads/master
Commit: 228662bf52f283bddd395b2e8d60e54ed1e60972
Parents: 2290527
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Dec 4 14:01:05 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Dec 4 14:01:05 2017 -0800

--
 pom.xml | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/228662bf/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 6201c77..da9c7d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -10,8 +10,7 @@
WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See 
the 
License for the * specific language governing permissions and 
limitations 
* under the License. -->
-http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
-   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0

org.apache
@@ -19,7 +18,7 @@
18

org.apache.systemml
-   1.0.0-SNAPSHOT
+   1.0.0
systemml
jar
SystemML
@@ -35,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.0.0-rc1


JIRA
@@ -580,9 +579,7 @@
package


-   
+   

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc1 [created] 12edf2856

svn commit: r23387 - /dev/systemml/1.0.0-rc2/

[reinwald]

Author: reinwald
Date: Tue Dec  5 01:10:28 2017
New Revision: 23387

Log:
Apache SystemML 1.0.0-rc2

Added:
dev/systemml/1.0.0-rc2/
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.sha512

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz
==
Binary file - no diff available.

Propchange: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc Tue Dec  5 01:10:28 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJaJfHHAAoJEGOG/rIi7k//zkQP/3nGqjjMeD2uroXbbNHMcRQ6
+0Tr71D5lHRjgOfLfcXq5ChkYT/SzauFarTVuR1mCgZZO3Y50fAADiiU1gDdrnnDI
+vgKMlREVkFnMkIcHrPjUm/x8y/maA7NhkhClrbgGqw4VusqH1EuFTYyP3fsGATtq
+qoL8XSxNty2/MSYo71exN9LYvUJJ3SUcVgVKillCBel9IgRNhrgGZykVyFo19GXU
+PwrmRNDDmw/MWW9frtnngDC429JINspsKnWqv3yAvIndsPLpKBgazdFtD1DVXbyU
+vQTIi17V8Ik7hG4F8Eq6sBdUBPuTLA2+7daqD4xGt3MLd3ahJXHuaUfYoZPxOEok
+GxWBfYEwbkYkPu/hWgp7Eu1fJRw6I3wLMrvMfVnGQJXTpzg49N1SmGW5mcZH5uow
+VRJbQfR1t9lkz8CkXVKgvhEnUGUxET9NlEivZKz551QZnjWM0Lj48LHMjsHeCFjx
+EC29t4DMTegXrYx3kDdepGIyuK+86GVIa69zu1sl7NAtXakRNPW7Xsvi/p6iv8Jn
+cdIzAWX/nxhHIzzmIHPlzQZLZor4uREPcfyYTE9FE6JlfcID1WAll97gkTRHYH2C
+TZmXUVpy0NFK4k1mED/SamqgWS9yhDY8VsE3kbc86gsFmAC/tP4AV5Ymj82dZIKf
+RWEpwMImvOz34rplENyg
+=n5RV
+-END PGP SIGNATURE-

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5 (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5 Tue Dec  5 01:10:28 2017
@@ -0,0 +1 @@
+c5259134d26fea053b4ae06009c630a1 *systemml-1.0.0-bin.tgz

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512 (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512 Tue Dec  5 01:10:28 
2017
@@ -0,0 +1 @@
+122099035c0ff7496fe01e4b4a1070f9528014fd1af1fdf9811288e9f2c2f581b2035e948717009348aaedd43e83355deacab9f89037e0ac80cb6814dc80541a
  systemml-1.0.0-bin.tgz

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip
==
Binary file - no diff available.

Propchange: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc Tue Dec  5 01:10:28 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJaJfHHAAoJEGOG/rIi7k//X/4P/05dKnNk+54jSmRGyxrm4DQT
+16pSG+wr6/RJOWIZ7u6rwxSC4q4JJH9Bo0qh23PJ/AOpt2H/jW44p+CtomMFC3U5
+qpTTuddMdo9mjSI9N99KM8WGCuClMFxMduDioH055T6uzqr6pHS+9Z8NNu1jAlQb
+/HlMz3ZBWFEsgQ8uhJST3sszb3iaJbEVZXZ5mR//mlh/ta0T53Sm68O+u2hwqOiL
+FOIjlqz8QnSDLfK8nNUFJos/+F10mBa2CsJXzpubZOlH5g+DLHWk44PiHxQOTNJU
+UCCHNAG9NRVlqTy/0uBrS9tQhmnoLFR+utfOrejYf8nERQSQuzM43JwDiAMvVJOv
+HWgWa5EJvTr5q2sdszK8/3G8JuefbXQGMiv0Tw41MrelYU2UxtoEH0eCNJTN1MPD
+HIziDHK3uOB3BkQ6UErYE1TRYpE2yobHsjPZYS9Vr06rONWMBJi8vdr0IgaQehdl
+yeLPB73f6nrHds3vhbfE2vP0T1ntrLRsJhax0EXhPHL4TTQgDLjEbBKmlW4D1ZGW
+ZKE3NT5oMXALiWJH2KuDPnWGjzsWQ93aJM600K/guWQzYpMjHBv6rbxiTezH/lq5

systemml git commit: [maven-release-plugin] prepare release v1.0.0-rc3

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master e4c2880ac -> 0f984ebf4


[maven-release-plugin] prepare release v1.0.0-rc3


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/0f984ebf
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/0f984ebf
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/0f984ebf

Branch: refs/heads/master
Commit: 0f984ebf473678229419e51f0f812e6571478dc9
Parents: e4c2880
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Dec 4 20:57:04 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Dec 4 20:57:04 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/0f984ebf/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 78b0428..4ee4641 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.1.0-SNAPSHOT
+   1.0.0
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.0.0-rc3


JIRA

systemml git commit: [maven-release-plugin] prepare for next development iteration

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 0f984ebf4 -> cf9396b5c


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/cf9396b5
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/cf9396b5
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/cf9396b5

Branch: refs/heads/master
Commit: cf9396b5c1a6ddffb0b64e8cd4bd45d5cd237323
Parents: 0f984eb
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon Dec 4 20:57:20 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon Dec 4 20:57:20 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/cf9396b5/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 4ee4641..78b0428 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.0.0
+   1.1.0-SNAPSHOT
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   v1.0.0-rc3
+   HEAD


JIRA

svn commit: r23694 - /dev/systemml/1.0.0-rc2/ /release/systemml/1.0.0/

[reinwald]

Author: reinwald
Date: Wed Dec 13 01:40:06 2017
New Revision: 23694

Log:
Move SystemML 1.0.0 from dev to release

Added:
release/systemml/1.0.0/
  - copied from r23693, dev/systemml/1.0.0-rc2/
Removed:
dev/systemml/1.0.0-rc2/

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/branch-1.0.0 [created] 6519f3383

svn commit: r1818739 - in /systemml/site/docs/1.0.0: ./ api/ api/java/ api/java/org/ api/java/org/apache/ api/java/org/apache/sysml/ api/java/org/apache/sysml/api/ api/java/org/apache/sysml/api/class-

[reinwald]

Author: reinwald
Date: Wed Dec 20 06:24:37 2017
New Revision: 1818739

URL: http://svn.apache.org/viewvc?rev=1818739=rev
Log:
Add 1.0.0 docs to website


[This commit notification would consist of 267 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

[35/50] [abbrv] systemml git commit: [SYSTEMML-1952] Fix codegen row ops w/ non-allocated dense side inputs

[reinwald]

[SYSTEMML-1952] Fix codegen row ops w/ non-allocated dense side inputs

This patch fixes special cases with empty (and not allocated) dense side
inputs. Before we always converted sparse and empty dense inputs to
dense blocks. With the introduction of sparse side inputs, the special
case of empty dense blocks was not yet handled correctly. We now simply
construct sparse side inputs and make these robust to work with
non-existing sparse blocks. 

Furthermore, this includes a minor performance improvement, where sparse
row side inputs do not need to reset the temporary row for each access
in case of non-existing sparse blocks.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/897d29d0
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/897d29d0
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/897d29d0

Branch: refs/heads/master
Commit: 897d29d04a8d6b89a7467e65d350aa98362f67b0
Parents: dd513ff
Author: Matthias Boehm 
Authored: Thu Oct 26 22:45:06 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Oct 26 23:38:21 2017 -0700

--
 .../sysml/runtime/codegen/SpoofOperator.java  | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/897d29d0/src/main/java/org/apache/sysml/runtime/codegen/SpoofOperator.java
--
diff --git a/src/main/java/org/apache/sysml/runtime/codegen/SpoofOperator.java 
b/src/main/java/org/apache/sysml/runtime/codegen/SpoofOperator.java
index a614ded..21fdd35 100644
--- a/src/main/java/org/apache/sysml/runtime/codegen/SpoofOperator.java
+++ b/src/main/java/org/apache/sysml/runtime/codegen/SpoofOperator.java
@@ -104,10 +104,10 @@ public abstract class SpoofOperator implements 
Serializable
else {
b[i-offset] = new 
SideInput(DataConverter.convertToDoubleVector(in), null, clen);
LOG.warn(getClass().getName()+": 
Converted "+in.getNumRows()+"x"+in.getNumColumns()+
-   ", nnz="+in.getNonZeros()+" 
sideways input matrix from sparse to dense.");  
+   ", nnz="+in.getNonZeros()+" 
sideways input matrix from sparse to dense.");
}
}
-   else if( in.isInSparseFormat() && in.isAllocated() ) {
+   else if( in.isInSparseFormat() || !in.isAllocated() ) {
b[i-offset] = new SideInput(null, in, clen);
}
else {
@@ -128,16 +128,14 @@ public abstract class SpoofOperator implements 
Serializable
boolean containsSparse = false;
for( int i=0; i 
1);
+   containsSparse |= (tmp.mdat != null && tmp.clen > 1);
}
if( !containsSparse )
return input;
SideInput[] ret = new SideInput[input.length];
for( int i=0; i 
1) ?
+   ret[i] = (tmp.mdat != null && tmp.clen > 1) ?
(row ? new SideInputSparseRow(tmp) : 
new SideInputSparseCell(tmp)) : tmp;
}
@@ -274,9 +272,10 @@ public abstract class SpoofOperator implements Serializable

private void nextRow(int r) {
currRowIndex = r;
-   Arrays.fill(values, 0);
SparseBlock sblock = mdat.getSparseBlock();
-   if( sblock != null && !sblock.isEmpty(r) ) {
+   if( sblock == null ) return;
+   Arrays.fill(values, 0);
+   if( !sblock.isEmpty(r) ) {
int apos = sblock.pos(r);
int alen = sblock.size(r);
int[] aix = sblock.indexes(r);
@@ -298,7 +297,8 @@ public abstract class SpoofOperator implements Serializable
super(in.ddat, in.mdat, in.clen);
}
public double next(int rowIndex, int 

[44/50] [abbrv] systemml git commit: [MINOR][SYSTEMML-1979] Fix codegen plan enumeration (cost bound)

[reinwald]

[MINOR][SYSTEMML-1979] Fix codegen plan enumeration (cost bound)

A recent fix re-enabled the structual pruning in the codegen optimizer.
So far, we passed the current cost bound into recursive calls for
conditionally independent subproblems. In rare cases, this lead to
invalid pruning because the subproblem is solved with a constant
assignment for points not included in the subproblem.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/381d1d6a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/381d1d6a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/381d1d6a

Branch: refs/heads/master
Commit: 381d1d6a9ac356d4b834963c71f8872837acf35e
Parents: d907efc
Author: Matthias Boehm 
Authored: Mon Oct 30 22:13:33 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 30 22:13:33 2017 -0700

--
 .../hops/codegen/opt/PlanSelectionFuseCostBasedV2.java  | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/381d1d6a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
index 4d8a7bc..1f670b3 100644
--- 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
+++ 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
@@ -160,8 +160,8 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
}
 
//enumerate and cost plans, returns optional plan
-   boolean[] bestPlan = enumPlans(memo, part, costs, 
rgraph, 
-   part.getMatPointsExt(), 0, 
Double.MAX_VALUE);
+   boolean[] bestPlan = enumPlans(memo, part,
+   costs, rgraph, part.getMatPointsExt(), 0);

//prune memo table wrt best plan and select plans
HashSet visited = new HashSet<>();
@@ -194,17 +194,17 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
 * @param rgraph reachability graph of interesting materialization 
points
 * @param matPoints sorted materialization points (defined the search 
space)
 * @param off offset for recursive invocation, indicating the fixed 
plan part
-* @param bestC currently known best plan costs (used of upper bound)
 * @return optimal assignment of materialization points
 */
private static boolean[] enumPlans(CPlanMemoTable memo, PlanPartition 
part, StaticCosts costs, 
-   ReachabilityGraph rgraph, InterestingPoint[] matPoints, int 
off, double bestC)
+   ReachabilityGraph rgraph, InterestingPoint[] matPoints, int off)
{
//scan linearized search space, w/ skips for branch and bound 
pruning
//and structural pruning (where we solve conditionally 
independent problems)
//bestC is monotonically non-increasing and serves as the upper 
bound
-   long len = UtilFunctions.pow(2, matPoints.length-off);
+   final long len = UtilFunctions.pow(2, matPoints.length-off);
boolean[] bestPlan = null;
+   double bestC = Double.MAX_VALUE;
long numEvalPlans = 0, numEvalPartPlans = 0;

for( long i=0; i

[20/50] [abbrv] systemml git commit: [MINOR] Improved autoencoder scripts (ordering row-shuffle/z-transform)

[reinwald]

[MINOR] Improved autoencoder scripts (ordering row-shuffle/z-transform)

This patch makes a minor performance improvement to the autoencoder
script. So far, we first applied z-transform followed but a random row
shuffling. Since the z-transform turns sparse datasets into dense, we
now first perform the row shuffling, which makes this permutation matrix
multiply significantly faster and can avoid unnecessary evictions.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a51f8e81
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a51f8e81
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a51f8e81

Branch: refs/heads/master
Commit: a51f8e8190281e4c32978d430c08c1b83083faa2
Parents: 1191dbf
Author: Matthias Boehm 
Authored: Sun Oct 22 21:01:37 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 21:01:37 2017 -0700

--
 scripts/staging/autoencoder-2layer.dml | 8 
 1 file changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/a51f8e81/scripts/staging/autoencoder-2layer.dml
--
diff --git a/scripts/staging/autoencoder-2layer.dml 
b/scripts/staging/autoencoder-2layer.dml
index a17d86e..9eee8ae 100644
--- a/scripts/staging/autoencoder-2layer.dml
+++ b/scripts/staging/autoencoder-2layer.dml
@@ -150,15 +150,15 @@ max_epochs = $EPOCH
 n = nrow(X)
 m = ncol(X)
 
+#randomly reordering rows
+permut = table(seq(1,n,1), order(target=Rand(rows=n, cols=1, min=0, max=1, 
pdf="uniform"), by=1, index.return=TRUE), n, n)
+X = permut %*% X
+
 #z-transform, whitening operator is better
 means = colSums(X)/n
 stds = sqrt((colSums(X^2)/n - means*means)*n/(n-1)) + 1e-17
 X = (X - means)/stds
 
-#randomly reordering rows
-permut = table(seq(1,n,1), order(target=Rand(rows=n, cols=1, min=0, max=1, 
pdf="uniform"), by=1, index.return=TRUE), n, n)
-X = permut %*% X
-
 W1 = sqrt(6)/sqrt(m + num_hidden1) * Rand(rows=num_hidden1, cols=m, min=-1, 
max=1, pdf="uniform")
 b1 = matrix(0, rows=num_hidden1, cols=1)
 W2 = sqrt(6)/sqrt(num_hidden1 + num_hidden2) * Rand(rows=num_hidden2, 
cols=num_hidden1, min=-1, max=1, pdf="uniform")

[25/50] [abbrv] systemml git commit: [SYSTEMML-1903, 1968] Fix codegen row templates w/ partial unknowns

[reinwald]

[SYSTEMML-1903,1968] Fix codegen row templates w/ partial unknowns

After recent codegen optimizer changes, GLM was failing during initial
compilation when used through JMLC. The reason was an incorrect handling
of partial unknowns that led to vector operations although the output
was known to be scalar. 


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/8f4ecdce
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/8f4ecdce
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/8f4ecdce

Branch: refs/heads/master
Commit: 8f4ecdce23780a4b820cb79865322d05ba1b9411
Parents: 70ab072
Author: Matthias Boehm 
Authored: Tue Oct 24 20:39:22 2017 -0700
Committer: Matthias Boehm 
Committed: Tue Oct 24 20:39:22 2017 -0700

--
 .../org/apache/sysml/hops/codegen/template/TemplateRow.java | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/8f4ecdce/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
index 9da04dc..e14fbd3 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
@@ -361,8 +361,9 @@ public class TemplateRow extends TemplateBase
// if one input is a matrix then we need to do vector 
by scalar operations
if( (hop.getInput().get(0).getDim1() > 1 && 
hop.getInput().get(0).getDim2() > 1)
|| (hop.getInput().get(1).getDim1() > 1 && 
hop.getInput().get(1).getDim2() > 1)
-   || (!(hop.dimsKnown() && 
hop.getInput().get(0).dimsKnown() && hop.getInput().get(1).dimsKnown()) 
-   && 
(cdata1.getDataType().isMatrix() || cdata2.getDataType().isMatrix(
+   || (!(hop.dimsKnown() && 
hop.getInput().get(0).dimsKnown() && hop.getInput().get(1).dimsKnown())
+   && (hop.getDim2() != 1) //not a known 
vector output
+   && (cdata1.getDataType().isMatrix() || 
cdata2.getDataType().isMatrix(
{
if( HopRewriteUtils.isBinary(hop, 
SUPPORTED_VECT_BINARY) ) {
if( TemplateUtils.isMatrix(cdata1) && 
(TemplateUtils.isMatrix(cdata2) 

[37/50] [abbrv] systemml git commit: [SYSTEMML-1976] Performance codegen outer ops w/ ultra-sparse inputs

[reinwald]

[SYSTEMML-1976] Performance codegen outer ops w/ ultra-sparse inputs

This patch improves the performance of codegen outer operations
(especially left mm), for ultra-sparse inputs. On ultra-sparse inputs,
the allocation and maintenance of column indexes can become a
bottleneck. Accordingly, this patch adds a special case for ultra-sparse
matrices.

For the core update rules of ALS-CG, this patch improved the performance
as follows (10 iterations over the amazon review dataset):
Left (t(t(U) %*% (W * (U %*% t(V): 125.7s -> 14.2s
Right ((W * (U %*% t(V))) %*% V): 25.2s -> 17.9s


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/ede870de
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/ede870de
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/ede870de

Branch: refs/heads/master
Commit: ede870de8a7b01bd44ac3b6bcfe7f0e86b1c93c8
Parents: 83e01b0
Author: Matthias Boehm 
Authored: Fri Oct 27 22:01:36 2017 -0700
Committer: Matthias Boehm 
Committed: Fri Oct 27 22:01:36 2017 -0700

--
 .../runtime/codegen/SpoofOuterProduct.java  | 77 +---
 1 file changed, 52 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/ede870de/src/main/java/org/apache/sysml/runtime/codegen/SpoofOuterProduct.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/codegen/SpoofOuterProduct.java 
b/src/main/java/org/apache/sysml/runtime/codegen/SpoofOuterProduct.java
index ac8dc57..26a661a 100644
--- a/src/main/java/org/apache/sysml/runtime/codegen/SpoofOuterProduct.java
+++ b/src/main/java/org/apache/sysml/runtime/codegen/SpoofOuterProduct.java
@@ -28,6 +28,7 @@ import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 
+import org.apache.sysml.hops.OptimizerUtils;
 import org.apache.sysml.runtime.DMLRuntimeException;
 import org.apache.sysml.runtime.compress.CompressedMatrixBlock;
 import org.apache.sysml.runtime.instructions.cp.DoubleObject;
@@ -391,7 +392,7 @@ public abstract class SpoofOuterProduct extends 
SpoofOperator
c[0] = sum;
}

-   private void executeSparse(SparseBlock sblock,  double[] u, double[] v, 
double[][] b, double[] scalars,
+   private void executeSparse(SparseBlock sblock, double[] u, double[] v, 
double[][] b, double[] scalars,
double[] c, int m, int n, int k, long nnz, OutProdType type, 
int rl, int ru, int cl, int cu) 
{
boolean left = (_outerProductType== 
OutProdType.LEFT_OUTER_PRODUCT);
@@ -401,37 +402,63 @@ public abstract class SpoofOuterProduct extends 
SpoofOperator
//blocksize is chosen such that we reuse each  Ui/Vj vector on 
average 8 times,
//with custom blocksizeJ for wdivmm_left to avoid LLC misses on 
output.
final int blocksizeI = (int) (8L*m*n/nnz);
-   final int blocksizeJ = left ? 
Math.max(8,Math.min(L2_CACHESIZE/(k*8), blocksizeI)) : blocksizeI;
-   int[] curk = new int[Math.min(blocksizeI,ru-rl)];

-   for( int bi = rl; bi < ru; bi+=blocksizeI ) 
+   if( OptimizerUtils.getSparsity(m, n, nnz) < 
MatrixBlock.ULTRA_SPARSITY_TURN_POINT ) //ultra-sparse
{
-   int bimin = Math.min(ru, bi+blocksizeI);
-   //prepare starting indexes for block row
-   for( int i=bi; i=0) ? index : n;
+   index = wpos + ((index>=0) ? index : n);
+   for( ; index

[45/50] [abbrv] systemml git commit: [SYSTEMML-1979] Improved codegen cost model (sparsity, minor fixes)

[reinwald]

[SYSTEMML-1979] Improved codegen cost model (sparsity, minor fixes)

This patch improves the codegen cost model to correctly account for the
compute workload of sparse matrix multiplications as well as sparse and
dense input sizes. Furthermore, this patch also includes minor fixes of
eviction- and broadcast-aware cost corrections. Overall, these changes
address special cases of sparse large-scale (i.e., distributed)
scenarios, where the codegen optimizer picked suboptimal plans.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/cb1d7928
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/cb1d7928
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/cb1d7928

Branch: refs/heads/master
Commit: cb1d792826411b144909b2168929c4f33620b02a
Parents: 381d1d6
Author: Matthias Boehm 
Authored: Tue Oct 31 22:25:38 2017 -0700
Committer: Matthias Boehm 
Committed: Wed Nov 1 02:25:34 2017 -0700

--
 .../opt/PlanSelectionFuseCostBasedV2.java   | 29 ++--
 1 file changed, 21 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/cb1d7928/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
index 1f670b3..9302573 100644
--- 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
+++ 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
@@ -89,7 +89,7 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
private static final double WRITE_BANDWIDTH_IO  =  512*1024*1024;  
//512MB/s
private static final double WRITE_BANDWIDTH_MEM =  2d*1024*1024*1024;  
//2GB/s
private static final double READ_BANDWIDTH_MEM  = 32d*1024*1024*1024;  
//32GB/s
-   private static final double READ_BANDWIDTH_BROADCAST = 
WRITE_BANDWIDTH_MEM/4;
+   private static final double READ_BANDWIDTH_BROADCAST = 
WRITE_BANDWIDTH_IO/4;
private static final double COMPUTE_BANDWIDTH  =   2d*1024*1024*1024   
//1GFLOPs/core
* 
InfrastructureAnalyzer.getLocalParallelism();

@@ -329,7 +329,7 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
//get partition input reads (at least read once)
for( Long hopID : part.getInputs() ) {
Hop hop = memo.getHopRefs().get(hopID);
-   costs += getSize(hop) * 8 / READ_BANDWIDTH_MEM;
+   costs += getSafeMemEst(hop) / READ_BANDWIDTH_MEM;
}
return costs;
}
@@ -355,6 +355,16 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
.mapToDouble(e -> e.getValue()).sum();
}

+   private static double sumInputMemoryEstimates(CPlanMemoTable memo, 
CostVector vect) {
+   return vect.inSizes.keySet().stream()
+   .mapToDouble(e -> 
getSafeMemEst(memo.getHopRefs().get(e))).sum();
+   }
+   
+   private static double getSafeMemEst(Hop hop) {
+   return !hop.dimsKnown() ? getSize(hop) * 8
+   : hop.getMemEstimate();
+   }
+   
private static long getSize(Hop hop) {
return Math.max(hop.getDim1(),1) 
* Math.max(hop.getDim2(),1);
@@ -603,7 +613,6 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection

private static boolean isRowAggOp(Hop hop){
return (hop instanceof AggUnaryOp || hop instanceof AggBinaryOp
-   || (hop instanceof IndexingOp && 
HopRewriteUtils.isColumnRangeIndexing((IndexingOp)hop))
|| HopRewriteUtils.isBinary(hop, OpOp2.CBIND));
}

@@ -840,19 +849,20 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection

//add costs for opened fused operator
if( opened ) {
+   double memInputs = sumInputMemoryEstimates(memo, 
costVect);
double tmpCosts = costVect.outSize * 8 / 
WRITE_BANDWIDTH_MEM
-   + Math.max(costVect.getInputSize() * 8 / 
READ_BANDWIDTH_MEM,
+   + Math.max(memInputs / READ_BANDWIDTH_MEM,
costVect.computeCosts/ COMPUTE_BANDWIDTH);
//read correction for distributed computation
-   Hop 

[33/50] [abbrv] systemml git commit: [SYSTEMML-446] Minimize the chances of eviction for right indexing operation

[reinwald]

[SYSTEMML-446] Minimize the chances of eviction for right indexing
operation

Indexing is only supported on GPU if:
1. the input is of type matrix AND
2. the input is less than 2GB. 

The second condition is added for following reason:
1. Indexing is a purely memory-bound operation and doesnot benefit
drastically from pushing down to GPU.
2. By forcing larger matrices to GPU (for example: training dataset), we
run into risk of unnecessary evictions of parameters and the gradients.
For single precision, there is additional overhead of converting
training dataset to single precision every single time it is evicted.

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/591a0f77
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/591a0f77
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/591a0f77

Branch: refs/heads/master
Commit: 591a0f7754d85e0b9a170cb4385bc84d52e641e8
Parents: f040674
Author: Niketan Pansare 
Authored: Wed Oct 25 21:10:21 2017 -0700
Committer: Niketan Pansare 
Committed: Wed Oct 25 21:10:21 2017 -0700

--
 src/main/java/org/apache/sysml/hops/IndexingOp.java | 11 +--
 1 file changed, 9 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/591a0f77/src/main/java/org/apache/sysml/hops/IndexingOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/IndexingOp.java 
b/src/main/java/org/apache/sysml/hops/IndexingOp.java
index 5989c66..0b8509a 100644
--- a/src/main/java/org/apache/sysml/hops/IndexingOp.java
+++ b/src/main/java/org/apache/sysml/hops/IndexingOp.java
@@ -102,8 +102,15 @@ public class IndexingOp extends Hop
return false;
}
else {
-   // only matrix indexing is supported on GPU
-   return (getDataType() == DataType.MATRIX);
+   // Indexing is only supported on GPU if:
+   // 1. the input is of type matrix AND
+   // 2. the input is less than 2GB. 
+   // The second condition is added for following reason:
+   // 1. Indexing is a purely memory-bound operation and 
doesnot benefit drastically from pushing down to GPU.
+   // 2. By forcing larger matrices to GPU (for example: 
training dataset), we run into risk of unnecessary evictions of 
+   // parameters and the gradients. For single precision, 
there is additional overhead of converting training dataset 
+   // to single precision every single time it is evicted.
+   return (getDataType() == DataType.MATRIX) && 
getInputMemEstimate() < 2e+9;
}
}
 

[18/50] [abbrv] systemml git commit: [MINOR] Fix analysis of sparse-safeness for codegen cell/magg ops

[reinwald]

[MINOR] Fix analysis of sparse-safeness for codegen cell/magg ops

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/c70cb116
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/c70cb116
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/c70cb116

Branch: refs/heads/master
Commit: c70cb1166f4ec6c79d10248727a3eb7b85f70360
Parents: 78a3808
Author: Matthias Boehm 
Authored: Sun Oct 22 18:57:35 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 18:57:35 2017 -0700

--
 .../apache/sysml/hops/codegen/template/TemplateCell.java  | 10 ++
 1 file changed, 6 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/c70cb116/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
index c9b0734..4f3d4f4 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateCell.java
@@ -322,10 +322,12 @@ public class TemplateCell extends TemplateBase
protected boolean isSparseSafe(List roots, Hop mainInput, 
List outputs, List aggOps, boolean onlySum) {
boolean ret = true;
for( int i=0; i

[26/50] [abbrv] systemml git commit: [SYSTEMML-540] Include the memory requirement of each layer in the summary table of Caffe2DML

[reinwald]

[SYSTEMML-540] Include the memory requirement of each layer in the summary 
table of Caffe2DML

- This helps the user to estimate the batch size she should set for
  optimal performance.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/881caa9b
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/881caa9b
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/881caa9b

Branch: refs/heads/master
Commit: 881caa9ba508b029f72f27d468bb33805704c7cb
Parents: 8f4ecdc
Author: Niketan Pansare 
Authored: Wed Oct 25 15:40:21 2017 -0700
Committer: Niketan Pansare 
Committed: Wed Oct 25 15:42:04 2017 -0700

--
 docs/beginners-guide-caffe2dml.md   | 37 +---
 .../org/apache/sysml/api/dl/Caffe2DML.scala | 46 ++--
 2 files changed, 63 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/881caa9b/docs/beginners-guide-caffe2dml.md
--
diff --git a/docs/beginners-guide-caffe2dml.md 
b/docs/beginners-guide-caffe2dml.md
index 4d6b7fd..8814283 100644
--- a/docs/beginners-guide-caffe2dml.md
+++ b/docs/beginners-guide-caffe2dml.md
@@ -64,22 +64,27 @@ lenet.summary()
 Output:
 
 ```
-+-+---+--++-+---+-+
-| Name|   Type|Output|  Weight| Bias|Top|   
Bottom|
-+-+---+--++-+---+-+
-|mnist|   Data| (, 1, 28, 28)|| |mnist,mnist|  
   |
-|conv1|Convolution|(, 32, 28, 28)|   [32 X 25]| [32 X 1]|  conv1|
mnist|
-|relu1|   ReLU|(, 32, 28, 28)|| |  relu1|
conv1|
-|pool1|Pooling|(, 32, 14, 14)|| |  pool1|
relu1|
-|conv2|Convolution|(, 64, 14, 14)|  [64 X 800]| [64 X 1]|  conv2|
pool1|
-|relu2|   ReLU|(, 64, 14, 14)|| |  relu2|
conv2|
-|pool2|Pooling|  (, 64, 7, 7)|| |  pool2|
relu2|
-|  ip1|   InnerProduct| (, 512, 1, 1)|[3136 X 512]|[1 X 512]|ip1|
pool2|
-|relu3|   ReLU| (, 512, 1, 1)|| |  relu3|  
ip1|
-|drop1|Dropout| (, 512, 1, 1)|| |  drop1|
relu3|
-|  ip2|   InnerProduct|  (, 10, 1, 1)|  [512 X 10]| [1 X 10]|ip2|
drop1|
-| loss|SoftmaxWithLoss|  (, 10, 1, 1)|| |   
loss|ip2,mnist|
-+-+---+--++-+---+-+
++-+---+--++-+---+-++
+| Name|   Type|Output|  Weight| Bias|Top|   
Bottom|Memory* (train/test)|
++-+---+--++-+---+-++
+|mnist|   Data| (, 1, 28, 28)|| |mnist,mnist|  
   | 1/0|
+|conv1|Convolution|(, 32, 28, 28)|   [32 X 25]| [32 X 1]|  conv1|
mnist|   25/12|
+|relu1|   ReLU|(, 32, 28, 28)|| |  relu1|
conv1|   25/12|
+|pool1|Pooling|(, 32, 14, 14)|| |  pool1|
relu1| 6/3|
+|conv2|Convolution|(, 64, 14, 14)|  [64 X 800]| [64 X 1]|  conv2|
pool1|38/7|
+|relu2|   ReLU|(, 64, 14, 14)|| |  relu2|
conv2|12/6|
+|pool2|Pooling|  (, 64, 7, 7)|| |  pool2|
relu2| 3/2|
+|  ip1|   InnerProduct| (, 512, 1, 1)|[3136 X 512]|[1 X 512]|ip1|
pool2|  797/13|
+|relu3|   ReLU| (, 512, 1, 1)|| |  relu3|  
ip1| 1/0|
+|drop1|Dropout| (, 512, 1, 1)|| |  drop1|
relu3| 1/0|
+|  ip2|   InnerProduct|  (, 10, 1, 1)|  [512 X 10]| [1 X 10]|ip2|
drop1| 3/0|
+| loss|SoftmaxWithLoss|  (, 10, 1, 1)|| |   
loss|ip2,mnist| 0/0|
++-+---+--++-+---+-++
+
+Total number of layer outputs/errors/weights/bias/gradients: 
5568768/5568768/1662752/618/106455680
+Total memory requirements for parameters* for train/test: 910/55
+[Advanced] Key network statistics to compute intermediate CP overhead 
batchSize/maxThreads/1-thread im2col*(sum, max)/1-thread reshape_col*(sum, 
max): 64/48/(1, 1)/(0, 0).
+* => memory in megabytes assuming the parameters are in double precision and 
in dense 

[09/50] [abbrv] systemml git commit: [MINOR] [SYSTEMML-540] Reset the _cachedParams to avoid incorrect sizes

[reinwald]

[MINOR] [SYSTEMML-540] Reset the _cachedParams to avoid incorrect sizes

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/259814e6
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/259814e6
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/259814e6

Branch: refs/heads/master
Commit: 259814e6c00021c643c33867906d0c5d8dc4bc5e
Parents: 3a7f38e
Author: Niketan Pansare 
Authored: Tue Oct 17 16:52:19 2017 -0700
Committer: Niketan Pansare 
Committed: Tue Oct 17 16:52:19 2017 -0700

--
 src/main/java/org/apache/sysml/hops/ConvolutionOp.java | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/259814e6/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java 
b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
index e4ed32b..50a7ca3 100644
--- a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
+++ b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
@@ -639,6 +639,9 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
return;
}

+   // Reset the _cachedParams to avoid incorrect sizes
+   _cachedParams = new ConvolutionParameters(-1, -1, -1, -1, -1, 
-1, -1, -1, -1, -1, -1, _maxNumThreads);
+   
switch(op) 
{
case MAX_POOLING:

[11/50] [abbrv] systemml git commit: [MINOR] Fix missing warning on truncation of matrix/frame toString

[reinwald]

[MINOR] Fix missing warning on truncation of matrix/frame toString 

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/5b8d6265
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/5b8d6265
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/5b8d6265

Branch: refs/heads/master
Commit: 5b8d62659b2e5727bebcaf0d2681fc4ecd4ea85f
Parents: 60ad522
Author: Matthias Boehm 
Authored: Tue Oct 17 20:54:01 2017 -0700
Committer: Matthias Boehm 
Committed: Tue Oct 17 21:39:58 2017 -0700

--
 .../cp/ParameterizedBuiltinCPInstruction.java   | 16 +++-
 1 file changed, 15 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/5b8d6265/src/main/java/org/apache/sysml/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java
 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java
index e8a5f4a..f6532d7 100644
--- 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java
+++ 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/ParameterizedBuiltinCPInstruction.java
@@ -26,6 +26,7 @@ import org.apache.sysml.lops.Lop;
 import org.apache.sysml.parser.ParameterizedBuiltinFunctionExpression;
 import org.apache.sysml.parser.Statement;
 import org.apache.sysml.runtime.DMLRuntimeException;
+import org.apache.sysml.runtime.controlprogram.caching.CacheBlock;
 import org.apache.sysml.runtime.controlprogram.caching.CacheableData;
 import org.apache.sysml.runtime.controlprogram.caching.FrameObject;
 import org.apache.sysml.runtime.controlprogram.caching.MatrixObject;
@@ -328,10 +329,12 @@ public class ParameterizedBuiltinCPInstruction extends 
ComputationCPInstruction
String out = null;
if( data instanceof MatrixObject ) {
MatrixBlock matrix = (MatrixBlock) 
data.acquireRead();
+   warnOnTrunction(matrix, rows, cols);
out = DataConverter.toString(matrix, sparse, 
separator, lineseparator, rows, cols, decimal);
}
else if( data instanceof FrameObject ) {
FrameBlock frame = (FrameBlock) 
data.acquireRead();
+   warnOnTrunction(frame, rows, cols);
out = DataConverter.toString(frame, sparse, 
separator, lineseparator, rows, cols, decimal);
}
else {
@@ -342,6 +345,17 @@ public class ParameterizedBuiltinCPInstruction extends 
ComputationCPInstruction
}
else {
throw new DMLRuntimeException("Unknown opcode : " + 
opcode);
-   }   
+   }
+   }
+   
+   private void warnOnTrunction(CacheBlock data, int rows, int cols) {
+   //warn on truncation because users might not be aware and use 
toString for verification
+   if( (getParam("rows")==null && data.getNumRows()>rows)
+   || (getParam("cols")==null && 
data.getNumColumns()>cols) )
+   {
+   LOG.warn("Truncating 
"+data.getClass().getSimpleName()+" of size "
+   + data.getNumRows()+"x"+data.getNumColumns()+" 
to "+rows+"x"+cols+". "
+   + "Use toString(X, rows=..., cols=...) if 
necessary.");
+   }
}
 }

[47/50] [abbrv] systemml git commit: [SYSTEMML-1981] Fix graceful value type casts on function invocations

[reinwald]

[SYSTEMML-1981] Fix graceful value type casts on function invocations

The existing graceful value type casts - on function invocations with
wrong value type - incorrectly took the input parameter type instead of
the function parameter type for comparison. This patch fixes this
casting issue, which avoids unnecessary warnings on function invocations
and recompilation exceptions for boolean parameters.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a2f0598c
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a2f0598c
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a2f0598c

Branch: refs/heads/master
Commit: a2f0598c606db16e75790cdbc3dbe37dc32d89a0
Parents: fc47891
Author: Matthias Boehm 
Authored: Wed Nov 1 21:37:49 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Nov 2 00:39:16 2017 -0700

--
 .../cp/FunctionCallCPInstruction.java   | 29 
 .../functions/misc/FunctionReturnTest.java  | 78 
 .../functions/misc/FunctionReturnBoolean.dml| 34 +
 .../functions/misc/FunctionReturnInt.dml| 34 +
 .../functions/misc/ZPackageSuite.java   |  1 +
 5 files changed, 160 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/a2f0598c/src/main/java/org/apache/sysml/runtime/instructions/cp/FunctionCallCPInstruction.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/FunctionCallCPInstruction.java
 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/FunctionCallCPInstruction.java
index 402d4a5..b901dfc 100644
--- 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/FunctionCallCPInstruction.java
+++ 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/FunctionCallCPInstruction.java
@@ -60,7 +60,6 @@ public class FunctionCallCPInstruction extends CPInstruction {
private FunctionCallCPInstruction(String namespace, String functName, 
ArrayList boundInParamOperands,
ArrayList boundInParamNames, ArrayList 
boundOutParamNames, String istr) {
super(null, functName, istr);
-
_cptype = CPINSTRUCTION_TYPE.External;
_functionName = functName;
_namespace = namespace;
@@ -72,7 +71,7 @@ public class FunctionCallCPInstruction extends CPInstruction {
 
public static FunctionCallCPInstruction parseInstruction(String str) 
throws DMLRuntimeException 
-   {   
+   {
//schema: extfunct, fname, num inputs, num outputs, inputs, 
outputs
String[] parts = 
InstructionUtils.getInstructionPartsWithValueType ( str );
String namespace = parts[1];
@@ -94,8 +93,7 @@ public class FunctionCallCPInstruction extends CPInstruction {
return new FunctionCallCPInstruction ( namespace,functionName, 
boundInParamOperands, boundInParamNames, 
boundOutParamNames, str );
}
-
-   
+   
@Override
public Instruction preprocessInstruction(ExecutionContext ec)
throws DMLRuntimeException 
@@ -114,7 +112,7 @@ public class FunctionCallCPInstruction extends 
CPInstruction {
@Override
public void processInstruction(ExecutionContext ec) 
throws DMLRuntimeException 
-   {   
+   {
if( LOG.isTraceEnabled() ){
LOG.trace("Executing instruction : " + this.toString());
}
@@ -130,19 +128,19 @@ public class FunctionCallCPInstruction extends 
CPInstruction {

// create bindings to formal parameters for given function call
// These are the bindings passed to the FunctionProgramBlock 
for function execution 
-   LocalVariableMap functionVariables = new LocalVariableMap();

+   LocalVariableMap functionVariables = new LocalVariableMap();
for( int i=0; i

[22/50] [abbrv] systemml git commit: [SYSTEMML-1970] Performance sparse conv2d operations w/ native libs

[reinwald]

[SYSTEMML-1970] Performance sparse conv2d operations w/ native libs

This patch improves the performance of sparse conv2d operations with
enabled native libraries (BLAS and native conv2d ops). So far, we called
custom native ops for sparse or dense inputs/filters, which for certain
scenarios internally converted the sparse to dense inputs/filters
respectively. With this patch, we now decide the implementation with
sparsity awareness. Additionally, this also includes a minor improvement
that avoids unnecessary nnz maintenance after native BLAS class.

On an end-to-end cnn application this patch improved performance from
605s to 349s per epoch w/ selective native ops (490s with Java ops).

Furthermore, this also fixes a recently introduced issues of
thread-local nnz maintenance in the native conv2d backward data op.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/2c37d9f0
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/2c37d9f0
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/2c37d9f0

Branch: refs/heads/master
Commit: 2c37d9f03117d118c9d10c6839c4b1d60a4a9afc
Parents: 596005a
Author: Matthias Boehm 
Authored: Mon Oct 23 19:33:04 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 23 23:44:02 2017 -0700

--
 .../cp/ConvolutionCPInstruction.java|  6 +-
 .../sysml/runtime/matrix/data/LibMatrixDNN.java | 10 ---
 .../LibMatrixDNNConv2dBackwardDataHelper.java   |  3 +-
 .../runtime/matrix/data/LibMatrixDNNHelper.java | 71 +---
 4 files changed, 50 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/2c37d9f0/src/main/java/org/apache/sysml/runtime/instructions/cp/ConvolutionCPInstruction.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/ConvolutionCPInstruction.java
 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/ConvolutionCPInstruction.java
index 2c7b972..c6b4698 100644
--- 
a/src/main/java/org/apache/sysml/runtime/instructions/cp/ConvolutionCPInstruction.java
+++ 
b/src/main/java/org/apache/sysml/runtime/instructions/cp/ConvolutionCPInstruction.java
@@ -363,10 +363,8 @@ public class ConvolutionCPInstruction extends 
UnaryCPInstruction {
}
else {
outputBlock = new MatrixBlock(N, C*H*W, 
false).allocateBlock();
-   
if(instOpcode.equalsIgnoreCase("maxpooling_backward"))
-   
LibMatrixDNN.maxpoolingBackward(matBlock, dout, outputBlock, params, false);
-   else
-   
LibMatrixDNN.maxpoolingBackward(matBlock, dout, outputBlock, params, true);
+   LibMatrixDNN.maxpoolingBackward(matBlock, dout, 
outputBlock, params, 
+   
!instOpcode.equalsIgnoreCase("maxpooling_backward"));
}
ec.releaseMatrixInput(_in2.getName(), 
getExtendedOpcode());
}

http://git-wip-us.apache.org/repos/asf/systemml/blob/2c37d9f0/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
index ac66e51..096574a 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
@@ -34,7 +34,6 @@ import org.apache.sysml.runtime.DMLRuntimeException;
 import org.apache.sysml.runtime.instructions.InstructionUtils;
 import org.apache.sysml.runtime.matrix.operators.BinaryOperator;
 import org.apache.sysml.runtime.util.ConvolutionUtils;
-import org.apache.sysml.utils.Statistics;
 
 /*
  * This class allows users to invoke deep learning related operations 
@@ -161,9 +160,6 @@ public class LibMatrixDNN {
if(params.bias != null && params.bias.isInSparseFormat())
params.bias.sparseToDense(); // Since bias is extremely 
small array

-   if(isEligibleForConv2dSparse(params))
-   Statistics.numNativeSparseConv2dCalls.increment();
-   
long nnz = execute(LibMatrixDNNHelper.getConv2dWorkers(params), 
params);

//post-processing: maintain nnz
@@ -183,9 +179,6 @@ public class LibMatrixDNN {
public static void conv2dBackwardData(MatrixBlock filter, MatrixBlock 
dout, MatrixBlock outputBlock, ConvolutionParameters params) 

[49/50] [abbrv] systemml git commit: [SYSTEMML-1983] New codegen cplan rewrite framework (micro optims)

[reinwald]

[SYSTEMML-1983] New codegen cplan rewrite framework (micro optims)

This patch refactors the code generator by moving smaller micro
optimizations from the templates into a new cplan rewrite framework to
avoid redundancy and inconsistency across templates and to improve
debuggability. The goal is NOT to provide a fusion-aware rewrite
framework, but simply to apply smaller rewrites for better code quality.
An example rewrite is rowSums(X!=0) -> rowNnzs(X), which avoids an
unnecessary row intermediate and is realized as a meta data operation
for sparse input rows.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/e1f5866a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/e1f5866a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/e1f5866a

Branch: refs/heads/master
Commit: e1f5866a5c2e2100d68124b74a33a9022e89dd09
Parents: ee6060b
Author: Matthias Boehm 
Authored: Thu Nov 2 00:15:11 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Nov 2 00:39:18 2017 -0700

--
 .../sysml/hops/codegen/SpoofCompiler.java   |  43 +-
 .../hops/codegen/template/CPlanOpRewriter.java  | 130 +++
 .../hops/codegen/template/TemplateCell.java |  11 +-
 .../codegen/template/TemplateOuterProduct.java  |   9 +-
 .../hops/codegen/template/TemplateRow.java  |  10 --
 .../functions/codegen/RowAggTmplTest.java   |  20 ++-
 .../scripts/functions/codegen/rowAggPattern34.R |  32 +
 .../functions/codegen/rowAggPattern34.dml   |  29 +
 8 files changed, 219 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/e1f5866a/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java 
b/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
index 4af8540..51cd0a2 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
@@ -39,7 +39,6 @@ import org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM;
 import org.apache.sysml.conf.ConfigurationManager;
 import org.apache.sysml.conf.DMLConfig;
 import org.apache.sysml.hops.codegen.cplan.CNode;
-import org.apache.sysml.hops.codegen.cplan.CNodeBinary.BinType;
 import org.apache.sysml.hops.codegen.cplan.CNodeCell;
 import org.apache.sysml.hops.codegen.cplan.CNodeData;
 import org.apache.sysml.hops.codegen.cplan.CNodeMultiAgg;
@@ -53,7 +52,6 @@ import 
org.apache.sysml.hops.codegen.opt.PlanSelectionFuseCostBased;
 import org.apache.sysml.hops.codegen.opt.PlanSelectionFuseCostBasedV2;
 import org.apache.sysml.hops.codegen.opt.PlanSelectionFuseNoRedundancy;
 import org.apache.sysml.hops.codegen.cplan.CNodeTpl;
-import org.apache.sysml.hops.codegen.cplan.CNodeUnary.UnaryType;
 import org.apache.sysml.hops.codegen.template.TemplateBase;
 import org.apache.sysml.hops.codegen.template.TemplateBase.CloseType;
 import org.apache.sysml.hops.codegen.template.TemplateBase.TemplateType;
@@ -61,6 +59,7 @@ import 
org.apache.sysml.hops.codegen.template.CPlanCSERewriter;
 import org.apache.sysml.hops.codegen.template.CPlanMemoTable;
 import org.apache.sysml.hops.codegen.template.CPlanMemoTable.MemoTableEntry;
 import org.apache.sysml.hops.codegen.template.CPlanMemoTable.MemoTableEntrySet;
+import org.apache.sysml.hops.codegen.template.CPlanOpRewriter;
 import org.apache.sysml.hops.codegen.template.TemplateUtils;
 import org.apache.sysml.hops.recompile.RecompileStatus;
 import org.apache.sysml.hops.recompile.Recompiler;
@@ -68,7 +67,6 @@ import org.apache.sysml.hops.AggUnaryOp;
 import org.apache.sysml.hops.Hop;
 import org.apache.sysml.hops.Hop.OpOp1;
 import org.apache.sysml.hops.HopsException;
-import org.apache.sysml.hops.LiteralOp;
 import org.apache.sysml.hops.OptimizerUtils;
 import org.apache.sysml.hops.rewrite.HopRewriteUtils;
 import org.apache.sysml.hops.rewrite.ProgramRewriteStatus;
@@ -684,13 +682,15 @@ public class SpoofCompiler
private static HashMap> 
cleanupCPlans(CPlanMemoTable memo, HashMap> cplans) 
{
HashMap> cplans2 = new HashMap<>();
+   CPlanOpRewriter rewriter = new CPlanOpRewriter();
CPlanCSERewriter cse = new CPlanCSERewriter();

for( Entry> e : cplans.entrySet() ) {
CNodeTpl tpl = e.getValue().getValue();
Hop[] inHops = e.getValue().getKey();

-   //perform common subexpression elimination
+   //perform 

[07/50] [abbrv] systemml git commit: [SYSTEMML-540] Reduce the number of unknowns in ConvolutionOp

[reinwald]

[SYSTEMML-540] Reduce the number of unknowns in ConvolutionOp

- This commit reduces the unknowns during dynamic recompilation by inferring the
input's height/width of ConvolutionOp based on its parent's output's
height/width.
- Additionally, for developer debugging, I have guarded the functionality
with the flag INFER_TENSOR_SHAPE_FROM_PARENT_CONV_OP and have added
sufficient documentation to explain how these dimensions are inferred.

Closes #685.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/5adb330d
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/5adb330d
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/5adb330d

Branch: refs/heads/master
Commit: 5adb330deffa5479475338316bf47193d0c31da4
Parents: 2ca2d8a
Author: Niketan Pansare 
Authored: Mon Oct 16 15:44:37 2017 -0700
Committer: Niketan Pansare 
Committed: Mon Oct 16 15:45:39 2017 -0700

--
 .../org/apache/sysml/hops/ConvolutionOp.java| 170 ---
 1 file changed, 144 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/5adb330d/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java 
b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
index e732fb8..e4ed32b 100644
--- a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
+++ b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
@@ -32,11 +32,21 @@ import org.apache.sysml.runtime.DMLRuntimeException;
 import org.apache.sysml.runtime.instructions.gpu.context.GPUContextPool;
 import org.apache.sysml.runtime.matrix.MatrixCharacteristics;
 import org.apache.sysml.runtime.matrix.data.ConvolutionParameters;
-
 import java.util.ArrayList;
 
 public class ConvolutionOp extends Hop  implements MultiThreadedHop
 {  
+   // 
-
+   // This flag allows us to compile plans with less unknowns and also 
serves as future tensorblock integration.
+   // By default, these flags are turned on.
+   
+   // When this flag is turned on, we attempt to check the parent 
convolution hop for unknown dimensions.
+   // For example: in case of conv -> maxpool, the input 
channel/height/width of maxpool will match output channel/height/width of conv.
+   private static final boolean INFER_TENSOR_SHAPE_FROM_PARENT_CONV_OP = 
true;
+   // This guards us from cases where the user provides incorrect C,H,W 
parameters.
+   private static final boolean THROW_ERROR_IF_INFERRED_SHAPE_MISMATCH = 
true;
+   // 
-
+   
private Hop.ConvOp op;
 
private int _maxNumThreads = -1; //-1 for unlimited
@@ -475,17 +485,21 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
// input_shape1, input_shape2, input_shape3, input_shape4, 
// filter_shape1, filter_shape2, filter_shape3, filter_shape4
ConvolutionParameters parseInput() throws DMLRuntimeException {
+   
+   Hop imageHeightHop = null; Hop filterHeightHop = null;
if(op == ConvOp.MAX_POOLING_BACKWARD 
|| op == ConvOp.DIRECT_CONV2D 
|| op == ConvOp.DIRECT_CONV2D_BACKWARD_FILTER
|| op == ConvOp.DIRECT_CONV2D_BACKWARD_DATA) {
+   imageHeightHop = getInput().get(8);
+   filterHeightHop = getInput().get(12);
_cachedParams.setIfUnknown(
getInput().get(6),
getInput().get(7), 
-   getInput().get(8), 
+   imageHeightHop, 
getInput().get(9), 
getInput().get(10), 
-   getInput().get(12), 
+   filterHeightHop, 
getInput().get(13), 
getInput().get(2), 
getInput().get(3), 
@@ -493,22 +507,127 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
getInput().get(5), _maxNumThreads);
}
else {
+   imageHeightHop = getInput().get(7);
+   filterHeightHop = getInput().get(11);
_cachedParams.setIfUnknown(
 

[36/50] [abbrv] systemml git commit: [SYSTEMML-1693] New IPA pass for function inlining after rewrites

[reinwald]

[SYSTEMML-1693] New IPA pass for function inlining after rewrites

The existing function inlining (of single-statement-block functions)
happens during validate, i.e., before rewrites. However, after constant
propagation, constant folding, branch removal, and statement block
merge, often additional opportunities arise. This patch exploits such
opportunities by adding a new inter-procedural analysis pass for
inlining single-statement-block functions. To limit the potential
exponential increase of program size, we only inline functions with less
or equal than 10 operations other than dataops and literals.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/83e01b02
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/83e01b02
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/83e01b02

Branch: refs/heads/master
Commit: 83e01b02891e54612e2ab82d3f2f805eee2f09f1
Parents: 897d29d
Author: Matthias Boehm 
Authored: Fri Oct 27 20:23:56 2017 -0700
Committer: Matthias Boehm 
Committed: Fri Oct 27 20:23:56 2017 -0700

--
 .../sysml/hops/ipa/FunctionCallGraph.java   |  31 +++-
 .../sysml/hops/ipa/IPAPassInlineFunctions.java  | 156 +++
 .../sysml/hops/ipa/InterProceduralAnalysis.java |   4 +-
 .../org/apache/sysml/parser/DMLTranslator.java  |   8 +-
 .../functions/misc/IPAFunctionInliningTest.java | 122 +++
 .../test/integration/functions/misc/IfTest.java | 155 +-
 .../scripts/functions/misc/IPAFunInline1.dml|  34 
 .../scripts/functions/misc/IPAFunInline2.dml|  36 +
 .../scripts/functions/misc/IPAFunInline3.dml|  39 +
 .../scripts/functions/misc/IPAFunInline4.dml|  36 +
 .../functions/misc/ZPackageSuite.java   |   1 +
 11 files changed, 532 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/83e01b02/src/main/java/org/apache/sysml/hops/ipa/FunctionCallGraph.java
--
diff --git a/src/main/java/org/apache/sysml/hops/ipa/FunctionCallGraph.java 
b/src/main/java/org/apache/sysml/hops/ipa/FunctionCallGraph.java
index d719da7..4735f47 100644
--- a/src/main/java/org/apache/sysml/hops/ipa/FunctionCallGraph.java
+++ b/src/main/java/org/apache/sysml/hops/ipa/FunctionCallGraph.java
@@ -55,8 +55,9 @@ public class FunctionCallGraph
//program-wide function call operators per target function
//(mapping from function keys to set of its function calls)
private final HashMap _fCalls;
+   private final HashMap _fCallsSB;

-   //subset of direct or indirect recursive functions  
+   //subset of direct or indirect recursive functions
private final HashSet _fRecursive;

/**
@@ -68,6 +69,7 @@ public class FunctionCallGraph
public FunctionCallGraph(DMLProgram prog) {
_fGraph = new HashMap<>();
_fCalls = new HashMap<>();
+   _fCallsSB = new HashMap<>();
_fRecursive = new HashSet<>();

constructFunctionCallGraph(prog);
@@ -82,6 +84,7 @@ public class FunctionCallGraph
public FunctionCallGraph(StatementBlock sb) {
_fGraph = new HashMap<>();
_fCalls = new HashMap<>();
+   _fCallsSB = new HashMap<>();
_fRecursive = new HashSet<>();

constructFunctionCallGraph(sb);
@@ -125,6 +128,21 @@ public class FunctionCallGraph
}

/**
+* Returns all statement blocks that contain a function operator
+* calling the given function.
+* 
+* @param fkey function key of called function,
+*  null indicates the main program and returns an empty list
+* @return list of statement blocks
+*/
+   public List getFunctionCallsSB(String fkey) {
+   //main program cannot have function calls
+   if( fkey == null )
+   return Collections.emptyList();
+   return _fCallsSB.get(fkey);
+   }
+   
+   /**
 * Indicates if the given function is either directly or indirectly 
recursive.
 * An example of an indirect recursive function is foo2 in the 
following call
 * chain: foo1 - foo2 - foo1.
@@ -135,7 +153,7 @@ public class FunctionCallGraph
 */
public boolean isRecursiveFunction(String fnamespace, String fname) {
return isRecursiveFunction(
-   DMLProgram.constructFunctionKey(fnamespace, fname));

+   DMLProgram.constructFunctionKey(fnamespace, fname));
}
  

[14/50] [abbrv] systemml git commit: [SYSTEMML-446] [SYSTEMML-702] Updated the sparse matrix multiplication to minimize sparse-to-dense as well as dense-to-sparse conversion

[reinwald]

[SYSTEMML-446] [SYSTEMML-702] Updated the sparse matrix multiplication to 
minimize sparse-to-dense as well as dense-to-sparse conversion

1. The goal of this PR is not to improve performance (for example: by 
considering the cost of sparse-to-dense vs FLOPs required given a memory 
budget) but instead to minimize sparse-to-dense conversion in the GPU matrix 
multiplication operator.

2. If matmult uses unnecessary sparse-to-dense conversion, then we run into
  risk of one of the two situations:
- In best case some of the matmult won't be pushed to GPU under worst-case 
memory budget.
- On other hand, if these conversion are not accounted for, they may cause OOMs.

3. Every operator (except dense-sparse matrix multiplication) uses only memory 
allocated to input and output matrices.

4. Since there is no CuSPARSE kernel for dense-sparse matrix multiplication 
operator, we either have to transpose the output after performing sparse-dense 
matrix multiplication or perform dense-dense matrix multiplication after 
converting the sparse input to dense format.

Closes #686.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/6de8f051
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/6de8f051
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/6de8f051

Branch: refs/heads/master
Commit: 6de8f051daaefdab403c0edd3c7beb30c9619033
Parents: 323dd72
Author: Niketan Pansare 
Authored: Fri Oct 20 12:29:20 2017 -0800
Committer: Niketan Pansare 
Committed: Fri Oct 20 13:29:20 2017 -0700

--
 .../java/org/apache/sysml/hops/AggBinaryOp.java |  30 +-
 .../gpu/AggregateBinaryGPUInstruction.java  |   3 +-
 .../instructions/gpu/GPUInstruction.java|   1 +
 .../instructions/gpu/context/CSRPointer.java|   7 +-
 .../instructions/gpu/context/GPUObject.java |   2 +-
 .../runtime/matrix/data/LibMatrixCUDA.java  | 541 +--
 .../runtime/matrix/data/LibMatrixCuMatMult.java | 480 
 .../test/gpu/MatrixMultiplicationOpTest.java|  98 +++-
 8 files changed, 581 insertions(+), 581 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/6de8f051/src/main/java/org/apache/sysml/hops/AggBinaryOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/AggBinaryOp.java 
b/src/main/java/org/apache/sysml/hops/AggBinaryOp.java
index cfa99a4..d733d6a 100644
--- a/src/main/java/org/apache/sysml/hops/AggBinaryOp.java
+++ b/src/main/java/org/apache/sysml/hops/AggBinaryOp.java
@@ -371,40 +371,16 @@ public class AggBinaryOp extends Hop implements 
MultiThreadedHop
double ret = 0;
 
if (isGPUEnabled()) {
-   // In GPU Mode, intermediate memory is only needed in 
case of one of the matrix blocks is sparse
-   // When sparse block is converted to dense and a dense 
MM takes place, we need (dim1 * dim2)
-   // When dense block is converted to sparse and a sparse 
MM takes place, we need (dim1 * dim2 * 2)
-
Hop in1 = _input.get(0);
Hop in2 = _input.get(1);
double in1Sparsity = 
OptimizerUtils.getSparsity(in1.getDim1(), in1.getDim2(), in1.getNnz());
double in2Sparsity = 
OptimizerUtils.getSparsity(in2.getDim1(), in2.getDim2(), in2.getNnz());
-
boolean in1Sparse = in1Sparsity < 
MatrixBlock.SPARSITY_TURN_POINT;
boolean in2Sparse = in2Sparsity < 
MatrixBlock.SPARSITY_TURN_POINT;
-
-   boolean in1UltraSparse = in1Sparsity < 
MatrixBlock.ULTRA_SPARSITY_TURN_POINT;
-   boolean in2UltraSparse = in2Sparsity < 
MatrixBlock.ULTRA_SPARSITY_TURN_POINT;
-
-   // For Matmult X * Y, if X is sparse, Y is dense, X is 
converted to dense
-   // If X is ultrasparse, Y is converted to sparse
-   if (in1Sparse ^ in2Sparse) { // one sparse, one dense
-   if (in1Sparse) {
-   if (in1UltraSparse) {
-   ret += 2 * 
OptimizerUtils.estimateSizeExactSparsity(in2.getDim1(), in2.getDim2(), 
in2.getNnz());
-   } else {
-   ret += 
OptimizerUtils.estimateSizeExactSparsity(in1.getDim1(), in1.getDim2(), 
in1.getNnz());
-   }
-   } else if (in2Sparse) {
-   if (in2UltraSparse) {
-   ret += 2 * 

[12/50] [abbrv] systemml git commit: [SYSTEMML-1967] Fix spark rand instruction (#partitions for sparse)

[reinwald]

[SYSTEMML-1967] Fix spark rand instruction (#partitions for sparse)

This patch fixes the spark rand instruction to create the correct number
of partitions under awareness of sparsity. So far, this method called a
primitive for size estimation with the number of non-zeros instead of
the sparsity, which led to dense estimates. 

Furthermore, this patch also fixes minor configuration issues of enabled
hand-coded fused operators.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/4f29b348
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/4f29b348
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/4f29b348

Branch: refs/heads/master
Commit: 4f29b3485f4eb8a58aebd41eef22c5d0f92d632f
Parents: 5b8d626
Author: Matthias Boehm 
Authored: Tue Oct 17 23:09:40 2017 -0700
Committer: Matthias Boehm 
Committed: Tue Oct 17 23:09:40 2017 -0700

--
 src/main/java/org/apache/sysml/hops/BinaryOp.java | 14 --
 .../java/org/apache/sysml/hops/OptimizerUtils.java|  2 +-
 .../runtime/instructions/spark/RandSPInstruction.java |  2 +-
 3 files changed, 10 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/4f29b348/src/main/java/org/apache/sysml/hops/BinaryOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/BinaryOp.java 
b/src/main/java/org/apache/sysml/hops/BinaryOp.java
index 58bbc8f..76c1a64 100644
--- a/src/main/java/org/apache/sysml/hops/BinaryOp.java
+++ b/src/main/java/org/apache/sysml/hops/BinaryOp.java
@@ -1373,8 +1373,8 @@ public class BinaryOp extends Hop
 
private static boolean requiresPartitioning( Hop rightInput )
{
-   return (   rightInput.dimsKnown() //known input size 
-&& rightInput.getDim1()*rightInput.getDim2() > 
DistributedCacheInput.PARTITION_SIZE);
+   return ( rightInput.dimsKnown() //known input size 
+   && rightInput.getDim1()*rightInput.getDim2() > 
DistributedCacheInput.PARTITION_SIZE);
}

public static boolean requiresReplication( Hop left, Hop right )
@@ -1393,9 +1393,10 @@ public class BinaryOp extends Hop
long m1_cpb = left.getColsInBlock();

//MR_BINARY_UAGG_CHAIN only applied if result is column/row 
vector of MV binary operation.
-   if( right instanceof AggUnaryOp && right.getInput().get(0) == 
left  //e.g., P / rowSums(P)
+   if( OptimizerUtils.ALLOW_OPERATOR_FUSION
+   && right instanceof AggUnaryOp && 
right.getInput().get(0) == left  //e.g., P / rowSums(P)
&& AggUnaryOp) right).getDirection() == 
Direction.Row && m1_dim2 > 1 && m1_dim2 <= m1_cpb ) //single column block
-   ||  (((AggUnaryOp) right).getDirection() == Direction.Col 
&& m1_dim1 > 1 && m1_dim1 <= m1_rpb ))) //single row block
+   || (((AggUnaryOp) right).getDirection() == 
Direction.Col && m1_dim1 > 1 && m1_dim1 <= m1_rpb ))) //single row block
{
return MMBinaryMethod.MR_BINARY_UAGG_CHAIN;
}
@@ -1430,9 +1431,10 @@ public class BinaryOp extends Hop
}

//MR_BINARY_UAGG_CHAIN only applied if result is column/row 
vector of MV binary operation.
-   if( right instanceof AggUnaryOp && right.getInput().get(0) == 
left  //e.g., P / rowSums(P)
+   if( OptimizerUtils.ALLOW_OPERATOR_FUSION
+   && right instanceof AggUnaryOp && 
right.getInput().get(0) == left  //e.g., P / rowSums(P)
&& AggUnaryOp) right).getDirection() == 
Direction.Row && m1_dim2 > 1 && m1_dim2 <= m1_cpb ) //single column block
-   ||  (((AggUnaryOp) right).getDirection() == Direction.Col 
&& m1_dim1 > 1 && m1_dim1 <= m1_rpb ))) //single row block
+   || (((AggUnaryOp) right).getDirection() == 
Direction.Col && m1_dim1 > 1 && m1_dim1 <= m1_rpb ))) //single row block
{
return MMBinaryMethod.MR_BINARY_UAGG_CHAIN;
}

http://git-wip-us.apache.org/repos/asf/systemml/blob/4f29b348/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
--
diff --git a/src/main/java/org/apache/sysml/hops/OptimizerUtils.java 
b/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
index 5d831e5..d67e086 100644
--- a/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
+++ b/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
@@ -709,7 +709,7 @@ public class OptimizerUtils
//check for guaranteed 

[30/50] [abbrv] systemml git commit: [SYSTEMML-1969] Support single-precision operations on GPU backend

[reinwald]

[SYSTEMML-1969] Support single-precision operations on GPU backend

- Since single-precision operations are faster on most GPU, we should allow our 
users to perform the instructions on GPU in single precision.
- The GPU backend has been refactored to support arbitrary precision.
- This feature can be enabled via configuration property 
sysml.floating.point.precision.
- The valid values for this property are double and float. We can support 
half/mixed precision in a separate PR.

Closes #688.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/abbffc55
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/abbffc55
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/abbffc55

Branch: refs/heads/master
Commit: abbffc55ef8f47f10b6e59b0ae5e1f311f4a8f3e
Parents: 881caa9
Author: Niketan Pansare 
Authored: Wed Oct 25 19:25:20 2017 -0700
Committer: Niketan Pansare 
Committed: Wed Oct 25 19:26:50 2017 -0700

--
 conf/SystemML-config.xml.template   |3 +
 src/main/cpp/kernels/SystemML.cu| 1874 ++--
 src/main/cpp/kernels/SystemML.ptx   | 9579 ++
 .../java/org/apache/sysml/api/DMLScript.java|1 +
 .../apache/sysml/api/ScriptExecutorUtils.java   |4 +
 .../java/org/apache/sysml/conf/DMLConfig.java   |4 +-
 .../controlprogram/caching/CacheableData.java   |4 +-
 .../instructions/gpu/context/CSRPointer.java|   52 +-
 .../instructions/gpu/context/GPUContext.java|   31 +-
 .../instructions/gpu/context/GPUObject.java |   91 +-
 .../instructions/gpu/context/JCudaKernels.java  |9 +-
 .../matrix/data/CudaSupportFunctions.java   |   87 +
 .../DoublePrecisionCudaSupportFunctions.java|  175 +
 .../runtime/matrix/data/LibMatrixCUDA.java  |  144 +-
 .../runtime/matrix/data/LibMatrixCuDNN.java |   38 +-
 .../LibMatrixCuDNNConvolutionAlgorithm.java |5 +-
 .../data/LibMatrixCuDNNInputRowFetcher.java |8 +-
 .../data/LibMatrixCuDNNPoolingDescriptors.java  |3 +-
 .../runtime/matrix/data/LibMatrixCuMatMult.java |   34 +-
 .../sysml/runtime/matrix/data/MatrixBlock.java  |5 +-
 .../SinglePrecisionCudaSupportFunctions.java|  208 +
 .../org/apache/sysml/test/gpu/GPUTests.java |   20 +-
 .../test/gpu/MatrixMultiplicationOpTest.java|   22 +-
 23 files changed, 9423 insertions(+), 2978 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/conf/SystemML-config.xml.template
--
diff --git a/conf/SystemML-config.xml.template 
b/conf/SystemML-config.xml.template
index 511e215..8452e75 100644
--- a/conf/SystemML-config.xml.template
+++ b/conf/SystemML-config.xml.template
@@ -93,6 +93,9 @@
 
 false

+
+double
+

30
 

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/cpp/kernels/SystemML.cu
--
diff --git a/src/main/cpp/kernels/SystemML.cu b/src/main/cpp/kernels/SystemML.cu
index c243564..d176f8f 100644
--- a/src/main/cpp/kernels/SystemML.cu
+++ b/src/main/cpp/kernels/SystemML.cu
@@ -26,11 +26,28 @@ nvcc -ptx -arch=sm_30 SystemML.cu
 #include 
 #include 
 
+extern "C" __global__ void double2float_f(double *A, float *ret, int N) {
+  int tid = blockIdx.x * blockDim.x + threadIdx.x;
+  if (tid < N) {
+   // TODO: Use __double2float_rd or __double2float_rn  or 
__double2float_ru or __double2float_rz after 
+ret[tid] = (float)A[tid];
+  }
+}
+
+extern "C" __global__ void float2double_f(float *A, double *ret, int N) {
+  int tid = blockIdx.x * blockDim.x + threadIdx.x;
+  if (tid < N) {
+ret[tid] = (double)A[tid];
+  }
+}
+
 /**
- * Performs a slice operation where the input matrix is sparse and the output 
matrix is dense.
- * This function avoids unnecessary sparse to dense conversion of the input 
matrix.
+ * Performs a slice operation where the input matrix is sparse and the output
+ * matrix is dense.
+ * This function avoids unnecessary sparse to dense conversion of the input
+ * matrix.
  * Parallelization: rows of output matrix.
- * 
+ *
  * @params inVal input val pointer
  * @params inRowPtr input row pointer
  * @params colInd input col index pointer
@@ -41,49 +58,73 @@ nvcc -ptx -arch=sm_30 SystemML.cu
  * @param cu column upper
  * @param retClen number of columns of output matrix
  */
-extern "C"
-__global__ void slice_sparse_dense_row(double* inVal, int* inRowPtr, int* 
colInd, double* ret, 
-int rl, int ru, int cl, int cu, int retClen) {
-   int index = blockIdx.x * blockDim.x + threadIdx.x;
-   int rowIndex = index + rl;
-   if (rowIndex <= ru){
-   /*
-* TODO: Alternative 

[42/50] [abbrv] systemml git commit: [SYSTEMML-540] Added a rewrite to support a common tensor operation (sum over channels)

[reinwald]

[SYSTEMML-540] Added a rewrite to support a common tensor operation (sum over 
channels)

- Added a rewrite to convert out = rowSums(matrix(colSums(A), rows=C, cols=HW)) 
to out = channel_sums(A) when nrow(A) > 1 and exectype is CP or GPU.
- This avoids unnecessary intermediates and GPU-CP-GPU transfer (for
  reshape). This saves about ~150 seconds on sentence CNN for 200 epochs.
- When we move to a higher CuDNN version, we can replace the custom 
channel_sums kernel with possibly more optimized CuDNN reduce tensor kernel.
- Added the corresponding CPU and GPU tests.
- Updated T_MAX(val) to MAX(). Interestingly enough, nvcc was smart enough
  to remove the parameter automatically, hence the ptx remained the same
  after the change.

Closes #693.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/d916ba5b
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/d916ba5b
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/d916ba5b

Branch: refs/heads/master
Commit: d916ba5bd8ceec591a04f4d16c6d24f3985e3e4f
Parents: 2896f33
Author: Niketan Pansare 
Authored: Mon Oct 30 10:32:53 2017 -0700
Committer: Niketan Pansare 
Committed: Mon Oct 30 10:32:53 2017 -0700

--
 src/main/cpp/kernels/Makefile   |   2 +-
 src/main/cpp/kernels/SystemML.cu| 308 ++-
 .../java/org/apache/sysml/hops/AggUnaryOp.java  |  95 --
 .../apache/sysml/lops/ConvolutionTransform.java |  42 ++-
 .../instructions/CPInstructionParser.java   |   1 +
 .../instructions/GPUInstructionParser.java  |   1 +
 .../cp/ConvolutionCPInstruction.java|  86 ++
 .../gpu/ConvolutionGPUInstruction.java  |  47 +++
 .../spark/QuantilePickSPInstruction.java|   2 +-
 .../runtime/matrix/data/LibMatrixCUDA.java  |  31 ++
 .../runtime/matrix/data/LibMatrixCuDNN.java |   4 +-
 .../sysml/test/gpu/AggregateUnaryOpTests.java   |  31 ++
 .../apache/sysml/test/gpu/UnaryOpTestsBase.java |   8 +-
 .../functions/tensor/ChannelSumTest.java| 146 +
 .../scripts/functions/tensor/ChannelSumTest.R   |  39 +++
 .../scripts/functions/tensor/ChannelSumTest.dml |  35 +++
 16 files changed, 690 insertions(+), 188 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/d916ba5b/src/main/cpp/kernels/Makefile
--
diff --git a/src/main/cpp/kernels/Makefile b/src/main/cpp/kernels/Makefile
index 5feae69..ec10317 100644
--- a/src/main/cpp/kernels/Makefile
+++ b/src/main/cpp/kernels/Makefile
@@ -16,7 +16,7 @@
 # under the License.
 
 NVCC=nvcc
-CUDAFLAGS= -ptx -c -arch=sm_30 
+CUDAFLAGS= -ptx -c -arch=sm_30 --std c++11
 
 # Use these flags for precise math
 #CUDAFLAGS= -ptx -c -arch=sm_30 -ftz=false -prec-div=true -prec-sqrt=true

http://git-wip-us.apache.org/repos/asf/systemml/blob/d916ba5b/src/main/cpp/kernels/SystemML.cu
--
diff --git a/src/main/cpp/kernels/SystemML.cu b/src/main/cpp/kernels/SystemML.cu
index d176f8f..ade2dd1 100644
--- a/src/main/cpp/kernels/SystemML.cu
+++ b/src/main/cpp/kernels/SystemML.cu
@@ -20,7 +20,7 @@
 /**
 When updating a kernel or adding a new one,
 please compile the ptx file and commit it:
-nvcc -ptx -arch=sm_30 SystemML.cu
+nvcc -ptx -arch=sm_30 --std c++11 SystemML.cu
 ***/
 
 #include 
@@ -29,7 +29,8 @@ nvcc -ptx -arch=sm_30 SystemML.cu
 extern "C" __global__ void double2float_f(double *A, float *ret, int N) {
   int tid = blockIdx.x * blockDim.x + threadIdx.x;
   if (tid < N) {
-   // TODO: Use __double2float_rd or __double2float_rn  or 
__double2float_ru or __double2float_rz after 
+// TODO: Use __double2float_rd or __double2float_rn  or __double2float_ru 
or
+// __double2float_rz after
 ret[tid] = (float)A[tid];
   }
 }
@@ -84,15 +85,14 @@ __device__ void slice_sparse_dense_row(T *inVal, int 
*inRowPtr, int *colInd,
  *
  * int size = inRowPtr[rowIndex+1] - inRowPtr[rowIndex];
  * double numThreads = (double)min(size, MAX_NUM_THREADS_CHILD_KERNEL);
- * slice_sparse_dense_row_helper<<< ceil(numThreads/
-*MAX_NUM_THREADS_CHILD_KERNEL), MAX_NUM_THREADS_CHILD_KERNEL>>>(inVal, 
inRowPtr,
-*colInd, ret,
-*  rl, ru, cl, cu, retClen, inRowPtr[rowIndex],
-*inRowPtr[rowIndex+1], index);
-*
-* Two-step compilation and linking process in JCudaKernels's constructor:
-* cuLinkAddFile(linkState, CUjitInputType.CU_JIT_INPUT_LIBRARY,
-*"/usr/local/cuda/lib64/libcudadevrt.a", jitOptions);
+ * slice_sparse_dense_row_helper
+ * <<< ceil(numThreads/MAX_NUM_THREADS_CHILD_KERNEL), 
MAX_NUM_THREADS_CHILD_KERNEL>>>
+ * (inVal, inRowPtr, 

[31/50] [abbrv] systemml git commit: [SYSTEMML-446] Bugfix for GPU sparse right indexing with empty output

[reinwald]

[SYSTEMML-446] Bugfix for GPU sparse right indexing with empty output


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/d3917eff
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/d3917eff
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/d3917eff

Branch: refs/heads/master
Commit: d3917effd988de0e0977a310c73c4f232214632e
Parents: abbffc5
Author: Niketan Pansare 
Authored: Wed Oct 25 19:57:28 2017 -0700
Committer: Niketan Pansare 
Committed: Wed Oct 25 19:57:28 2017 -0700

--
 .../gpu/context/ExecutionConfig.java| 29 ++--
 .../runtime/matrix/data/LibMatrixCUDA.java  |  8 --
 2 files changed, 7 insertions(+), 30 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/d3917eff/src/main/java/org/apache/sysml/runtime/instructions/gpu/context/ExecutionConfig.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/instructions/gpu/context/ExecutionConfig.java
 
b/src/main/java/org/apache/sysml/runtime/instructions/gpu/context/ExecutionConfig.java
index 7f8eb9e..cae0660 100644
--- 
a/src/main/java/org/apache/sysml/runtime/instructions/gpu/context/ExecutionConfig.java
+++ 
b/src/main/java/org/apache/sysml/runtime/instructions/gpu/context/ExecutionConfig.java
@@ -69,6 +69,8 @@ public class ExecutionConfig {
 * @throws DMLRuntimeException if DMLRuntimeException occurs
 */
public static ExecutionConfig getConfigForSimpleVectorOperations(int 
numCells) throws DMLRuntimeException {
+   if(numCells == 0)
+   throw new DMLRuntimeException("Attempting to invoke a 
kernel with 0 threads");
int deviceNumber = 0;
int blockDimX = getMaxBlockDim(deviceNumber);
int gridDimX = (int) Math.ceil((double) numCells / blockDimX);
@@ -76,32 +78,6 @@ public class ExecutionConfig {
}
 
/**
-* Use this for simple matrix operations and use following in the kernel
-* 
-* int ix = blockIdx.x * blockDim.x + threadIdx.x;
-* int iy = blockIdx.y * blockDim.y + threadIdx.y;
-* 
-* 
-* This tries to schedule as minimum grids as possible.
-*
-* @param rlen number of rows
-* @param clen number of columns
-* @return execution configuration
-* @throws DMLRuntimeException if DMLRuntimeException occurs
-*/
-   public static ExecutionConfig getConfigForMatrixOperations(int rlen, 
int clen) throws DMLRuntimeException {
-   int deviceNumber = 0;
-   int maxBlockDim = getMaxBlockDim(deviceNumber);
-   int blockDimX = (int) Math.min(maxBlockDim, rlen);
-   int gridDimX = (int) Math.ceil((double) rlen / blockDimX);
-   int blockDimY = (int) Math.min(Math.floor(((double) 
maxBlockDim) / blockDimX), clen);
-   int gridDimY = (int) Math.ceil((double) clen / blockDimY);
-   if (gridDimY > 65535)
-   throw new DMLRuntimeException("Internal Error: gridDimY 
must be less than 65535 for all supported CUDA compute capabilites!");
-   return new ExecutionConfig(gridDimX, gridDimY, blockDimX, 
blockDimY);
-   }
-
-   /**
 * Use this for simple vector operations and use following in the kernel
 * 
 * int index = blockIdx.x * blockDim.x + threadIdx.x
@@ -116,7 +92,6 @@ public class ExecutionConfig {
return getConfigForSimpleVectorOperations(rlen * clen);
}
 
-
public ExecutionConfig(int gridDimX, int blockDimX) {
this.gridDimX = gridDimX;
this.blockDimX = blockDimX;

http://git-wip-us.apache.org/repos/asf/systemml/blob/d3917eff/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCUDA.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCUDA.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCUDA.java
index eb17e69..2cccde0 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCUDA.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCUDA.java
@@ -1821,17 +1821,19 @@ public class LibMatrixCUDA {
 */
protected static void sliceSparseDense(GPUContext gCtx, String 
instName, CSRPointer inPointer, Pointer outPointer, 
int rl, int ru, int cl, int cu, int inClen) throws 
DMLRuntimeException {
+   int size = getNnz(inPointer, rl, ru);
+   // Return since nnz of the output is 0 as outPointer is 
expected to be zeroed out.
+   

[24/50] [abbrv] systemml git commit: [HOTFIX][SYSTEMML-1648] Fix l2svm and msvm algorithm scripts

[reinwald]

[HOTFIX][SYSTEMML-1648] Fix l2svm and msvm algorithm scripts

This patch fixes the recently changed l2svm and msvm algorithm scripts
with regard to (1) use of non-existing variables, (2) corrupted
convergence checks (before update), and (3) various smaller issues
(unused variables, commented code, formatting).


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/70ab072a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/70ab072a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/70ab072a

Branch: refs/heads/master
Commit: 70ab072ae764a9abffaead3431ca11e8e1efec68
Parents: a472ae9
Author: Matthias Boehm 
Authored: Tue Oct 24 19:48:07 2017 -0700
Committer: Matthias Boehm 
Committed: Tue Oct 24 19:48:07 2017 -0700

--
 scripts/algorithms/l2-svm.dml | 18 --
 scripts/algorithms/m-svm.dml  | 18 +++---
 2 files changed, 15 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/70ab072a/scripts/algorithms/l2-svm.dml
--
diff --git a/scripts/algorithms/l2-svm.dml b/scripts/algorithms/l2-svm.dml
index 141ef82..2446610 100644
--- a/scripts/algorithms/l2-svm.dml
+++ b/scripts/algorithms/l2-svm.dml
@@ -134,31 +134,32 @@ while(continue & iter < maxiterations)  {
 h = dd + sum(Xd * sv * Xd)
 step_sz = step_sz - g/h
 
-continue1 = (gg/h >= 0.01);
+continue1 = (g*g/h >= 0.01);
   }
 
   #update weights
   w = w + step_sz*s
   Xw = Xw + step_sz*Xd
-   
+  
   out = 1 - Y * Xw
   sv = (out > 0)
   out = sv * out
   obj = 0.5 * sum(out * out) + lambda/2 * sum(w * w)
   g_new = t(X) %*% (out * Y) - lambda * w
-
+  
   print("ITER " + iter + ": OBJ=" + obj)
   debug_str = append(debug_str, iter + "," + obj)
-   
+  
   tmp = sum(s * g_old)
-  continue = (step_sz*tmp >= epsilon*obj & sum(s^2) != 0);
-
+  
   #non-linear CG step
   be = sum(g_new * g_new)/sum(g_old * g_old)
   s = be * s + g_new
   g_old = g_new
-
+  
+  continue = (step_sz*tmp >= epsilon*obj & sum(s^2) != 0);
   iter = iter + 1
+  
 }
 
 extra_model_params = matrix(0, rows=4, cols=1)
@@ -167,11 +168,8 @@ extra_model_params[2,1] = negative_label
 extra_model_params[3,1] = intercept
 extra_model_params[4,1] = dimensions
 
-weights = w
 w = t(cbind(t(w), t(extra_model_params)))
 write(w, $model, format=cmdLine_fmt)
-# write(extra_model_params, " ", format=cmdLine_fmt)
-# write(weights, " ", format=cmdLine_fmt)
 
 logFile = $Log
 if(logFile != " ") {

http://git-wip-us.apache.org/repos/asf/systemml/blob/70ab072a/scripts/algorithms/m-svm.dml
--
diff --git a/scripts/algorithms/m-svm.dml b/scripts/algorithms/m-svm.dml
index 6c11811..253764c 100644
--- a/scripts/algorithms/m-svm.dml
+++ b/scripts/algorithms/m-svm.dml
@@ -25,7 +25,7 @@
 # Example Usage:
 # Assume SVM_HOME is set to the home of the dml script
 # Assume input and output directories are on hdfs as INPUT_DIR and OUTPUT_DIR
-# Assume epsilon = 0.001, lambda=1.0, max_iterations = 100
+# Assume epsilon = 0.001, lambda=1.0, maxiterations = 100
 #
 # INPUT PARAMETERS:
 # 
-
@@ -92,8 +92,8 @@ lambda = cmdLine_reg
 if(lambda < 0)
   stop("Stopping due to invalid argument: Regularization constant (reg) must 
be non-negative")
 
-max_iterations = cmdLine_maxiter
-if(max_iterations < 1)
+maxiterations = cmdLine_maxiter
+if(maxiterations < 1)
   stop("Stopping due to invalid argument: Maximum iterations should be a 
positive integer")
 
 num_samples = nrow(X)
@@ -110,7 +110,7 @@ if(intercept == 1){
 }
 w = matrix(0, rows=num_rows_in_w, cols=num_classes)
 
-debug_mat = matrix(-1, rows=max_iterations, cols=num_classes)
+debug_mat = matrix(-1, rows=maxiterations, cols=num_classes)
 
 parfor(iter_class in 1:num_classes){ 
   Y_local = 2 * (Y == iter_class) - 1
@@ -145,7 +145,6 @@ parfor(iter_class in 1:num_classes){
   step_sz = step_sz - g/h
   
   continue1 = (g*g/h >= 0.01)
-  
 }
 
 #update weights
@@ -162,15 +161,14 @@ parfor(iter_class in 1:num_classes){
   
 train_acc = sum(Y_local*(X%*%w_class) >= 0)/num_samples*100
 print("For class " + iter_class + " iteration " + iter + " training 
accuracy: " + train_acc)
-debug_mat[iter+1,iter_class] = obj
+debug_mat[iter+1,iter_class] = obj

-continue = (step_sz*tmp >= epsilon*obj & sum(s^2) != 0);
-
 #non-linear CG step
 be = sum(g_new * g_new)/sum(g_old * g_old)
 s = be * s + g_new
 g_old = g_new
 
+continue = (step_sz*tmp >= epsilon*obj & sum(s^2) != 0);
 iter = iter + 1
  

[43/50] [abbrv] systemml git commit: [SYSTEMML-1979] Improved codegen optimizer (cost model, various fixes)

[reinwald]

[SYSTEMML-1979] Improved codegen optimizer (cost model, various fixes)

This patch makes a number of improvements to the codegen optimizer,
which help to exploit missed fusion potential for Kmeans over large
distributed datasets (i.e., with spark codegen operations). In detail,
this includes the following changes:

1) Eviction-aware cost model: So far we only took the write memory
bandwidth into account. With this change we also account for known
evictions whenever the output and temporary intermediate inputs are
known not to fit into the buffer pool. 

2) Generalized exploration of row fusion plans: This generalization now
allows to fuse matrix-matrix multiplications onto arbitrary row
operations, which allows to fuse the entire Kmeans inner loop if
beneficial.

3) Row sumSq vector primitives: Additionally, we now compile sumSq
vector primitives instead of sum(pow(,2)) which helps to avoid
unnecessary dense row vector intermediates. 

4) Fix missing dense-sparse outer vector operations: So far we only
supported sparse-dense outer vector operations. With the above change
(2), the sparse input can also occur on the right hand side.

5) Fix cost model for unary aggregates: The compute costs for all types
of unary aggregates were incorrectly computed based on the output size
instead of based on the input size.

6) Fix row to cell conversion: This patch also makes some
smaller corrections for the conversion of row to cell templates if there
are no aggregations or vector operations (e.g., only convert if a cell
template exist with exactly the same fusion references).

7) Fix robustness of temporary memory management: We use a preallocated
ring buffer for row vector intermediates of different sizes. This patch
restricts the size of preallocated vectors to 4MB to avoid OOM in case
the large vectors are not used by an operator.

On a 200M x 100, dense input matrix (160GB), this patch improved the
end-to-end runtime of Kmeans (20 iterations, 5 centroids) w/ codegen
from 5319s to 281s.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/d907efc1
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/d907efc1
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/d907efc1

Branch: refs/heads/master
Commit: d907efc17456d7536e1a7344a614aa8a122721ee
Parents: d916ba5
Author: Matthias Boehm 
Authored: Sun Oct 29 23:21:30 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 30 18:37:34 2017 -0700

--
 .../sysml/hops/codegen/cplan/CNodeBinary.java   |  3 +-
 .../sysml/hops/codegen/cplan/CNodeUnary.java| 11 +--
 .../opt/PlanSelectionFuseCostBasedV2.java   | 72 +---
 .../hops/codegen/template/CPlanMemoTable.java   | 10 +++
 .../hops/codegen/template/TemplateRow.java  | 47 ++---
 .../runtime/codegen/LibSpoofPrimitives.java | 27 
 .../gpu/ConvolutionGPUInstruction.java  |  5 --
 .../functions/codegen/RowAggTmplTest.java   | 18 -
 .../scripts/functions/codegen/rowAggPattern33.R | 36 ++
 .../functions/codegen/rowAggPattern33.dml   | 33 +
 10 files changed, 215 insertions(+), 47 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/d907efc1/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java 
b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
index d188afd..8c3c73d 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
@@ -77,7 +77,8 @@ public class CNodeBinary extends CNode
return sparseLhs ? "double[] %TMP% 
= LibSpoofPrimitives.vectMatrixMult(%IN1v%, %IN2%, %IN1i%, %POS1%, %POS2%, 
alen, len);\n" :
"
double[] %TMP% = LibSpoofPrimitives.vectMatrixMult(%IN1%, %IN2%, %POS1%, 
%POS2%, %LEN%);\n";
case VECT_OUTERMULT_ADD:
-   return sparseLhs ? "
LibSpoofPrimitives.vectOuterMultAdd(%IN1v%, %IN2%, %OUT%, %IN1i%, %POS1%, 
%POS2%, %POSOUT%, alen, %LEN1%, %LEN2%);\n" :
+   return  sparseLhs ? "
LibSpoofPrimitives.vectOuterMultAdd(%IN1v%, %IN2%, %OUT%, %IN1i%, %POS1%, 
%POS2%, %POSOUT%, alen, %LEN1%, %LEN2%);\n" :
+   sparseRhs ? "
LibSpoofPrimitives.vectOuterMultAdd(%IN1%, %IN2v%, %OUT%, %POS1%, %IN2i%, 
%POS2%, %POSOUT%, alen, %LEN1%, %LEN2%);\n" :
" 

[29/50] [abbrv] systemml git commit: [SYSTEMML-1969] Support single-precision operations on GPU backend

[reinwald]

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/cpp/kernels/SystemML.ptx
--
diff --git a/src/main/cpp/kernels/SystemML.ptx 
b/src/main/cpp/kernels/SystemML.ptx
index 73b057e..d382fc5 100644
--- a/src/main/cpp/kernels/SystemML.ptx
+++ b/src/main/cpp/kernels/SystemML.ptx
@@ -1,8 +1,8 @@
 //
 // Generated by NVIDIA NVVM Compiler
 //
-// Compiler Build ID: CL-21124049
-// Cuda compilation tools, release 8.0, V8.0.44
+// Compiler Build ID: CL-21554848
+// Cuda compilation tools, release 8.0, V8.0.61
 // Based on LLVM 3.4svn
 //
 
@@ -10,7 +10,7 @@
 .target sm_30
 .address_size 64
 
-   // .globl   slice_sparse_dense_row
+   // .globl   double2float_f
 .func  (.param .b64 func_retval0) __internal_trig_reduction_slowpathd
 (
.param .b64 __internal_trig_reduction_slowpathd_param_0,
@@ -23,20 +23,97 @@
.param .b64 __internal_accurate_pow_param_1
 )
 ;
-.extern .shared .align 8 .b8 sdata[];
+.extern .shared .align 1 .b8 my_sdata[];
+.const .align 4 .b8 __cudart_i2opi_f[24] = {65, 144, 67, 60, 153, 149, 98, 
219, 192, 221, 52, 245, 209, 87, 39, 252, 41, 21, 68, 78, 110, 131, 249, 162};
 .const .align 8 .b8 __cudart_i2opi_d[144] = {8, 93, 141, 31, 177, 95, 251, 
107, 234, 146, 82, 138, 247, 57, 7, 61, 123, 241, 229, 235, 199, 186, 39, 117, 
45, 234, 95, 158, 102, 63, 70, 79, 183, 9, 203, 39, 207, 126, 54, 109, 31, 109, 
10, 90, 139, 17, 47, 239, 15, 152, 5, 222, 255, 151, 248, 31, 59, 40, 249, 189, 
139, 95, 132, 156, 244, 57, 83, 131, 57, 214, 145, 57, 65, 126, 95, 180, 38, 
112, 156, 233, 132, 68, 187, 46, 245, 53, 130, 232, 62, 167, 41, 177, 28, 235, 
29, 254, 28, 146, 209, 9, 234, 46, 73, 6, 224, 210, 77, 66, 58, 110, 36, 183, 
97, 197, 187, 222, 171, 99, 81, 254, 65, 144, 67, 60, 153, 149, 98, 219, 192, 
221, 52, 245, 209, 87, 39, 252, 41, 21, 68, 78, 110, 131, 249, 162};
 .const .align 8 .b8 __cudart_sin_cos_coeffs[128] = {186, 94, 120, 249, 101, 
219, 229, 61, 70, 210, 176, 44, 241, 229, 90, 190, 146, 227, 172, 105, 227, 29, 
199, 62, 161, 98, 219, 25, 160, 1, 42, 191, 24, 8, 17, 17, 17, 17, 129, 63, 84, 
85, 85, 85, 85, 85, 197, 191, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
100, 129, 253, 32, 131, 255, 168, 189, 40, 133, 239, 193, 167, 238, 33, 62, 
217, 230, 6, 142, 79, 126, 146, 190, 233, 188, 221, 25, 160, 1, 250, 62, 71, 
93, 193, 22, 108, 193, 86, 191, 81, 85, 85, 85, 85, 85, 165, 63, 0, 0, 0, 0, 0, 
0, 224, 191, 0, 0, 0, 0, 0, 0, 240, 63};
 
-.visible .entry slice_sparse_dense_row(
-   .param .u64 slice_sparse_dense_row_param_0,
-   .param .u64 slice_sparse_dense_row_param_1,
-   .param .u64 slice_sparse_dense_row_param_2,
-   .param .u64 slice_sparse_dense_row_param_3,
-   .param .u32 slice_sparse_dense_row_param_4,
-   .param .u32 slice_sparse_dense_row_param_5,
-   .param .u32 slice_sparse_dense_row_param_6,
-   .param .u32 slice_sparse_dense_row_param_7,
-   .param .u32 slice_sparse_dense_row_param_8
+.visible .entry double2float_f(
+   .param .u64 double2float_f_param_0,
+   .param .u64 double2float_f_param_1,
+   .param .u32 double2float_f_param_2
+)
+{
+   .reg .pred  %p<2>;
+   .reg .f32   %f<2>;
+   .reg .b32   %r<6>;
+   .reg .f64   %fd<2>;
+   .reg .b64   %rd<9>;
+
+
+   ld.param.u64%rd1, [double2float_f_param_0];
+   ld.param.u64%rd2, [double2float_f_param_1];
+   ld.param.u32%r2, [double2float_f_param_2];
+   mov.u32 %r3, %ctaid.x;
+   mov.u32 %r4, %ntid.x;
+   mov.u32 %r5, %tid.x;
+   mad.lo.s32  %r1, %r4, %r3, %r5;
+   setp.ge.s32 %p1, %r1, %r2;
+   @%p1 braBB0_2;
+
+   cvta.to.global.u64  %rd3, %rd1;
+   mul.wide.s32%rd4, %r1, 8;
+   add.s64 %rd5, %rd3, %rd4;
+   ld.global.f64   %fd1, [%rd5];
+   cvt.rn.f32.f64  %f1, %fd1;
+   cvta.to.global.u64  %rd6, %rd2;
+   mul.wide.s32%rd7, %r1, 4;
+   add.s64 %rd8, %rd6, %rd7;
+   st.global.f32   [%rd8], %f1;
+
+BB0_2:
+   ret;
+}
+
+   // .globl   float2double_f
+.visible .entry float2double_f(
+   .param .u64 float2double_f_param_0,
+   .param .u64 float2double_f_param_1,
+   .param .u32 float2double_f_param_2
+)
+{
+   .reg .pred  %p<2>;
+   .reg .f32   %f<2>;
+   .reg .b32   %r<6>;
+   .reg .f64   %fd<2>;
+   .reg .b64   %rd<9>;
+
+
+   ld.param.u64%rd1, [float2double_f_param_0];
+   ld.param.u64%rd2, [float2double_f_param_1];
+   ld.param.u32%r2, [float2double_f_param_2];
+   mov.u32 %r3, %ctaid.x;
+   mov.u32 %r4, %ntid.x;
+   mov.u32 %r5, %tid.x;
+   mad.lo.s32  %r1, %r4, %r3, %r5;
+   setp.ge.s32 %p1, %r1, %r2;
+   @%p1 braBB1_2;
+
+   cvta.to.global.u64  %rd3, %rd1;
+   mul.wide.s32%rd4, %r1, 4;
+   

[32/50] [abbrv] systemml git commit: [MINOR] Enable single precision GPU tests

[reinwald]

[MINOR] Enable single precision GPU tests

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/f0406746
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/f0406746
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/f0406746

Branch: refs/heads/master
Commit: f040674661ae818d0379abbcac624a726d3b3e3a
Parents: d3917ef
Author: Niketan Pansare 
Authored: Wed Oct 25 20:29:55 2017 -0700
Committer: Niketan Pansare 
Committed: Wed Oct 25 20:29:55 2017 -0700

--
 src/test/java/org/apache/sysml/test/gpu/GPUTests.java | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/f0406746/src/test/java/org/apache/sysml/test/gpu/GPUTests.java
--
diff --git a/src/test/java/org/apache/sysml/test/gpu/GPUTests.java 
b/src/test/java/org/apache/sysml/test/gpu/GPUTests.java
index d7d1ad5..a83b110 100644
--- a/src/test/java/org/apache/sysml/test/gpu/GPUTests.java
+++ b/src/test/java/org/apache/sysml/test/gpu/GPUTests.java
@@ -55,7 +55,7 @@ public abstract class GPUTests extends AutomatedTestBase {
private static final boolean PRINT_MAT_ERROR = false;

// We will use this flag until lower precision is supported on CP. 
-   private final static String DATA_TYPE = "double";  
+   private final static String FLOATING_POINT_PRECISION = "double";  
protected final double SINGLE_PRECISION_THRESHOLD = 1e-3;// for 
relative error


@@ -75,9 +75,9 @@ public abstract class GPUTests extends AutomatedTestBase {
 * @return a valid threshold
 */
protected double getTHRESHOLD() {
-   if(DATA_TYPE.equals("double"))  return 
DOUBLE_PRECISION_THRESHOLD;
-   else if(DATA_TYPE.equals("float"))  return 
SINGLE_PRECISION_THRESHOLD;
-   else throw new RuntimeException("Unsupported datatype:" + 
DATA_TYPE);
+   if(FLOATING_POINT_PRECISION.equals("double"))  return 
DOUBLE_PRECISION_THRESHOLD;
+   else if(FLOATING_POINT_PRECISION.equals("single"))  return 
SINGLE_PRECISION_THRESHOLD;
+   else throw new RuntimeException("Unsupported precision:" + 
FLOATING_POINT_PRECISION);
}
 
@After
@@ -263,7 +263,7 @@ public abstract class GPUTests extends AutomatedTestBase {
format.format(
"Relative 
error(%f) is more than threshold (%f). Expected = %f, Actual = %f, differed at 
[%d, %d]",
relativeError, 
getTHRESHOLD(), expectedDouble, actualDouble, i, j);
-   if(DATA_TYPE.equals("double"))
+   
if(FLOATING_POINT_PRECISION.equals("double"))

Assert.assertTrue(format.toString(), relativeError < getTHRESHOLD());
else

Assert.assertTrue(format.toString(), relativeError < getTHRESHOLD() || 
absoluteError < getTHRESHOLD());
@@ -324,7 +324,7 @@ public abstract class GPUTests extends AutomatedTestBase {
protected List runOnGPU(SparkSession spark, String scriptStr, 
Map inputs,
List outStrs) {
MLContext gpuMLC = new MLContext(spark);
-   gpuMLC.setConfigProperty("sysml.gpu.dataType", DATA_TYPE);
+   gpuMLC.setConfigProperty("sysml.floating.point.precision", 
FLOATING_POINT_PRECISION);
gpuMLC.setGPU(true);
gpuMLC.setForceGPU(true);
gpuMLC.setStatistics(true);

[23/50] [abbrv] systemml git commit: [SYSTEMML-1972] Fix rewrite remove right indexing (w/ invalid ix range)

[reinwald]

[SYSTEMML-1972] Fix rewrite remove right indexing (w/ invalid ix range)

This patch hardens the existing rewrite of removing unnecessary right
indexing operations whose input and output are of equal size, which is
only valid with valid indexing ranges. Although we check this during
validation, there are scenarios with unknown sizes or index expressions
that cause invalid results despite invalid index ranges. We now simply
check for valid row-lower and column-lower indexing ranges which both
needs to be 1 for the rewrite to be valid.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a472ae92
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a472ae92
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a472ae92

Branch: refs/heads/master
Commit: a472ae922827b437e00ca8331ff3db5f6c19f443
Parents: 2c37d9f
Author: Matthias Boehm 
Authored: Mon Oct 23 23:43:46 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 23 23:44:04 2017 -0700

--
 .../java/org/apache/sysml/hops/IndexingOp.java  |  5 +
 .../sysml/hops/rewrite/HopRewriteUtils.java | 13 +++
 .../RewriteAlgebraicSimplificationDynamic.java  | 23 ++--
 3 files changed, 21 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/a472ae92/src/main/java/org/apache/sysml/hops/IndexingOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/IndexingOp.java 
b/src/main/java/org/apache/sysml/hops/IndexingOp.java
index 23d0630..5989c66 100644
--- a/src/main/java/org/apache/sysml/hops/IndexingOp.java
+++ b/src/main/java/org/apache/sysml/hops/IndexingOp.java
@@ -118,10 +118,7 @@ public class IndexingOp extends Hop
Hop input = getInput().get(0);

//rewrite remove unnecessary right indexing
-   if( dimsKnown() && input.dimsKnown() 
-   && getDim1() == input.getDim1() && getDim2() == 
input.getDim2()
-   && !(getDim1()==1 && getDim2()==1))
-   {
+   if( HopRewriteUtils.isUnnecessaryRightIndexing(this) ) {
setLops( input.constructLops() );
}
//actual lop construction, incl operator selection 

http://git-wip-us.apache.org/repos/asf/systemml/blob/a472ae92/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
--
diff --git a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java 
b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
index 68068eb..ad2392a 100644
--- a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
+++ b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
@@ -1000,6 +1000,19 @@ public class HopRewriteUtils
&& hop.getInput().get(4) instanceof LiteralOp;
}

+   public static boolean isUnnecessaryRightIndexing(Hop hop) {
+   if( !(hop instanceof IndexingOp) )
+   return false;
+   //note: in addition to equal sizes, we also check a valid
+   //starting row and column ranges of 1 in order to guard against
+   //invalid modifications in the presence of invalid index ranges
+   //(e.g., X[,2] on a column vector needs to throw an error)
+   return isEqualSize(hop, hop.getInput().get(0))
+   && !(hop.getDim1()==1 && hop.getDim2()==1)
+   && isLiteralOfValue(hop.getInput().get(1), 1)  //rl
+   && isLiteralOfValue(hop.getInput().get(3), 1); //cl
+   }
+   
public static boolean isScalarMatrixBinaryMult( Hop hop ) {
return hop instanceof BinaryOp && 
((BinaryOp)hop).getOp()==OpOp2.MULT
&& 
((hop.getInput().get(0).getDataType()==DataType.SCALAR && 
hop.getInput().get(1).getDataType()==DataType.MATRIX)

http://git-wip-us.apache.org/repos/asf/systemml/blob/a472ae92/src/main/java/org/apache/sysml/hops/rewrite/RewriteAlgebraicSimplificationDynamic.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/rewrite/RewriteAlgebraicSimplificationDynamic.java
 
b/src/main/java/org/apache/sysml/hops/rewrite/RewriteAlgebraicSimplificationDynamic.java
index 5437535..eba06fc 100644
--- 
a/src/main/java/org/apache/sysml/hops/rewrite/RewriteAlgebraicSimplificationDynamic.java
+++ 
b/src/main/java/org/apache/sysml/hops/rewrite/RewriteAlgebraicSimplificationDynamic.java
@@ -230,23 +230,14 @@ public class RewriteAlgebraicSimplificationDynamic 
extends HopRewriteRule
 

[05/50] [abbrv] systemml git commit: [SYSTEMML-1836] Fix unnecessary GC overhead in codegen row operations

[reinwald]

[SYSTEMML-1836] Fix unnecessary GC overhead in codegen row operations

This patch fixes issues of huge garbage collection overhead in special
cases of codegen row operations with relatively large intermediate rows,
matrix side inputs, and especially for multi-threaded operations. The
major problem was that, for aggregations, the temporary output per
thread was allocated as the product of the two temporary vector sizes
(as necessary for matrix multiplication outputs), which however is often
completely unnecessary.

On a scenario of row operations for sum( (X != 0) * (UV - X) ^ 2) over
small inputs of 10K x 3K, this patch improved performance for 10
iterations from 80s (74s GC) to 1.4s (0.074s GC). 


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/586f8229
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/586f8229
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/586f8229

Branch: refs/heads/master
Commit: 586f82299292019583b68dd98ccb8fbb71a2fcd3
Parents: b6b6772
Author: Matthias Boehm 
Authored: Sun Oct 15 21:50:17 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 15 21:50:17 2017 -0700

--
 .../sysml/runtime/codegen/LibSpoofPrimitives.java | 11 +--
 .../apache/sysml/runtime/codegen/SpoofRowwise.java| 14 +++---
 2 files changed, 16 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/586f8229/src/main/java/org/apache/sysml/runtime/codegen/LibSpoofPrimitives.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/codegen/LibSpoofPrimitives.java 
b/src/main/java/org/apache/sysml/runtime/codegen/LibSpoofPrimitives.java
index 8444b5f..7624d96 100644
--- a/src/main/java/org/apache/sysml/runtime/codegen/LibSpoofPrimitives.java
+++ b/src/main/java/org/apache/sysml/runtime/codegen/LibSpoofPrimitives.java
@@ -1829,9 +1829,9 @@ public class LibSpoofPrimitives
private int _len2;

public VectorBuffer(int num, int len1, int len2) {
-   int lnum = (len2 > 0) ? 2*num : num;
+   int lnum = (len2>0 && len1!=len2) ? 2*num : num;
_data = new double[lnum][];
-   for( int i=0; i num ) {
_data[2*i] = new double[len1];
_data[2*i+1] = new double[len2];
@@ -1839,6 +1839,7 @@ public class LibSpoofPrimitives
else {
_data[i] = new double[len1];
}
+   }
_pos = -1;
_len1 = len1;
_len2 = len2;
@@ -1851,5 +1852,11 @@ public class LibSpoofPrimitives
} while( _data[_pos].length!=len );
return _data[_pos];
}
+   @SuppressWarnings("unused")
+   public boolean isReusable(int num, int len1, int len2) {
+   int lnum = (len2>0 && len1!=len2) ? 2*num : num;
+   return (_len1 == len1 && _len2 == len2
+   && _data.length == lnum);
+   }
}
 }

http://git-wip-us.apache.org/repos/asf/systemml/blob/586f8229/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
--
diff --git a/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java 
b/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
index d8a747b..9d5675b 100644
--- a/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
+++ b/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
@@ -209,8 +209,9 @@ public abstract class SpoofRowwise extends SpoofOperator
if( _type.isColumnAgg() || _type == RowType.FULL_AGG ) {
//execute tasks
ArrayList tasks = new 
ArrayList<>();
+   int outLen = out.getNumRows() * 
out.getNumColumns();
for( int i=0; i> taskret = 
pool.invokeAll(tasks); 
//aggregate partial results
int len = _type.isColumnAgg() ? 
out.getNumRows()*out.getNumColumns() : 1;
@@ -343,17 +344,16 @@ public abstract class SpoofRowwise extends SpoofOperator
private final MatrixBlock _a;
private final SideInput[] _b;
private final double[] _scalars;
-

[27/50] [abbrv] systemml git commit: [SYSTEMML-1969] Support single-precision operations on GPU backend

[reinwald]

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCuMatMult.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCuMatMult.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCuMatMult.java
index 21a2a35..d962027 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCuMatMult.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixCuMatMult.java
@@ -23,13 +23,6 @@ import static 
jcuda.jcusparse.cusparseOperation.CUSPARSE_OPERATION_TRANSPOSE;
 import static jcuda.runtime.JCuda.cudaMemcpy;
 import static jcuda.runtime.cudaMemcpyKind.cudaMemcpyHostToDevice;
 import jcuda.Pointer;
-import jcuda.Sizeof;
-import jcuda.jcublas.JCublas2;
-import jcuda.jcublas.cublasHandle;
-import jcuda.jcublas.cublasOperation;
-import jcuda.jcusparse.JCusparse;
-import jcuda.jcusparse.cusparseHandle;
-import jcuda.runtime.JCuda;
 
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
@@ -43,6 +36,11 @@ import 
org.apache.sysml.runtime.instructions.gpu.context.GPUContext;
 import org.apache.sysml.utils.GPUStatistics;
 import org.apache.sysml.utils.Statistics;
 
+import jcuda.jcusparse.cusparseHandle;
+import jcuda.jcublas.cublasHandle;
+import jcuda.jcublas.cublasOperation;
+import jcuda.runtime.JCuda;
+
 public class LibMatrixCuMatMult extends LibMatrixCUDA {
 
private static final Log LOG = 
LogFactory.getLog(LibMatrixCuMatMult.class.getName());
@@ -175,7 +173,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
 
// Step 3: Invoke the kernel
long t1 = GPUStatistics.DISPLAY_STATISTICS ? 
System.nanoTime() : 0;
-   JCusparse.cusparseDcsrgemm(getCusparseHandle(gCtx), 
transa, transb, params.m, params.n, params.k, A.descr,
+   
cudaSupportFunctions.cusparsecsrgemm(getCusparseHandle(gCtx), transa, transb, 
params.m, params.n, params.k, A.descr,
(int) A.nnz, A.val, A.rowPtr, A.colInd, 
B.descr, (int) B.nnz, B.val, B.rowPtr, B.colInd, C.descr,
C.val, C.rowPtr, C.colInd);
if (GPUStatistics.DISPLAY_STATISTICS)
@@ -239,7 +237,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
 * allocated in dense row-major format and A is sparse.
 * 
 * Other than input and output, this method requires additional memory =
-* outRLen * outCLen * Sizeof.DOUBLE
+* outRLen * outCLen * sizeOfDataType
 * 
 * @param gCtx
 *a valid {@link GPUContext}
@@ -276,7 +274,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
// t(C) = t(B) %*% t(A)
Pointer output = null;
if (outRLen != 1 && outCLen != 1) {
-   output = gCtx.allocate(outRLen * outCLen * 
Sizeof.DOUBLE);
+   output = gCtx.allocate(outRLen * outCLen * 
sizeOfDataType);
} else {
// no transpose required for vector output
output = C;
@@ -287,7 +285,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
if (outRLen != 1 && outCLen != 1) {
// Transpose: C = t(output)
long t0 = GPUStatistics.DISPLAY_STATISTICS ? 
System.nanoTime() : 0;
-   JCublas2.cublasDgeam(gCtx.getCublasHandle(), 
cublasOperation.CUBLAS_OP_T, cublasOperation.CUBLAS_OP_T,
+   cudaSupportFunctions.cublasgeam(gCtx.getCublasHandle(), 
cublasOperation.CUBLAS_OP_T, cublasOperation.CUBLAS_OP_T,
toInt(outCLen), toInt(outRLen), one(), 
output, toInt(outRLen), zero(), new Pointer(),
toInt(outRLen), C, toInt(outCLen));
if (!DMLScript.EAGER_CUDA_FREE)
@@ -331,7 +329,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
int m = toInt(param.rightNumRows);
int n = toInt(param.rightNumCols);
int transa = 
reverseCusparseOp(cusparseOp(param.isLeftTransposed));
-   JCusparse.cusparseDcsrmv(handle, transa, m, n, 
toInt(B.nnz), one(), B.descr, B.val, B.rowPtr, B.colInd, A,
+   cudaSupportFunctions.cusparsecsrmv(handle, transa, m, 
n, toInt(B.nnz), one(), B.descr, B.val, B.rowPtr, B.colInd, A,
zero(), C);
kernel = 
GPUInstruction.MISC_TIMER_SPARSE_MATRIX_DENSE_VECTOR_LIB;
} else {
@@ -342,7 +340,7 @@ public class LibMatrixCuMatMult extends LibMatrixCUDA {
int transa = 
reverseCusparseOp(cusparseOp(param.isLeftTransposed));

[04/50] [abbrv] systemml git commit: [SYSTEMML-1963] Fix missing codegen dense-sparse vector primitives

[reinwald]

[SYSTEMML-1963] Fix missing codegen dense-sparse vector primitives

This patch fixes issues that showed up with the fuse-all heuristic on
different scenarios of ALS-CG. In particular, this adds missing vector
primitives for dense-sparse vector operations and modifies the code
generator accordingly. Most of these operations are invariant to the
ordering of inputs and hence simply call the existing primitives with
permuted inputs.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/b6b67727
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/b6b67727
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/b6b67727

Branch: refs/heads/master
Commit: b6b67727b9ec271995520e47dc3044eccaed2b65
Parents: 06b4b9d
Author: Matthias Boehm 
Authored: Sun Oct 15 20:27:51 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 15 20:27:51 2017 -0700

--
 .../sysml/hops/codegen/cplan/CNodeBinary.java   | 33 -
 .../runtime/codegen/LibSpoofPrimitives.java | 70 
 2 files changed, 88 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/b6b67727/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java 
b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
index 1ca4aa6..cac8ab8 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeBinary.java
@@ -68,16 +68,16 @@ public class CNodeBinary extends CNode
return ssComm || vsComm || vvComm;
}

-   public String getTemplate(boolean sparse, boolean scalarVector, 
boolean scalarInput) {
+   public String getTemplate(boolean sparseLhs, boolean sparseRhs, 
boolean scalarVector, boolean scalarInput) {
switch (this) {
case DOT_PRODUCT:   
-   return sparse ? "double %TMP% = 
LibSpoofPrimitives.dotProduct(%IN1v%, %IN2%, %IN1i%, %POS1%, %POS2%, alen);\n" :
+   return sparseLhs ? "double %TMP% = 
LibSpoofPrimitives.dotProduct(%IN1v%, %IN2%, %IN1i%, %POS1%, %POS2%, alen);\n" :
"
double %TMP% = LibSpoofPrimitives.dotProduct(%IN1%, %IN2%, %POS1%, %POS2%, 
%LEN%);\n";
case VECT_MATRIXMULT:   
-   return sparse ? "double[] %TMP% = 
LibSpoofPrimitives.vectMatrixMult(%IN1v%, %IN2%, %IN1i%, %POS1%, %POS2%, alen, 
len);\n" :
+   return sparseLhs ? "double[] %TMP% 
= LibSpoofPrimitives.vectMatrixMult(%IN1v%, %IN2%, %IN1i%, %POS1%, %POS2%, 
alen, len);\n" :
"
double[] %TMP% = LibSpoofPrimitives.vectMatrixMult(%IN1%, %IN2%, %POS1%, 
%POS2%, %LEN%);\n";
case VECT_OUTERMULT_ADD:   
-   return sparse ? "
LibSpoofPrimitives.vectOuterMultAdd(%IN1v%, %IN2%, %OUT%, %IN1i%, %POS1%, 
%POS2%, %POSOUT%, alen, %LEN1%, %LEN2%);\n" :
+   return sparseLhs ? "
LibSpoofPrimitives.vectOuterMultAdd(%IN1v%, %IN2%, %OUT%, %IN1i%, %POS1%, 
%POS2%, %POSOUT%, alen, %LEN1%, %LEN2%);\n" :
"
LibSpoofPrimitives.vectOuterMultAdd(%IN1%, %IN2%, %OUT%, %POS1%, %POS2%, 
%POSOUT%, %LEN1%, %LEN2%);\n";

//vector-scalar-add operations
@@ -96,10 +96,10 @@ public class CNodeBinary extends CNode
case VECT_GREATEREQUAL_ADD: {
String vectName = 
getVectorPrimitiveName();
if( scalarVector )
-   return sparse ? "
LibSpoofPrimitives.vect"+vectName+"Add(%IN1%, %IN2v%, %OUT%, %IN2i%, %POS2%, 
%POSOUT%, alen, %LEN%);\n" : 
+   return sparseLhs ? "
LibSpoofPrimitives.vect"+vectName+"Add(%IN1%, %IN2v%, %OUT%, %IN2i%, %POS2%, 
%POSOUT%, alen, %LEN%);\n" : 

"LibSpoofPrimitives.vect"+vectName+"Add(%IN1%, %IN2%, %OUT%, %POS2%, 
%POSOUT%, %LEN%);\n";
else
-   return sparse ? "

[34/50] [abbrv] systemml git commit: [SYSTEMML-1970] Performance conv2d backward filter (dense/sparse-sparse)

[reinwald]

[SYSTEMML-1970] Performance conv2d backward filter (dense/sparse-sparse)

This patch makes a number of smaller performance improvements to the
existing conv2d backward filter function. 

1) Dense/sparse-sparse: Whenever the rhs is sparse and the lhs has a
higher sparsity in terms of nnz/cells, we now flip the computation to
t(t(rhs)%*%t(lhs)), where the t(rhs) and t(lhs) are pckibacked into the
im2col and rotate calls, and the final transpose collapses with the
existing transpose add to a simple add.

2) Avoid unnecessary allocations in sparse-dense and sparse-sparse
matrix multiplications (limit temporary arrays to min(blocksize,ru-rl)).

3) Avoid class locking: So far the transAdd used class-wide
synchronization which causes unnecessary contention in scenarios where
multiple builtin functions are ran concurrently (e.g., parfor, jmlc). We
now simply synchronize over the monitor of the allocated output.

On an end-to-end cnn application w/ native blas enabled, this patch
improved performance from 349s to 335s.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/dd513ffe
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/dd513ffe
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/dd513ffe

Branch: refs/heads/master
Commit: dd513ffee87a4efaf1d5f771a8d0ee4bae67
Parents: 591a0f7
Author: Matthias Boehm 
Authored: Thu Oct 26 20:26:14 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Oct 26 23:38:19 2017 -0700

--
 .../LibMatrixDNNConv2dBackwardFilterHelper.java | 90 +---
 .../runtime/matrix/data/LibMatrixDNNHelper.java |  2 +
 .../matrix/data/LibMatrixDNNIm2ColHelper.java   | 27 +++---
 .../runtime/matrix/data/LibMatrixMult.java  |  4 +-
 4 files changed, 95 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/dd513ffe/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
index f0fd002..9698725 100644
--- 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
+++ 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
@@ -124,21 +124,83 @@ public class LibMatrixDNNConv2dBackwardFilterHelper {
}
}

-   private static synchronized void inplaceTransAdd(double[] a, 
ConvolutionParameters params) {
-   // Perform transposed addition: output of size [K, CRS] += 
input of size [CRS,K]
-   double [] c = params.output.denseBlock;
-   final int CRS = params.C*params.R*params.S, K = params.K;
-   final int blocksizeIJ = 128; //L2 cache
+   public static class Conv2dBackwardFilterTrans implements Callable 
{
+   private final int _rl, _ru; 
+   private final ConvolutionParameters _params; 
+   
+   public Conv2dBackwardFilterTrans(int rl, int ru, 
ConvolutionParameters params) {
+   _rl = rl; _ru = ru;
+   _params = params;
+   }

-   //cache-conscious blocked execution
-   for( int bi=0; bi

[48/50] [abbrv] systemml git commit: [SYSTEMML-1982] Improved IPA pass for recompile_once functions

[reinwald]

[SYSTEMML-1982] Improved IPA pass for recompile_once functions

This patch improved the existing inter-procedural-analysis pass for
recompile_once functions that are recompiled on function entry. So far
all functions with loops were marked for recompile_once. The problem is
that recompilation information is not available during IPA. Hence, we
correct these flags now after generating the runtime program and making
the decisions on recompilation.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/ee6060bf
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/ee6060bf
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/ee6060bf

Branch: refs/heads/master
Commit: ee6060bfc1576cf777dfa99c48126a5a7b35db3a
Parents: a2f0598
Author: Matthias Boehm 
Authored: Wed Nov 1 22:51:46 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Nov 2 00:39:17 2017 -0700

--
 .../org/apache/sysml/parser/DMLTranslator.java  | 57 +---
 .../apache/sysml/parser/ForStatementBlock.java  | 15 ++
 .../apache/sysml/parser/IfStatementBlock.java   | 12 ++---
 .../org/apache/sysml/parser/StatementBlock.java |  7 +--
 .../sysml/parser/WhileStatementBlock.java   | 12 ++---
 5 files changed, 45 insertions(+), 58 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/ee6060bf/src/main/java/org/apache/sysml/parser/DMLTranslator.java
--
diff --git a/src/main/java/org/apache/sysml/parser/DMLTranslator.java 
b/src/main/java/org/apache/sysml/parser/DMLTranslator.java
index 75103d1..fb8404f 100644
--- a/src/main/java/org/apache/sysml/parser/DMLTranslator.java
+++ b/src/main/java/org/apache/sysml/parser/DMLTranslator.java
@@ -317,25 +317,21 @@ public class DMLTranslator
}

public void constructLops(DMLProgram dmlp) throws ParseException, 
LanguageException, HopsException, LopsException {
-
// for each namespace, handle function program blocks handle 
function 
-   for (String namespaceKey : dmlp.getNamespaces().keySet()){
-   for (String fname: 
dmlp.getFunctionStatementBlocks(namespaceKey).keySet()) {
-   FunctionStatementBlock current = 
dmlp.getFunctionStatementBlock(namespaceKey, fname);
-   constructLops(current);
-   }
-   }
+   for( String namespaceKey : dmlp.getNamespaces().keySet() )
+   for( FunctionStatementBlock fsb : 
dmlp.getFunctionStatementBlocks(namespaceKey).values() )
+   constructLops(fsb);

// handle regular program blocks
-   for (int i = 0; i < dmlp.getNumStatementBlocks(); i++) {
-   StatementBlock current = dmlp.getStatementBlock(i);
-   constructLops(current);
-   }
+   for( StatementBlock sb : dmlp.getStatementBlocks() )
+   constructLops(sb);
}
 
-   public void constructLops(StatementBlock sb) 
+   public boolean constructLops(StatementBlock sb) 
throws HopsException, LopsException 
-   {   
+   {
+   boolean ret = false;
+   
if (sb instanceof WhileStatementBlock)
{
WhileStatementBlock wsb = (WhileStatementBlock)sb;
@@ -348,13 +344,13 @@ public class DMLTranslator
}
// step through stmt blocks in while stmt body
for (StatementBlock stmtBlock : body){
-   constructLops(stmtBlock);
+   ret |= constructLops(stmtBlock);
}

// handle while stmt predicate
Lop l = wsb.getPredicateHops().constructLops();
wsb.set_predicateLops(l);   
-   wsb.updatePredicateRecompilationFlag();
+   ret |= wsb.updatePredicateRecompilationFlag();
}

else if (sb instanceof IfStatementBlock)
@@ -370,16 +366,16 @@ public class DMLTranslator
}
// step through stmt blocks in if stmt ifBody
for (StatementBlock stmtBlock : ifBody)
-   constructLops(stmtBlock);
+   ret |= constructLops(stmtBlock);

// step through stmt blocks in if stmt elseBody
for (StatementBlock stmtBlock : elseBody)
-  

[08/50] [abbrv] systemml git commit: [SYSTEMML-1903] Fix codegen row candidate exploration w/ row indexing

[reinwald]

[SYSTEMML-1903] Fix codegen row candidate exploration w/ row indexing

For Autoencoder w/ batch=512, the codegen fuse-all heuristic was failing
on cplan construction due to unknown row type. The root cause was an
invalid partial fusion plan of type row that covered the row batching
from the overall dataset although row template only support column range
indexing (which applies to all rows).
 

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/3a7f38e5
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/3a7f38e5
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/3a7f38e5

Branch: refs/heads/master
Commit: 3a7f38e58586cee2044c611ef5b3222fcf9561e5
Parents: 5adb330
Author: Matthias Boehm 
Authored: Mon Oct 16 21:45:54 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 16 22:16:36 2017 -0700

--
 .../org/apache/sysml/hops/codegen/template/CPlanMemoTable.java | 2 +-
 src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/3a7f38e5/src/main/java/org/apache/sysml/hops/codegen/template/CPlanMemoTable.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/CPlanMemoTable.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/CPlanMemoTable.java
index 882cde2..99ffc8d 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/CPlanMemoTable.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/CPlanMemoTable.java
@@ -439,7 +439,7 @@ public class CPlanMemoTable
sb.append(input(i));
}
if( !isValid() )
-   sb.append(", x");
+   sb.append("|x");
sb.append(")");
return sb.toString();
}

http://git-wip-us.apache.org/repos/asf/systemml/blob/3a7f38e5/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
--
diff --git a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java 
b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
index b0f46b7..7bbfa52 100644
--- a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
+++ b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
@@ -988,7 +988,7 @@ public class HopRewriteUtils

public static boolean isColumnRangeIndexing(IndexingOp hop) {
return ((isLiteralOfValue(hop.getInput().get(1), 1)
-   && isLiteralOfValue(hop.getInput().get(2), 
hop.getDim1()))
+   && isLiteralOfValue(hop.getInput().get(2), 
hop.getInput().get(0).getDim1()))
|| hop.getDim1() == hop.getInput().get(0).getDim1())
&& isLiteralOfValue(hop.getInput().get(3), 1)
&& hop.getInput().get(4) instanceof LiteralOp;

[21/50] [abbrv] systemml git commit: [SYSTEMML-1648] Making SVM scripts work with MLContext

[reinwald]

[SYSTEMML-1648] Making SVM scripts work with MLContext

Closes #687.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/596005a8
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/596005a8
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/596005a8

Branch: refs/heads/master
Commit: 596005a80d0b39fef9b33b55145ffda043a4573d
Parents: a51f8e8
Author: j143 
Authored: Sun Oct 22 21:35:08 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 21:35:08 2017 -0700

--
 scripts/algorithms/l2-svm-predict.dml |  82 ++---
 scripts/algorithms/l2-svm.dml | 118 +-
 scripts/algorithms/m-svm-predict.dml  |  45 +++
 scripts/algorithms/m-svm.dml  | 186 -
 4 files changed, 222 insertions(+), 209 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/596005a8/scripts/algorithms/l2-svm-predict.dml
--
diff --git a/scripts/algorithms/l2-svm-predict.dml 
b/scripts/algorithms/l2-svm-predict.dml
index 9052265..73e28b4 100644
--- a/scripts/algorithms/l2-svm-predict.dml
+++ b/scripts/algorithms/l2-svm-predict.dml
@@ -51,6 +51,7 @@ cmdLine_Y = ifdef($Y, " ")
 cmdLine_confusion = ifdef($confusion, " ")
 cmdLine_accuracy = ifdef($accuracy, " ")
 cmdLine_scores = ifdef($scores, " ")
+cmdLine_scoring_only = ifdef($scoring_only, FALSE)
 cmdLine_fmt = ifdef($fmt, "text")
 
 X = read($X)
@@ -59,7 +60,7 @@ w = read($model)
 
 dimensions = as.scalar(w[nrow(w),1])
 if(dimensions != ncol(X))
-   stop("Stopping due to invalid input: Model dimensions do not seem to 
match input data dimensions")
+  stop("Stopping due to invalid input: Model dimensions do not seem to match 
input data dimensions")

 intercept = as.scalar(w[nrow(w)-1,1])
 negative_label = as.scalar(w[nrow(w)-2,1])
@@ -68,56 +69,51 @@ w = w[1:(nrow(w)-4),]
 
 b = 0.0
 if(intercept == 1)
-   b = as.scalar(w[nrow(w),1])
+  b = as.scalar(w[nrow(w),1])
 
 scores = b + (X %*% w[1:ncol(X),])
 
 if(cmdLine_scores != " ")
-   write(scores, cmdLine_scores, format=cmdLine_fmt)
+  write(scores, cmdLine_scores, format=cmdLine_fmt)
 
-if(cmdLine_Y != " "){
-   y = read(cmdLine_Y)
+if(!cmdLine_scoring_only){
+  Y = read(cmdLine_Y)
 
-   pred = (scores >= 0)
-   pred_labels = pred*positive_label + (1-pred)*negative_label
-   num_correct = sum(pred_labels == y)
-   acc = 100*num_correct/nrow(X)
+  pred = (scores >= 0)
+  pred_labels = pred*positive_label + (1-pred)*negative_label
+  num_correct = sum(pred_labels == Y)
+  acc = 100*num_correct/nrow(X)
 
-   acc_str = "Accuracy (%): " + acc
-   print(acc_str)
-   if(cmdLine_accuracy != " ")
-   write(acc_str, cmdLine_accuracy)
+  acc_str = "Accuracy (%): " + acc
+  print(acc_str)
+  
+  if(cmdLine_accuracy != " ")
+write(acc_str, cmdLine_accuracy)
 
-   if(cmdLine_confusion != " "){
-   pred = 2*pred - 1
-   if(negative_label != -1 | positive_label != +1)
-   y = 2/(positive_label - negative_label)*y - (negative_label + 
positive_label)/(positive_label - negative_label)
-   
-   pred_is_minus = (pred == -1)
-   pred_is_plus = 1 - pred_is_minus
-   y_is_minus = (y == -1)
-   y_is_plus = 1 - y_is_minus
-
-   check_min_y_minus = sum(pred_is_minus*y_is_minus)
-   check_min_y_plus = sum(pred_is_minus*y_is_plus)
-   check_max_y_minus = sum(pred_is_plus*y_is_minus)
-   check_max_y_plus = sum(pred_is_plus*y_is_plus)
+  if(cmdLine_confusion != " "){
+  
+pred = 2*pred - 1
+
+if(negative_label != -1 | positive_label != +1)
+  Y = 2/(positive_label - negative_label)*Y - (negative_label + 
positive_label)/(positive_label - negative_label)
+   
+pred_is_minus = (pred == -1)
+pred_is_plus = 1 - pred_is_minus
+y_is_minus = (Y == -1)
+y_is_plus = 1 - y_is_minus
 
-   #s = check_min_y_minus + "," + check_min_y_plus
-   #s = append(s, check_max_y_minus + "," + check_max_y_plus)
-   #s = append(s, "")
-   #write(s, cmdLine_confusion)
+check_min_y_minus = sum(pred_is_minus*y_is_minus)
+check_min_y_plus = sum(pred_is_minus*y_is_plus)
+check_max_y_minus = sum(pred_is_plus*y_is_minus)
+check_max_y_plus = sum(pred_is_plus*y_is_plus)

-   confusion_mat = matrix(0, rows=3, cols=3)
-confusion_mat[1,2] = negative_label
-confusion_mat[1,3] = positive_label
-confusion_mat[2,1] = negative_label
-confusion_mat[3,1] = positive_label
-confusion_mat[2,2] = check_min_y_minus
-  

[28/50] [abbrv] systemml git commit: [SYSTEMML-1969] Support single-precision operations on GPU backend

[reinwald]

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/java/org/apache/sysml/api/DMLScript.java
--
diff --git a/src/main/java/org/apache/sysml/api/DMLScript.java 
b/src/main/java/org/apache/sysml/api/DMLScript.java
index ba447cf..4da874e 100644
--- a/src/main/java/org/apache/sysml/api/DMLScript.java
+++ b/src/main/java/org/apache/sysml/api/DMLScript.java
@@ -163,6 +163,7 @@ public class DMLScript
public static boolean   ENABLE_DEBUG_MODE   = 
DMLOptions.defaultOptions.debug;   // debug mode
public static ExplainType   EXPLAIN = 
DMLOptions.defaultOptions.explainType; // explain type
public static StringDML_FILE_PATH_ANTLR_PARSER = 
DMLOptions.defaultOptions.filePath; // filename of dml/pydml script
+   public static StringFLOATING_POINT_PRECISION = "double";
// data type to use internally
 
/**
 * Global variable indicating the script type (DML or PYDML). Can be 
used

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
--
diff --git a/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java 
b/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
index a49ffda..51ab6a1 100644
--- a/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
+++ b/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
@@ -81,6 +81,10 @@ public class ScriptExecutorUtils {
DMLScript.SYNCHRONIZE_GPU = 
dmlconf.getBooleanValue(DMLConfig.SYNCHRONIZE_GPU);
DMLScript.EAGER_CUDA_FREE = 
dmlconf.getBooleanValue(DMLConfig.EAGER_CUDA_FREE);
DMLScript.STATISTICS_MAX_WRAP_LEN = 
dmlconf.getIntValue(DMLConfig.STATS_MAX_WRAP_LEN);
+   if(DMLScript.USE_ACCELERATOR) {
+   DMLScript.FLOATING_POINT_PRECISION = 
dmlconf.getTextValue(DMLConfig.FLOATING_POINT_PRECISION);
+   
org.apache.sysml.runtime.matrix.data.LibMatrixCUDA.resetFloatingPointPrecision();
+   }
 
boolean exceptionThrown = false;
 

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/java/org/apache/sysml/conf/DMLConfig.java
--
diff --git a/src/main/java/org/apache/sysml/conf/DMLConfig.java 
b/src/main/java/org/apache/sysml/conf/DMLConfig.java
index 0b73ab0..e8bde56 100644
--- a/src/main/java/org/apache/sysml/conf/DMLConfig.java
+++ b/src/main/java/org/apache/sysml/conf/DMLConfig.java
@@ -92,6 +92,7 @@ public class DMLConfig
// Fraction of available memory to use. The available memory is 
computer when the GPUContext is created
// to handle the tradeoff on calling cudaMemGetInfo too often.
public static final String GPU_MEMORY_UTILIZATION_FACTOR = 
"sysml.gpu.memory.util.factor";
+   public static final String FLOATING_POINT_PRECISION = 
"sysml.floating.point.precision"; // String to specify the datatype to use 
internally: supported values are double, single
 
// supported prefixes for custom map/reduce configurations
public static final String PREFIX_MAPRED = "mapred";
@@ -139,6 +140,7 @@ public class DMLConfig
_defaultVals.put(AVAILABLE_GPUS, "-1");
_defaultVals.put(SYNCHRONIZE_GPU,"true" );
_defaultVals.put(EAGER_CUDA_FREE,"false" );
+   _defaultVals.put(FLOATING_POINT_PRECISION,   
"double" );
}

public DMLConfig()
@@ -421,7 +423,7 @@ public class DMLConfig
COMPRESSED_LINALG, 
CODEGEN, CODEGEN_COMPILER, CODEGEN_OPTIMIZER, 
CODEGEN_PLANCACHE, CODEGEN_LITERALS,
EXTRA_GPU_STATS, EXTRA_DNN_STATS, 
EXTRA_FINEGRAINED_STATS, STATS_MAX_WRAP_LEN,
-   AVAILABLE_GPUS, SYNCHRONIZE_GPU, EAGER_CUDA_FREE
+   AVAILABLE_GPUS, SYNCHRONIZE_GPU, 
EAGER_CUDA_FREE, FLOATING_POINT_PRECISION
}; 

StringBuilder sb = new StringBuilder();

http://git-wip-us.apache.org/repos/asf/systemml/blob/abbffc55/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
 
b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
index 5297e61..c7ffdb1 100644
--- 
a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
+++ 
b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
@@ -404,7 +404,7 @@ public 

[17/50] [abbrv] systemml git commit: [SYSTEMML-1970] Performance conv2d-backward-data (for sparse filter)

[reinwald]

[SYSTEMML-1970] Performance conv2d-backward-data (for sparse filter)

This patch follows-up on the recent modification of conv2d backward
filter, by similarly applying a sparse rotate for conv2d backward data.
Furthermore, this also includes the removal of unnecessary allocations
per input row, and thread-local nnz maintenance.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/78a3808e
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/78a3808e
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/78a3808e

Branch: refs/heads/master
Commit: 78a3808e0aaefb0c6f6959611ef119695d4d1d3e
Parents: b261661
Author: Matthias Boehm 
Authored: Sun Oct 22 17:57:29 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 17:57:29 2017 -0700

--
 .../sysml/runtime/matrix/data/LibMatrixDNN.java  |  4 ++--
 .../LibMatrixDNNConv2dBackwardDataHelper.java| 19 ++-
 .../LibMatrixDNNConv2dBackwardFilterHelper.java  | 17 -
 .../matrix/data/LibMatrixDNNConv2dHelper.java|  2 +-
 .../runtime/matrix/data/LibMatrixDNNHelper.java  |  2 +-
 5 files changed, 22 insertions(+), 22 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/78a3808e/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
index b967780..ac66e51 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNN.java
@@ -186,10 +186,10 @@ public class LibMatrixDNN {
if(isEligibleForConv2dBackwardDataDense(params))

Statistics.numNativeSparseConv2dBwdDataCalls.increment();

-   
execute(LibMatrixDNNHelper.getConv2dBackwardDataWorkers(params), params);
+   long nnz = 
execute(LibMatrixDNNHelper.getConv2dBackwardDataWorkers(params), params);

//post-processing: maintain nnz
-   outputBlock.recomputeNonZeros(); 
+   outputBlock.setNonZeros(nnz);
outputBlock.examSparsity();
}


http://git-wip-us.apache.org/repos/asf/systemml/blob/78a3808e/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
index 04c13e6..cd5 100644
--- 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
+++ 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
@@ -78,22 +78,22 @@ public class LibMatrixDNNConv2dBackwardDataHelper {
int PQ = _params.P*_params.Q; int K = _params.K; int 
CRS = _params.C*_params.R*_params.S;
MatrixBlock filter = _params.input1;
MatrixBlock dout = _params.input2;
-   MatrixBlock dout_reshaped = new MatrixBlock(PQ, K, 
false);
-   dout_reshaped.allocateDenseBlock();
+   MatrixBlock outRotate = new MatrixBlock(PQ, K, 
dout.sparse);
+   MatrixBlock outMM = new MatrixBlock(PQ, CRS, false);
+   outRotate.allocateBlock();
LibMatrixDNNRotate180Helper.Rotate180Worker 
rotate180Worker = 
-   
LibMatrixDNNRotate180Helper.Rotate180Worker.getWorker( dout, dout_reshaped, 
_params, true, false);
+   
LibMatrixDNNRotate180Helper.Rotate180Worker.getWorker( dout, outRotate, 
_params, true, false);
long time1 = 0; long time2 = 0;
for(int n = _rl; n < _ru; n++)  {
// rotate180(dout[n,]) => dout_reshaped
rotate180Worker.execute(n, 0);
-   
// dout_reshaped %*% filter => temp
-   MatrixBlock temp = new MatrixBlock(PQ, CRS, 
false);
long t1 = DMLScript.STATISTICS && 
LibMatrixDNN.DISPLAY_STATISTICS ? System.nanoTime() : 0;
-   
LibMatrixDNNHelper.singleThreadedMatMult(dout_reshaped, filter, temp, true, 
false, _params);
+   outMM.reset(PQ, CRS, false);
+  

[46/50] [abbrv] systemml git commit: [SYSTEMML-1980] HopDagValidator: Accept Integer Matrices

[reinwald]

[SYSTEMML-1980] HopDagValidator: Accept Integer Matrices

Under rare conditions a matrix can have INT ValueType and execute correctly.  
For example the program

```
X= Rand( rows=2, cols=2, min=1, max=2)
R = cbind(as.matrix(nrow(X)) * 2, as.matrix(ncol(X)))
```

would throw an exception by the HopValidator because the Hops produced look like
(here, "MI" means matrix data type and integer value type):

```
GENERIC (lines 28-0) [recompile=false]
--(33) u(cast_as_matrix) ([2]) [1,1,1000,1000,-1]MI [0,0,0 -> -MB]
--(35) b(*) (33,[2]) [1,1,1000,1000,-1]MI [0,0,0 -> -MB], CP
--(37) u(cast_as_matrix) ([2]) [1,1,1000,1000,-1]MI [0,0,0 -> -MB]
--(38) b(cbind) (35,37) [1,2,1000,1000,-1]MI [0,0,0 -> -MB], CP
--(44) PWrite R (38,[target/tes...],[false],[TEXT],[false],[,]) 
[1,2,-1,-1,-1]MI [0,0,0 -> -MB], CP
```

This patch relaxes the HopValidator to allow Integer matrices.

Closes #695.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/fc478916
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/fc478916
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/fc478916

Branch: refs/heads/master
Commit: fc47891656e5f804ce8a9ba1085a79d04153a138
Parents: cb1d792
Author: Dylan Hutchison 
Authored: Wed Nov 1 21:55:56 2017 -0700
Committer: Matthias Boehm 
Committed: Wed Nov 1 21:55:57 2017 -0700

--
 src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/fc478916/src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java
--
diff --git a/src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java 
b/src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java
index 7d14532..ce4648a 100644
--- a/src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java
+++ b/src/main/java/org/apache/sysml/hops/rewrite/HopDagValidator.java
@@ -126,7 +126,7 @@ public class HopDagValidator {
 
// check Matrix data type Hops must have Double Value type
if (dt == Expression.DataType.MATRIX )
-   check(vt == Expression.ValueType.DOUBLE, hop,
+   check(vt == Expression.ValueType.DOUBLE || vt == 
Expression.ValueType.INT, hop,
"has Matrix type but Value Type %s is not 
DOUBLE", vt);
 
//recursively process children

[01/50] [abbrv] systemml git commit: [MINOR] Refactoring lib matrixmult/bincell (instruction footprint) [Forced Update!]

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master eb15c5198 -> 0d4672207 (forced update)


[MINOR] Refactoring lib matrixmult/bincell (instruction footprint)

This patch makes a minor refactoring of the libraries for matrix
multiplications and binary cell-wise operations in order to reduce the
instruction footprint and simplify JIT compilation. Specifically, the
methods for dense-dense mm, sparse-dense mm, and safe binary operations
have been split into methods for the major individual cases.

On an end-to-end cnn application, this patch reduced the number of
L1-icache misses from 6,055,257,066 to 5,663,272,573 and the number of
iTLB misses from 289,601,812 to 161,268,707.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a347af3b
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a347af3b
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a347af3b

Branch: refs/heads/master
Commit: a347af3b7b488ac3b296b9b9692f7172d60ac6f5
Parents: 17c5d5a
Author: Matthias Boehm 
Authored: Sun Oct 15 02:22:55 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 15 02:22:55 2017 -0700

--
 .../runtime/matrix/data/LibMatrixBincell.java   | 446 +++--
 .../runtime/matrix/data/LibMatrixMult.java  | 665 ++-
 2 files changed, 580 insertions(+), 531 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/a347af3b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixBincell.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixBincell.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixBincell.java
index 7622137..2878b3b 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixBincell.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixBincell.java
@@ -210,12 +210,9 @@ public class LibMatrixBincell
{
return;
}
-   
-   int rlen = m1.rlen;
-   int clen = m1.clen;
-   BinaryAccessType atype = getBinaryAccessType(m1, m2);

-   if(atype == BinaryAccessType.MATRIX_COL_VECTOR //MATRIX - 
VECTOR
+   BinaryAccessType atype = getBinaryAccessType(m1, m2);
+   if( atype == BinaryAccessType.MATRIX_COL_VECTOR //MATRIX - 
VECTOR
|| atype == BinaryAccessType.MATRIX_ROW_VECTOR)  
{
//note: m2 vector and hence always dense
@@ -232,213 +229,24 @@ public class LibMatrixBincell
}
else //MATRIX - MATRIX
{
-   if(m1.sparse && m2.sparse)
-   {
-   if(ret.sparse)
-   ret.allocateSparseRowsBlock();  
-   
-   //both sparse blocks existing
-   if(m1.sparseBlock!=null && m2.sparseBlock!=null)
-   {
-   SparseBlock lsblock = m1.sparseBlock;
-   SparseBlock rsblock = m2.sparseBlock;
-   
-   if( ret.sparse && 
lsblock.isAligned(rsblock) )
-   {
-   SparseBlock c = ret.sparseBlock;
-   for(int r=0; r

[03/50] [abbrv] systemml git commit: [MINOR] Fix consistency ALS datagen script (factor names, cleanup)

[reinwald]

[MINOR] Fix consistency ALS datagen script (factor names, cleanup)

This patch cleans up the ALS data generation script to use the same
factor names as ALS-CG and remove unnecessary operations.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/06b4b9d5
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/06b4b9d5
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/06b4b9d5

Branch: refs/heads/master
Commit: 06b4b9d5ff04f09c61d44864d36a65b31527bbb3
Parents: 3355914
Author: Matthias Boehm 
Authored: Sun Oct 15 17:04:31 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 15 17:04:31 2017 -0700

--
 scripts/datagen/genRandData4ALS.dml | 19 +++
 1 file changed, 11 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/06b4b9d5/scripts/datagen/genRandData4ALS.dml
--
diff --git a/scripts/datagen/genRandData4ALS.dml 
b/scripts/datagen/genRandData4ALS.dml
index a5838d7..eb9a4a9 100644
--- a/scripts/datagen/genRandData4ALS.dml
+++ b/scripts/datagen/genRandData4ALS.dml
@@ -20,8 +20,8 @@
 #-
 
 Xfile = $X; # input matrix X of size m x n
-Wfile = $W; # original row factor of size m x r
-Hfile = $H; # original col factor of size r x n
+Ufile = $U; # original row factor of size m x r
+Vfile = $V; # original col factor of size r x n
 m = $rows; # no. of rows of X
 n = $cols; # no. of cols of X
 r = $rank; # rank of factorization
@@ -30,15 +30,18 @@ sigma = ifdef ($sigma, 0.01); # variance of Gaussian noise
 fmt = ifdef ($fmt, "binary"); # output format
 
 # generate original factors by sampling from a normal(0,1.0) distribution
-W = rand(rows = m, cols = r, pdf = "normal", seed = 123);
-H = rand(rows = r, cols = n, pdf = "normal", seed = 456);
+U = rand(rows = m, cols = r, pdf = "normal", seed = 123);
+V = rand(rows = n, cols = r, pdf = "normal", seed = 456);
 
 I = floor(rand(rows = nnz, cols = 1, min = 1, max = m + 0.9));
 J = floor(rand(rows = nnz, cols = 1, min = 1, max = n + 0.9));
 X = rand(rows = nnz, cols = 1, pdf = "normal") * sqrt(sigma);
 N = table(I, J, X);
-T = (N != 0);
-X = T * (W %*% H) + T * N;
+X = (N != 0) * (U %*% t(V)) + N;
 write(X, Xfile, format = fmt);
-write(W, Wfile, format = fmt);
-write(H, Hfile, format = fmt);
+if( Ufile != " " )
+  write(U, Ufile, format = fmt);
+if( Vfile != " " ) {
+  V = t(V);
+  write(V, Vfile, format = fmt);
+}

[39/50] [abbrv] systemml git commit: [SYSTEMML-540] Avoid redundant computation of cudnnPoolingForward in max_pool_backward

[reinwald]

[SYSTEMML-540] Avoid redundant computation of cudnnPoolingForward in 
max_pool_backward

- If the max_pool is invoked in the forward pass, then its output can be
  reused by the max_pool_backward rather than calling cudnnPoolingForward
  again. For sentence CNN with 2 epochs, this reduces the time for
  max_pool_backward from 6.361 to 2.966 seconds.

Closes #691.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/06d5bb07
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/06d5bb07
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/06d5bb07

Branch: refs/heads/master
Commit: 06d5bb073792345f7c4b7ecd0fb4454a335cc421
Parents: 118e3c0
Author: Niketan Pansare 
Authored: Sat Oct 28 13:44:37 2017 -0700
Committer: Niketan Pansare 
Committed: Sat Oct 28 13:45:52 2017 -0700

--
 .../org/apache/sysml/hops/ConvolutionOp.java| 163 +--
 .../gpu/ConvolutionGPUInstruction.java  |  43 -
 .../runtime/matrix/data/LibMatrixCuDNN.java |  51 +++---
 .../sysml/test/gpu/NeuralNetworkOpTests.java|  82 ++
 4 files changed, 260 insertions(+), 79 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/06d5bb07/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
--
diff --git a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java 
b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
index 50a7ca3..16a8b63 100644
--- a/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
+++ b/src/main/java/org/apache/sysml/hops/ConvolutionOp.java
@@ -47,14 +47,23 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
private static final boolean THROW_ERROR_IF_INFERRED_SHAPE_MISMATCH = 
true;
// 
-

+   // Specifies the type of this hop
private Hop.ConvOp op;
-
private int _maxNumThreads = -1; //-1 for unlimited
 
private ConvolutionOp() {
//default constructor for clone
}
 
+   /**
+* Create a hop from the builtin expression
+* 
+* @param l name of the hop
+* @param dt datatype (only supports matrix datatype)
+* @param vt valuetype  (only supports matrix valuetype) 
+* @param o type of this hop
+* @param inp input hops
+*/
public ConvolutionOp(String l, DataType dt, ValueType vt, ConvOp o, 
ArrayList inp) 
{
super(l, dt, vt);
@@ -75,8 +84,7 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
HopsException.check(_input.size() >= 1, this, "should have at 
least one input but has %d inputs", _input.size());
}
 
-   public ConvOp getOp()
-   {
+   public ConvOp getOp() {
return op;
}

@@ -163,77 +171,129 @@ public class ConvolutionOp extends Hop  implements 
MultiThreadedHop
return input instanceof ConvolutionOp && ((ConvolutionOp) 
input).getOp() == ConvOp.DIRECT_CONV2D;
}

+   /**
+* Compares the input parameters for max_pool/max_pool_backward 
operations
+* 
+* @return true if the following parameters match: stride=[stride, 
stride], padding=[pad, pad], input_shape=[numImg, numChannels, imgSize, 
imgSize], pool_size=[poolSize1, poolSize2]
+*/
+   private static boolean 
isPoolingParametersEqualAndKnown(ConvolutionParameters param1, 
ConvolutionParameters param2) {
+   return isEqualAndKnown(param1.stride_h, param2.stride_h) && 
isEqualAndKnown(param1.stride_w, param2.stride_w) && 
+   isEqualAndKnown(param1.pad_h, param2.pad_h) && 
isEqualAndKnown(param1.pad_w, param2.pad_w) &&
+   isEqualAndKnown(param1.R, param2.R) && 
isEqualAndKnown(param1.S, param2.S) &&
+   isEqualAndKnown(param1.N, param2.N) && 
isEqualAndKnown(param1.C, param2.C) &&
+   isEqualAndKnown(param1.H, param2.H) && 
isEqualAndKnown(param1.W, param2.W);
+   }
+   
+   private static boolean isEqualAndKnown(int val1, int val2) {
+   return val1 >= 0 && val2 >= 0 && val1 == val2;
+   }
+   
+   /**
+* Returns the output lop of maxpool operation with same parameters as 
this hop.
+* If corresponding output lop is not found or if this is not a 
max_pool_backward operation, this function returns null
+* 
+* @return output lop of maxpool operation with same parameters as this 
hop
+* @throws HopsException if error 
+* @throws LopsException if error
+*/
+   private Lop 

[16/50] [abbrv] systemml git commit: [SYSTEMML-1970] Performance conv2d-backward-filter (for sparse filter)

[reinwald]

[SYSTEMML-1970] Performance conv2d-backward-filter (for sparse filter)

This patch makes a number of performance improvements to sparse and
dense conv2d backward filter:

1) Conv2d backward w/ sparse filter: So far the rotate180 per input row,
converted any dense or sparse input into dense intermediates, which are
then fed into the matrix multiplication. We now rotate sparse filters
into sparse intermediates, which gives very good mm improvements due to
sparse-dense matrix multiplication. This patch also fixes the used
sparsity of related tests, which so far never created sparse inputs.

2) Minor dense conv2d improvements: Additional improvements includes the
removal of unnecessary allocations of matrix multiplication outputs,
more efficient output accumulation, and cache-conscious transpose
addition operations.

On an end-to-end cnn application, this patch improved the runtime from
610s to 498s per epoch.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/b261661a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/b261661a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/b261661a

Branch: refs/heads/master
Commit: b261661a834bfaef1eacc7fa0a14e885811082a1
Parents: 311e4aa
Author: Matthias Boehm 
Authored: Sun Oct 22 02:04:49 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 02:04:49 2017 -0700

--
 .../LibMatrixDNNConv2dBackwardDataHelper.java   |   2 +-
 .../LibMatrixDNNConv2dBackwardFilterHelper.java |  71 +
 .../runtime/matrix/data/LibMatrixDNNHelper.java |   8 +-
 .../matrix/data/LibMatrixDNNIm2ColHelper.java   |  48 +++---
 .../data/LibMatrixDNNRotate180Helper.java   |  67 -
 .../functions/tensor/Conv2DBackwardTest.java| 147 ---
 .../functions/tensor/Conv2DBackwardTest.R   |   4 +-
 .../functions/tensor/Conv2DBackwardTest.dml |   4 +-
 8 files changed, 162 insertions(+), 189 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/b261661a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
index 609af11..04c13e6 100644
--- 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
+++ 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardDataHelper.java
@@ -81,7 +81,7 @@ public class LibMatrixDNNConv2dBackwardDataHelper {
MatrixBlock dout_reshaped = new MatrixBlock(PQ, K, 
false);
dout_reshaped.allocateDenseBlock();
LibMatrixDNNRotate180Helper.Rotate180Worker 
rotate180Worker = 
-   
LibMatrixDNNRotate180Helper.Rotate180Worker.getWorker( dout, 
dout_reshaped.getDenseBlock(), _params, true);
+   
LibMatrixDNNRotate180Helper.Rotate180Worker.getWorker( dout, dout_reshaped, 
_params, true, false);
long time1 = 0; long time2 = 0;
for(int n = _rl; n < _ru; n++)  {
// rotate180(dout[n,]) => dout_reshaped

http://git-wip-us.apache.org/repos/asf/systemml/blob/b261661a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
index b89be82..de45b81 100644
--- 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
+++ 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixDNNConv2dBackwardFilterHelper.java
@@ -22,7 +22,7 @@ import java.util.concurrent.Callable;
 
 import org.apache.sysml.api.DMLScript;
 import 
org.apache.sysml.runtime.matrix.data.LibMatrixDNNIm2ColHelper.Im2colWorker;
-import org.apache.sysml.runtime.util.ConvolutionUtils;
+import 
org.apache.sysml.runtime.matrix.data.LibMatrixDNNRotate180Helper.Rotate180Worker;
 import org.apache.sysml.utils.NativeHelper;
 
 public class LibMatrixDNNConv2dBackwardFilterHelper {
@@ -43,12 +43,13 @@ public class LibMatrixDNNConv2dBackwardFilterHelper {

@Override
public Long call() throws Exception {
-   int CRS = _params.C*_params.R*_params.S; 
-   double [] dout_n = new 

[40/50] [abbrv] systemml git commit: [MINOR] Fix consistency task partitioning in mm, mmchain, codegen row

[reinwald]

[MINOR] Fix consistency task partitioning in mm, mmchain, codegen row

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/d75a669a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/d75a669a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/d75a669a

Branch: refs/heads/master
Commit: d75a669a46381a0a5b54109e7b207613e17ab54e
Parents: 06d5bb0
Author: Matthias Boehm 
Authored: Sun Oct 29 16:06:55 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 29 16:07:05 2017 -0700

--
 .../sysml/runtime/codegen/SpoofRowwise.java | 19 +++
 .../runtime/matrix/data/LibMatrixMult.java  | 56 
 2 files changed, 42 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/d75a669a/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
--
diff --git a/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java 
b/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
index 9d5675b..b0afd88 100644
--- a/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
+++ b/src/main/java/org/apache/sysml/runtime/codegen/SpoofRowwise.java
@@ -39,7 +39,6 @@ import org.apache.sysml.runtime.matrix.data.MatrixBlock;
 import org.apache.sysml.runtime.matrix.data.SparseBlock;
 import org.apache.sysml.runtime.matrix.data.SparseRow;
 import org.apache.sysml.runtime.matrix.data.SparseRowVector;
-import org.apache.sysml.runtime.util.UtilFunctions;
 
 
 public abstract class SpoofRowwise extends SpoofOperator
@@ -198,11 +197,9 @@ public abstract class SpoofRowwise extends SpoofOperator

//core parallel execute
ExecutorService pool = Executors.newFixedThreadPool( k );
-   int nk = (a instanceof CompressedMatrixBlock) ? k :
-   UtilFunctions.roundToNext(Math.min(8*k,m/32), k);
-   int blklen = (int)(Math.ceil((double)m/nk));
-   if( a instanceof CompressedMatrixBlock )
-   blklen = BitmapEncoder.getAlignedBlocksize(blklen);
+   ArrayList blklens = (a instanceof 
CompressedMatrixBlock) ?
+   LibMatrixMult.getAlignedBlockSizes(m, k, 
BitmapEncoder.BITMAP_BLOCK_SZ) :
+   LibMatrixMult.getBalancedBlockSizesDefault(m, k, false);

try
{
@@ -210,9 +207,9 @@ public abstract class SpoofRowwise extends SpoofOperator
//execute tasks
ArrayList tasks = new 
ArrayList<>();
int outLen = out.getNumRows() * 
out.getNumColumns();
-   for( int i=0; i> taskret = 
pool.invokeAll(tasks); 
+   for( int i=0, lb=0; i> taskret = 
pool.invokeAll(tasks);
//aggregate partial results
int len = _type.isColumnAgg() ? 
out.getNumRows()*out.getNumColumns() : 1;
for( Future task : taskret )
@@ -222,8 +219,8 @@ public abstract class SpoofRowwise extends SpoofOperator
else {
//execute tasks
ArrayList tasks = new 
ArrayList<>();
-   for( int i=0; i> taskret = 
pool.invokeAll(tasks);
//aggregate nnz, no need to aggregate results
long nnz = 0;

http://git-wip-us.apache.org/repos/asf/systemml/blob/d75a669a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixMult.java
--
diff --git 
a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixMult.java 
b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixMult.java
index 684f327..a1f648e 100644
--- a/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixMult.java
+++ b/src/main/java/org/apache/sysml/runtime/matrix/data/LibMatrixMult.java
@@ -209,8 +209,7 @@ public class LibMatrixMult
try {
ExecutorService pool = Executors.newFixedThreadPool( k 
);
ArrayList tasks = new ArrayList<>();
-   int nk = (pm2r||pm2c) ? k : 
UtilFunctions.roundToNext(Math.min(8*k,num/32), k);
-   ArrayList blklens = getBalancedBlockSizes(num, 
nk);
+   ArrayList 

[15/50] [abbrv] systemml git commit: [SYSTEMML-1968] Improved codegen optimizer (cost, mat points, pruning)

[reinwald]

[SYSTEMML-1968] Improved codegen optimizer (cost, mat points, pruning)

This patch improves the cost-based codegen optimizer to address wrong
fusion decision for large-scale computations. In detail, this includes:

1) Cost model: The cost model now accounts the broadcast cost for side
inputs in distributed spark operations. Furthermore, this also includes
a fix of calculating the compute costs in case of a mix of row and cell
operations of different dimensions. 

2) Interesting points: To enable the reasoning about side inputs, we now
also consider template switches from cell to row templates as
interesting points.

3) Pruning of row templates: The above changes also revealed hidden
issues in the pruning of unnecessary row templates (conversion to cell
templates), which mistakenly removed necessary row templates, which
ultimately led to runtime errors.

On a large-scale scenario of L2SVM over a 200M x 100 dense input
(160GB), this patch improved the end-to-end runtime for 20 outer
iterations from 942s to 273s (w/o codegen: 644s).


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/311e4aac
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/311e4aac
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/311e4aac

Branch: refs/heads/master
Commit: 311e4aac9833397908a083d0a48d5bd3ba086283
Parents: 6de8f05
Author: Matthias Boehm 
Authored: Sat Oct 21 16:41:53 2017 -0700
Committer: Matthias Boehm 
Committed: Sat Oct 21 17:15:38 2017 -0700

--
 .../sysml/hops/codegen/opt/PlanAnalyzer.java|   2 +-
 .../opt/PlanSelectionFuseCostBasedV2.java   | 131 ++-
 .../hops/codegen/template/CPlanMemoTable.java   |  25 ++--
 .../runtime/codegen/LibSpoofPrimitives.java |   6 +-
 4 files changed, 91 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/311e4aac/src/main/java/org/apache/sysml/hops/codegen/opt/PlanAnalyzer.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanAnalyzer.java 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanAnalyzer.java
index 9910814..7d522b3 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanAnalyzer.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanAnalyzer.java
@@ -267,7 +267,7 @@ public class PlanAnalyzer
for( int i=0; i<3; i++ ) {
if( refs[i] < 0 ) continue;
List tmp = 
memo.getDistinctTemplateTypes(hopID, i, true);
-   if( memo.containsNotIn(refs[i], tmp, true, 
true) )
+   if( memo.containsNotIn(refs[i], tmp, true) )
ret.add(new 
InterestingPoint(DecisionType.TEMPLATE_CHANGE, hopID, refs[i]));
}
}

http://git-wip-us.apache.org/repos/asf/systemml/blob/311e4aac/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
index d2ed3ac..10875e8 100644
--- 
a/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
+++ 
b/src/main/java/org/apache/sysml/hops/codegen/opt/PlanSelectionFuseCostBasedV2.java
@@ -86,6 +86,7 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
//to cover result allocation, write into main memory, and potential 
evictions
private static final double WRITE_BANDWIDTH = 2d*1024*1024*1024;  
//2GB/s
private static final double READ_BANDWIDTH = 32d*1024*1024*1024;  
//32GB/s
+   private static final double READ_BANDWIDTH_BROADCAST = 
WRITE_BANDWIDTH/4;
private static final double COMPUTE_BANDWIDTH = 2d*1024*1024*1024 
//2GFLOPs/core
* InfrastructureAnalyzer.getLocalParallelism();

@@ -146,7 +147,7 @@ public class PlanSelectionFuseCostBasedV2 extends 
PlanSelection
getComputeCosts(memo.getHopRefs().get(hopID), 
computeCosts);

//prepare pruning helpers and prune memo table w/ 
determined mat points
-   StaticCosts costs = new StaticCosts(computeCosts, 
getComputeCost(computeCosts, memo), 
+   StaticCosts costs = new StaticCosts(computeCosts, 
sumComputeCost(computeCosts), 
getReadCost(part, memo), 
getWriteCost(part.getRoots(), memo));
ReachabilityGraph rgraph = STRUCTURAL_PRUNING ? 

[50/50] [abbrv] systemml git commit: [SYSTEMML-1984] Robustness JMLC prepared scripts (outputs and configs)

[reinwald]

[SYSTEMML-1984] Robustness JMLC prepared scripts (outputs and configs)

This patch fixes the robustness of JMLC prepared scripts for deployments
where prepared scripts are not necessarily created and executed by the
same thread. Accordingly, we no longer maintain output variables (for
dynamic recompilation) and compiler configurations in a thread-local
manner, but attach the output variables to the symbol table as well as
set and cleanup thread-local compiler configurations individually at
connection level and on prepared script execute.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/0d467220
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/0d467220
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/0d467220

Branch: refs/heads/master
Commit: 0d4672207185ac183d22e157171a58c5f38a5bb0
Parents: e1f5866
Author: Matthias Boehm 
Authored: Thu Nov 2 20:02:22 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Nov 2 20:02:22 2017 -0700

--
 .../org/apache/sysml/api/jmlc/Connection.java   | 36 ++---
 .../org/apache/sysml/api/jmlc/JMLCProxy.java| 55 
 .../apache/sysml/api/jmlc/PreparedScript.java   | 39 ++
 .../apache/sysml/api/mlcontext/MLContext.java   | 11 
 .../sysml/api/mlcontext/ScriptExecutor.java |  8 +--
 .../apache/sysml/hops/recompile/Recompiler.java | 13 ++---
 .../controlprogram/LocalVariableMap.java| 25 ++---
 .../org/apache/sysml/utils/MLContextProxy.java  | 11 +---
 .../functions/jmlc/MulticlassSVMScoreTest.java  | 41 +--
 9 files changed, 102 insertions(+), 137 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/0d467220/src/main/java/org/apache/sysml/api/jmlc/Connection.java
--
diff --git a/src/main/java/org/apache/sysml/api/jmlc/Connection.java 
b/src/main/java/org/apache/sysml/api/jmlc/Connection.java
index f933396..4caa86c 100644
--- a/src/main/java/org/apache/sysml/api/jmlc/Connection.java
+++ b/src/main/java/org/apache/sysml/api/jmlc/Connection.java
@@ -95,9 +95,10 @@ import org.apache.wink.json4j.JSONObject;
  * 
  */
 public class Connection implements Closeable
-{  
-   private DMLConfig _dmlconf = null;
-
+{
+   private final DMLConfig _dmlconf;
+   private final CompilerConfig _cconf;
+   
/**
 * Connection constructor, the starting point for any other JMLC API 
calls.
 * 
@@ -122,14 +123,17 @@ public class Connection implements Closeable
cconf.set(ConfigType.ALLOW_INDIVIDUAL_SB_SPECIFIC_OPS, false);
cconf.set(ConfigType.ALLOW_CSE_PERSISTENT_READS, false);
cconf.set(ConfigType.CODEGEN_ENABLED, false);
-   ConfigurationManager.setLocalConfig(cconf);
+   _cconf = cconf;

//disable caching globally 
CacheableData.disableCaching();

-   //create thread-local default configuration
+   //create default configuration
_dmlconf = new DMLConfig();
+   
+   //set thread-local configurations for compilation
ConfigurationManager.setLocalConfig(_dmlconf);
+   ConfigurationManager.setLocalConfig(_cconf);
}

/**
@@ -143,10 +147,12 @@ public class Connection implements Closeable
this();

//set optional compiler configurations in current config
-   CompilerConfig cconf = ConfigurationManager.getCompilerConfig();
for( ConfigType configType : configs )
-   cconf.set(configType, true);
-   ConfigurationManager.setLocalConfig(cconf);
+   _cconf.set(configType, true);
+   
+   //set thread-local configurations for compilation
+   ConfigurationManager.setLocalConfig(_dmlconf);
+   ConfigurationManager.setLocalConfig(_cconf);
}

/**
@@ -202,7 +208,7 @@ public class Connection implements Closeable

//language validate
DMLTranslator dmlt = new DMLTranslator(prog);
-   dmlt.liveVariableAnalysis(prog);
+   dmlt.liveVariableAnalysis(prog);
dmlt.validateParseTree(prog);

//hop construct/rewrite
@@ -220,10 +226,6 @@ public class Connection implements Closeable

//final cleanup runtime prog

[13/50] [abbrv] systemml git commit: [SYSTEMML-1903] Fix robustness codegen row ops w/ unknowns

[reinwald]

[SYSTEMML-1903] Fix robustness codegen row ops w/ unknowns

This patch fixes special cases of codegen row templates with partial
unknowns, which is important for robustness during initial compilation
even though the unknowns led to dynamic recompilation during runtime.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/323dd72a
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/323dd72a
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/323dd72a

Branch: refs/heads/master
Commit: 323dd72a8ed18687aa3019387c4ab7b0598bd9d5
Parents: 4f29b34
Author: Matthias Boehm 
Authored: Thu Oct 19 15:07:54 2017 -0700
Committer: Matthias Boehm 
Committed: Thu Oct 19 16:06:14 2017 -0700

--
 .../hops/codegen/template/TemplateRow.java  | 38 ++--
 .../hops/codegen/template/TemplateUtils.java|  2 +-
 2 files changed, 21 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/323dd72a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
index 0389983..e664b9f 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
@@ -250,7 +250,7 @@ public class TemplateRow extends TemplateBase
else if (((AggUnaryOp)hop).getDirection() == 
Direction.Col && ((AggUnaryOp)hop).getOp() == AggOp.SUM ) {
//vector add without temporary copy
if( cdata1 instanceof CNodeBinary && 
((CNodeBinary)cdata1).getType().isVectorScalarPrimitive() )
-   out = new 
CNodeBinary(cdata1.getInput().get(0), cdata1.getInput().get(1), 
+   out = new 
CNodeBinary(cdata1.getInput().get(0), cdata1.getInput().get(1),

((CNodeBinary)cdata1).getType().getVectorAddPrimitive());
else
out = cdata1;
@@ -269,7 +269,7 @@ public class TemplateRow extends TemplateBase
{
//correct input under transpose
cdata1 = TemplateUtils.skipTranspose(cdata1, 
hop.getInput().get(0), tmp, compileLiterals);
-   inHops.remove(hop.getInput().get(0)); 
+   inHops.remove(hop.getInput().get(0));

inHops.add(hop.getInput().get(0).getInput().get(0));

//note: vectorMultAdd applicable to 
vector-scalar, and vector-vector
@@ -310,7 +310,8 @@ public class TemplateRow extends TemplateBase
CNode cdata1 = 
tmp.get(hop.getInput().get(0).getHopID());

// if one input is a matrix then we need to do vector 
by scalar operations
-   if(hop.getInput().get(0).getDim1() > 1 && 
hop.getInput().get(0).getDim2() > 1 ) 
+   if(hop.getInput().get(0).getDim1() > 1 && 
hop.getInput().get(0).getDim2() > 1 
+   || (!hop.dimsKnown() && 
cdata1.getDataType()==DataType.MATRIX ) ) 
{
if( HopRewriteUtils.isUnary(hop, 
SUPPORTED_VECT_UNARY) ) {
String opname = 
"VECT_"+((UnaryOp)hop).getOp().name();
@@ -320,12 +321,11 @@ public class TemplateRow extends TemplateBase
}
else 
throw new RuntimeException("Unsupported 
unary matrix "
-   + "operation: " + 
((UnaryOp)hop).getOp().name());
+   + "operation: " + 
((UnaryOp)hop).getOp().name());
}
else //general scalar case
{
cdata1 = 
TemplateUtils.wrapLookupIfNecessary(cdata1, hop.getInput().get(0));
-   
String primitiveOpName = 
((UnaryOp)hop).getOp().toString();
out = new CNodeUnary(cdata1, 
UnaryType.valueOf(primitiveOpName));
}
@@ -355,7 +355,9 @@ public class TemplateRow extends TemplateBase

// if one input is a 

[06/50] [abbrv] systemml git commit: [SYSTEMML-1964] Extended codegen outer template and rework close types

[reinwald]

[SYSTEMML-1964] Extended codegen outer template and rework close types

This patch makes a major change to the codegen outer template OFMC
conditions in order to increase its applicability, which is crucial for
sparsity exploitation in algorithms such as ALS-CG. In order to
guarantee correctness, this patch also cleans up the close types used
during candidate exploration and consolidates the redundant evaluation
of valid entry points during candidate selection.

Furthermore, this patch also improves the code generation of sparse
binary nodes and outer templates with neq 0 on the main input.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/2ca2d8aa
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/2ca2d8aa
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/2ca2d8aa

Branch: refs/heads/master
Commit: 2ca2d8aa73c4c0463a52d7f299320fc9b3865aea
Parents: 586f822
Author: Matthias Boehm 
Authored: Mon Oct 16 13:44:40 2017 -0700
Committer: Matthias Boehm 
Committed: Mon Oct 16 15:38:58 2017 -0700

--
 .../org/apache/sysml/hops/OptimizerUtils.java   |  5 ++-
 .../sysml/hops/codegen/SpoofCompiler.java   | 41 ++-
 .../sysml/hops/codegen/cplan/CNodeBinary.java   | 12 +++---
 .../sysml/hops/codegen/opt/PlanAnalyzer.java|  3 +-
 .../sysml/hops/codegen/opt/PlanSelection.java   | 19 +
 .../codegen/opt/PlanSelectionFuseCostBased.java |  2 +-
 .../opt/PlanSelectionFuseCostBasedV2.java   |  2 +-
 .../opt/PlanSelectionFuseNoRedundancy.java  |  2 +-
 .../hops/codegen/template/CPlanMemoTable.java   | 42 +---
 .../hops/codegen/template/TemplateBase.java | 31 ++-
 .../hops/codegen/template/TemplateCell.java | 11 +++--
 .../hops/codegen/template/TemplateMultiAgg.java |  6 +--
 .../codegen/template/TemplateOuterProduct.java  | 20 ++
 .../hops/codegen/template/TemplateRow.java  |  8 ++--
 .../hops/codegen/template/TemplateUtils.java| 23 ++-
 .../sysml/hops/rewrite/HopRewriteUtils.java | 12 +-
 .../functions/codegen/MiscPatternTest.java  | 41 +--
 .../scripts/functions/codegen/miscPattern3.R| 34 
 .../scripts/functions/codegen/miscPattern3.dml  | 34 
 .../scripts/functions/codegen/miscPattern4.R| 35 
 .../scripts/functions/codegen/miscPattern4.dml  | 35 
 21 files changed, 328 insertions(+), 90 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/2ca2d8aa/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
--
diff --git a/src/main/java/org/apache/sysml/hops/OptimizerUtils.java 
b/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
index e44e439..5d831e5 100644
--- a/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
+++ b/src/main/java/org/apache/sysml/hops/OptimizerUtils.java
@@ -1017,7 +1017,10 @@ public class OptimizerUtils
||(op==OpOp2.LESS && val==0)
||(op==OpOp2.NOTEQUAL && val==0)
||(op==OpOp2.EQUAL&& val!=0)
-   ||(op==OpOp2.MINUS&& val==0));
+   ||(op==OpOp2.MINUS&& val==0)
+   ||(op==OpOp2.PLUS && val==0)
+   ||(op==OpOp2.MAX  && val<=0)
+   ||(op==OpOp2.MIN  && val>=0));
}

public static double getBinaryOpSparsityConditionalSparseSafe( double 
sp1, OpOp2 op, LiteralOp lit ) {

http://git-wip-us.apache.org/repos/asf/systemml/blob/2ca2d8aa/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java 
b/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
index 0e5e194..5ff90fb 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/SpoofCompiler.java
@@ -39,6 +39,7 @@ import org.apache.sysml.api.DMLScript.RUNTIME_PLATFORM;
 import org.apache.sysml.conf.ConfigurationManager;
 import org.apache.sysml.conf.DMLConfig;
 import org.apache.sysml.hops.codegen.cplan.CNode;
+import org.apache.sysml.hops.codegen.cplan.CNodeBinary.BinType;
 import org.apache.sysml.hops.codegen.cplan.CNodeCell;
 import org.apache.sysml.hops.codegen.cplan.CNodeData;
 import org.apache.sysml.hops.codegen.cplan.CNodeMultiAgg;
@@ -52,6 +53,7 @@ import 
org.apache.sysml.hops.codegen.opt.PlanSelectionFuseCostBased;
 import org.apache.sysml.hops.codegen.opt.PlanSelectionFuseCostBasedV2;
 import 

[19/50] [abbrv] systemml git commit: [SYSTEMML-1971] New codegen vector primitive for counting nnz

[reinwald]

[SYSTEMML-1971] New codegen vector primitive for counting nnz

This patch adds a new codegen vector primitive for rowSums(X!=0), i.e.
for counting the number of non-zeros, which avoids unnecessary dense row
intermediates and is realized as a pure meta data operation for sparse
row inputs. 

After recent optimizer changes, we compiled a row template in ALS-CG for
rowSums(X!=0), which showed severe performance issues on the amazon
books dataset. This was because amazon is a ultra-sparse dataset with
huge number of features (2330066), for which the dense row intermediates
are more than 7 orders of magnitude larger than the sparse row input.
This patch completely removed these performance issues.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/1191dbfe
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/1191dbfe
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/1191dbfe

Branch: refs/heads/master
Commit: 1191dbfe2a20d85bf79f0106312f37df210053cf
Parents: c70cb11
Author: Matthias Boehm 
Authored: Sun Oct 22 19:53:52 2017 -0700
Committer: Matthias Boehm 
Committed: Sun Oct 22 19:53:52 2017 -0700

--
 .../org/apache/sysml/hops/codegen/cplan/CNodeUnary.java |  9 ++---
 .../apache/sysml/hops/codegen/template/TemplateRow.java |  5 +
 .../sysml/hops/codegen/template/TemplateUtils.java  |  3 ++-
 .../org/apache/sysml/hops/rewrite/HopRewriteUtils.java  |  6 ++
 .../sysml/runtime/codegen/LibSpoofPrimitives.java   | 12 
 5 files changed, 31 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/1191dbfe/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeUnary.java
--
diff --git a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeUnary.java 
b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeUnary.java
index b66423e..3a3dc79 100644
--- a/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeUnary.java
+++ b/src/main/java/org/apache/sysml/hops/codegen/cplan/CNodeUnary.java
@@ -30,7 +30,7 @@ public class CNodeUnary extends CNode
 {
public enum UnaryType {
LOOKUP_R, LOOKUP_C, LOOKUP_RC, LOOKUP0, //codegen specific
-   ROW_SUMS, ROW_MINS, ROW_MAXS, //codegen specific
+   ROW_SUMS, ROW_MINS, ROW_MAXS, ROW_COUNTNNZS, //codegen specific
VECT_EXP, VECT_POW2, VECT_MULT2, VECT_SQRT, VECT_LOG,
VECT_ABS, VECT_ROUND, VECT_CEIL, VECT_FLOOR, VECT_SIGN, 
VECT_SIN, VECT_COS, VECT_TAN, VECT_ASIN, VECT_ACOS, VECT_ATAN, 
@@ -52,8 +52,9 @@ public class CNodeUnary extends CNode
switch( this ) {
case ROW_SUMS:
case ROW_MINS:
-   case ROW_MAXS: {
-   String vectName = 
StringUtils.capitalize(this.toString().substring(4,7).toLowerCase());
+   case ROW_MAXS:
+   case ROW_COUNTNNZS: {
+   String vectName = 
StringUtils.capitalize(name().substring(4, name().length()-1).toLowerCase());
return sparse ? "double %TMP% = 
LibSpoofPrimitives.vect"+vectName+"(%IN1v%, %IN1i%, %POS1%, alen, len);\n": 
"
double %TMP% = LibSpoofPrimitives.vect"+vectName+"(%IN1%, %POS1%, %LEN%);\n"; 
}
@@ -244,6 +245,7 @@ public class CNodeUnary extends CNode
case ROW_SUMS:  return "u(R+)";
case ROW_MINS:  return "u(Rmin)";
case ROW_MAXS:  return "u(Rmax)";
+   case ROW_COUNTNNZS: return "u(Rnnz)";
case VECT_EXP:
case VECT_POW2:
case VECT_MULT2:
@@ -308,6 +310,7 @@ public class CNodeUnary extends CNode
case ROW_SUMS:
case ROW_MINS:
case ROW_MAXS:
+   case ROW_COUNTNNZS:
case EXP:
case LOOKUP_R:
case LOOKUP_C:

http://git-wip-us.apache.org/repos/asf/systemml/blob/1191dbfe/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
--
diff --git 
a/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java 
b/src/main/java/org/apache/sysml/hops/codegen/template/TemplateRow.java
index e664b9f..9da04dc 100644
--- 

systemml git commit: [MINOR] added additional examples

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 0d4672207 -> f76f2138a


[MINOR] added additional examples


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/f76f2138
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/f76f2138
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/f76f2138

Branch: refs/heads/master
Commit: f76f2138a4847f2ca52b5bf511907f4838b240b8
Parents: 0d46722
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Fri Nov 3 14:29:13 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Fri Nov 3 14:29:13 2017 -0700

--
 ...DML Tips and Tricks (aka Fun With DML).ipynb | 204 ++-
 1 file changed, 199 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/f76f2138/samples/jupyter-notebooks/DML
 Tips and Tricks (aka Fun With DML).ipynb
--
diff --git a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb
index c0391ce..b2d2fad 100644
--- a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
+++ b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
@@ -4,7 +4,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
-"1. [Cross Validation](#CrossValidation)\n",
+"1. [Replace NaN with mode](#NaN2Mode)\n",
+"* [Use sample builtin function to create sample from matrix](#sample)\n",
+"* [Count of Matching Values in two Matrices/Vectors](#MatchinRows)\n",
+"* [Cross Validation](#CrossValidation)\n",
 "* [Value-based join of two Matrices](#JoinMatrices)\n",
 "* [Filter Matrix to include only Frequent Column 
Values](#FilterMatrix)\n",
 "* [Construct (sparse) Matrix from (rowIndex, colIndex, values) 
triplets](#Construct_sparse_Matrix)\n",
@@ -17,17 +20,17 @@
   },
   {
"cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 15,
"metadata": {
 "collapsed": false,
-"scrolled": true
+"scrolled": false
},
"outputs": [
 {
  "name": "stdout",
  "output_type": "stream",
  "text": [
-  "2017-08-18 21:33:18 UTC\n"
+  "2017-09-22 07:57:57 UTC\n"
  ]
 }
],
@@ -42,6 +45,197 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+"## Replace NaN with mode"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"This functions replaces NaN in column with mode of column"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+  "Before: \n",
+  "1.000 NaN\n",
+  "1.000 NaN\n",
+  "1.000 2.000\n",
+  "2.000 1.000\n",
+  "1.000 2.000\n",
+  "\n",
+  "After: \n",
+  "1.000 2.000\n",
+  "1.000 2.000\n",
+  "1.000 2.000\n",
+  "2.000 1.000\n",
+  "1.000 2.000\n",
+  "\n",
+  "SystemML Statistics:\n",
+  "Total execution time:\t\t0.001 sec.\n",
+  "Number of executed Spark inst:\t0.\n",
+  "\n",
+  "\n"
+ ]
+}
+   ],
+   "source": [
+"prog=\"\"\"\n",
+"# Function for NaN-aware replacement with mode\n",
+"replaceNaNwithMode = function (matrix[double] X, integer colId) \n",
+"  return (matrix[double] X) \n",
+"{\n",
+"   Xi = replace (target=X[,colId], pattern=0/0, 
replacement=max(X[,colId])+1)   # replace NaN with largest value + 1\n",
+"   agg = aggregate (target=Xi, groups=Xi, fn=\"count\")   
# count each distinct value\n",
+"   mode = as.scalar (rowIndexMax(t(agg[1:nrow(agg)-1, ])))
  # mode is max frequent value except last value\n",
+"   X[,colId] = replace (target=Xi, pattern=max(Xi), replacement=mode)

systemml git commit: [Minor] update creation of lite jar

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 820b53046 -> ae819a253


[Minor] update creation of lite jar


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/ae819a25
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/ae819a25
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/ae819a25

Branch: refs/heads/master
Commit: ae819a253097da796e689764e4926d46228e055e
Parents: 820b530
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Fri Nov 3 22:24:04 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Fri Nov 3 22:30:59 2017 -0700

--
 pom.xml | 308 ++-
 src/assembly/lite.xml   |  22 +-
 .../org/apache/sysml/utils/lite/BuildLite.java  |  22 +-
 3 files changed, 198 insertions(+), 154 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/ae819a25/pom.xml
--
diff --git a/pom.xml b/pom.xml
index ff90986..6201c77 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1,23 +1,17 @@
 
-
-http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0

org.apache
@@ -78,45 +72,46 @@

 

-
-  central
-  https://repo1.maven.org/maven2
-  
-true
-  
-
-
+   
+   central
+   https://repo1.maven.org/maven2
+   
+   true
+   
+   
+   
 

 
-   
-   
-   
-   scripts
-   
-   algorithms/obsolete/*
-   datagen/obsolete/*
-   perftest/*
-   staging/**/*
-   nn/test/compare_backends/*
-   nn/test/compare_backends/*
-   nn/examples/caffe2dml/**/*
-
-   
-   scripts
-   
-   
-   src/main/cpp/kernels
-   
-   *.cu
-   
-   kernels
-   
-   
-   src/main/cpp/lib
-   lib
-   
-   
+   
+   
+   
+   scripts
+   
+   algorithms/obsolete/*
+   datagen/obsolete/*
+   perftest/*
+   staging/**/*
+   
nn/test/compare_backends/*
+   
nn/test/compare_backends/*
+   
nn/examples/caffe2dml/**/*
+
+   
+   scripts
+   
+   
+   src/main/cpp/kernels
+   
+   *.cu
+   
+   kernels
+   
+   
+   src/main/cpp/lib
+   lib
+   
+   
 


@@ -180,8 +175,8 @@

 

-   


@@ -209,8 +204,8 @@



-   
+   

${basedir}/src/test_suites/java

  

systemml git commit: [Minor]: - added main manifest to systemml-lite.jar - added several additional class files - regenerated lite.xml [Forced Update!]

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 0d4672207 -> 3ba5a1552 (forced update)


[Minor]: 
- added main manifest to systemml-lite.jar
- added several additional class files
- regenerated lite.xml

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/3ba5a155
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/3ba5a155
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/3ba5a155

Branch: refs/heads/master
Commit: 3ba5a15528ece6abe1ddde0f3a04a0dd839257bc
Parents: ea2a6e4
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Nov 2 23:49:40 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Nov 2 23:49:40 2017 -0700

--
 pom.xml | 11 ++
 src/assembly/lite.xml   | 22 +++-
 .../org/apache/sysml/utils/lite/BuildLite.java  | 36 +++-
 3 files changed, 59 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/3ba5a155/pom.xml
--
diff --git a/pom.xml b/pom.xml
index ff90986..615e93c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1162,6 +1162,17 @@



src/assembly/lite.xml


+   

+   
+   
org.apache.sysml.api.DMLScript
+   
+   

+   
${maven.build.timestamp}
+   
${project.groupId}
+   
${project.artifactId}-lite
+   
${project.version}
+   

+   




http://git-wip-us.apache.org/repos/asf/systemml/blob/3ba5a155/src/assembly/lite.xml
--
diff --git a/src/assembly/lite.xml b/src/assembly/lite.xml
index 908c72e..0212ace 100644
--- a/src/assembly/lite.xml
+++ b/src/assembly/lite.xml
@@ -53,7 +53,7 @@



-   *:antlr4-runtime
+   *:antlr4



@@ -210,6 +210,8 @@



+   
org/apache/commons/cli/OptionValidator.class
+   
org/apache/commons/cli/Util.class

org/apache/commons/cli/AlreadySelectedException.class

org/apache/commons/cli/CommandLine.class

org/apache/commons/cli/CommandLineParser.class
@@ -620,7 +622,12 @@



META-INF/services/org.apache.hadoop.fs.FileSystem
+   
common-version-info.properties

org/apache/hadoop/log/metrics/EventCounter.class
+   
org/apache/hadoop/security/Groups$CachedGroups.class
+   
org/apache/hadoop/security/token/Token.class
+   
org/apache/hadoop/security/token/TokenIdentifier.class
+   
org/apache/hadoop/util/GenericOptionsParser.class

org/apache/hadoop/util/ShutdownHookManager$2.class

org/apache/hadoop/HadoopIllegalArgumentException.class

org/apache/hadoop/conf/Configurable.class
@@ -1027,6 +1034,19 @@
 


+   
*:hadoop-mapreduce-c

systemml git commit: added addition tips and tricks

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 3ba5a1552 -> eb15c5198


added addition tips and tricks

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/eb15c519
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/eb15c519
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/eb15c519

Branch: refs/heads/master
Commit: eb15c5198b10e6fc37c54e5fc07f0241bff933ee
Parents: 3ba5a15
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Fri Nov 3 00:14:09 2017 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Fri Nov 3 00:14:09 2017 -0700

--
 ...DML Tips and Tricks (aka Fun With DML).ipynb | 204 ++-
 1 file changed, 199 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/eb15c519/samples/jupyter-notebooks/DML
 Tips and Tricks (aka Fun With DML).ipynb
--
diff --git a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb
index c0391ce..b2d2fad 100644
--- a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
+++ b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
@@ -4,7 +4,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
-"1. [Cross Validation](#CrossValidation)\n",
+"1. [Replace NaN with mode](#NaN2Mode)\n",
+"* [Use sample builtin function to create sample from matrix](#sample)\n",
+"* [Count of Matching Values in two Matrices/Vectors](#MatchinRows)\n",
+"* [Cross Validation](#CrossValidation)\n",
 "* [Value-based join of two Matrices](#JoinMatrices)\n",
 "* [Filter Matrix to include only Frequent Column 
Values](#FilterMatrix)\n",
 "* [Construct (sparse) Matrix from (rowIndex, colIndex, values) 
triplets](#Construct_sparse_Matrix)\n",
@@ -17,17 +20,17 @@
   },
   {
"cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 15,
"metadata": {
 "collapsed": false,
-"scrolled": true
+"scrolled": false
},
"outputs": [
 {
  "name": "stdout",
  "output_type": "stream",
  "text": [
-  "2017-08-18 21:33:18 UTC\n"
+  "2017-09-22 07:57:57 UTC\n"
  ]
 }
],
@@ -42,6 +45,197 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+"## Replace NaN with mode"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"This functions replaces NaN in column with mode of column"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [
+{
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+  "Before: \n",
+  "1.000 NaN\n",
+  "1.000 NaN\n",
+  "1.000 2.000\n",
+  "2.000 1.000\n",
+  "1.000 2.000\n",
+  "\n",
+  "After: \n",
+  "1.000 2.000\n",
+  "1.000 2.000\n",
+  "1.000 2.000\n",
+  "2.000 1.000\n",
+  "1.000 2.000\n",
+  "\n",
+  "SystemML Statistics:\n",
+  "Total execution time:\t\t0.001 sec.\n",
+  "Number of executed Spark inst:\t0.\n",
+  "\n",
+  "\n"
+ ]
+}
+   ],
+   "source": [
+"prog=\"\"\"\n",
+"# Function for NaN-aware replacement with mode\n",
+"replaceNaNwithMode = function (matrix[double] X, integer colId) \n",
+"  return (matrix[double] X) \n",
+"{\n",
+"   Xi = replace (target=X[,colId], pattern=0/0, 
replacement=max(X[,colId])+1)   # replace NaN with largest value + 1\n",
+"   agg = aggregate (target=Xi, groups=Xi, fn=\"count\")   
# count each distinct value\n",
+"   mode = as.scalar (rowIndexMax(t(agg[1:nrow(agg)-1, ])))
  # mode is max frequent value except last value\n",
+"   X[,colId] = replace (target=Xi, pattern=max(Xi), replacement=mode)

systemml git commit: [Minor]: minor additions to notebooks.

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master fdc24bb7d -> 0ef6b9246


[Minor]: minor additions to notebooks.

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/0ef6b924
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/0ef6b924
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/0ef6b924

Branch: refs/heads/master
Commit: 0ef6b924612951ccd003e8466fc9a911b098297f
Parents: fdc24bb
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Dec 7 16:16:07 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Dec 7 16:29:32 2017 -0800

--
 .../Deep_Learning_Image_Classification.ipynb| 316 --
 .../Linear_Regression_Algorithms_Demo.ipynb | 599 ---
 2 files changed, 915 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/0ef6b924/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb
--
diff --git a/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb 
b/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb
deleted file mode 100644
index 42f249f..000
--- a/samples/jupyter-notebooks/Deep_Learning_Image_Classification.ipynb
+++ /dev/null
@@ -1,316 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-"# Deep Learning Image Classification using Apache SystemML\n",
-"\n",
-"This notebook demonstrates how to train a deep learning model on SystemML 
for the classic [MNIST](http://yann.lecun.com/exdb/mnist/) problem of mapping 
images of single digit numbers to their corresponding numeric representations, 
using a classic [LeNet](http://yann.lecun.com/exdb/lenet/)-like convolutional 
neural network model. See [Neural Networks and Deep 
Learning](http://neuralnetworksanddeeplearning.com/chap6.html) for more 
information on neural networks and deep learning.\n",
-"\n",
-"The downloaded MNIST dataset contains labeled images of handwritten 
digits, where each example is a 28x28 pixel image of grayscale values in the 
range [0,255] stretched out as 784 pixels, and each label is one of 10 possible 
digits in [0,9].  We download 60,000 training examples, and 10,000 test 
examples, where the images and labels are stored in separate matrices.  We then 
train a SystemML LeNet-like convolutional neural network (i.e. \"convnet\", 
\"CNN\") model.  The resulting trained model has an accuracy of 98.6% on the 
test dataset.\n",
-"\n",
-"1. [Download the MNIST data](#download_data)\n",
-"1. [Train a CNN classifier for MNIST handwritten digits](#train)\n",
-"1. [Detect handwritten Digits](#predict)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-"\n",
-"![Image of Image to 
Digit](https://www.wolfram.com/mathematica/new-in-10/enhanced-image-processing/HTMLImages.en/handwritten-digits-classification/smallthumb_10.gif)\n",
-"Mapping images of numbers to numbers\n",
-""
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-"### Note: This notebook is supported with SystemML 0.14.0 and above."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-"collapsed": true
-   },
-   "outputs": [],
-   "source": [
-"!pip show systemml"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-"%matplotlib inline\n",
-"\n",
-"import matplotlib.pyplot as plt\n",
-"import numpy as np\n",
-"import pandas as pd\n",
-"from sklearn import datasets\n",
-"from sklearn.cross_validation import train_test_split  # module 
deprecated in 0.18\n",
-"#from sklearn.model_selection import train_test_split  # use this module 
for >=0.18\n",
-"from sklearn import metrics\n",
-"from systemml import MLContext, dml"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-"ml = MLContext(sc)\n",
-"print(\"Spark Version: {}\".format(sc.versio

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc1 [created] 3561f7352

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc1 [created] 3631d1395

systemml git commit: [maven-release-plugin] prepare release v1.0.0-rc1

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 4dafb85ce -> 054b85efd


[maven-release-plugin] prepare release v1.0.0-rc1


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/054b85ef
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/054b85ef
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/054b85ef

Branch: refs/heads/master
Commit: 054b85efdec92b8f66a125e014798076843e3492
Parents: 4dafb85
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Tue Dec 5 14:13:52 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Tue Dec 5 14:13:52 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/054b85ef/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 78b0428..da9c7d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.1.0-SNAPSHOT
+   1.0.0
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.0.0-rc1


JIRA

systemml git commit: [maven-release-plugin] prepare for next development iteration

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master e9ee73092 -> 4dafb85ce


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/4dafb85c
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/4dafb85c
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/4dafb85c

Branch: refs/heads/master
Commit: 4dafb85ce43300f6decb604ce7a058c2298ff94b
Parents: e9ee730
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Tue Dec 5 13:21:20 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Tue Dec 5 13:21:20 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/4dafb85c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index da9c7d6..78b0428 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.0.0
+   1.1.0-SNAPSHOT
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   v1.0.0-rc1
+   HEAD


JIRA

systemml git commit: [maven-release-plugin] prepare release v1.0.0-rc1

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master f63b8c6fa -> e9ee73092


[maven-release-plugin] prepare release v1.0.0-rc1


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/e9ee7309
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/e9ee7309
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/e9ee7309

Branch: refs/heads/master
Commit: e9ee730920880fa8e68574b4f9271356b0eca33e
Parents: f63b8c6
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Tue Dec 5 13:21:05 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Tue Dec 5 13:21:05 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/e9ee7309/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 78b0428..da9c7d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.1.0-SNAPSHOT
+   1.0.0
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.0.0-rc1


JIRA

svn commit: r23397 - /dev/systemml/1.0.0-rc3/

[reinwald]

Author: reinwald
Date: Tue Dec  5 21:52:00 2017
New Revision: 23397

Log:
deleting older rc3

Removed:
dev/systemml/1.0.0-rc3/

svn commit: r23395 - /dev/systemml/1.0.0-rc1/

[reinwald]

Author: reinwald
Date: Tue Dec  5 21:51:26 2017
New Revision: 23395

Log:
deleting older rc1

Removed:
dev/systemml/1.0.0-rc1/

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc3 [deleted] a62f1d590

systemml git commit: [Minor]: fix notebooks

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 0ef6b9246 -> bafed4986


[Minor]: fix notebooks

Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/bafed498
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/bafed498
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/bafed498

Branch: refs/heads/master
Commit: bafed4986fdbfc7977fa66d359c726c84ad65976
Parents: 0ef6b92
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Dec 7 19:43:50 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Dec 7 19:43:50 2017 -0800

--
 .../Deep Learning Image Classification.ipynb| 408 +
 .../Linear Regression Algorithms Demo.ipynb | 595 +++
 2 files changed, 1003 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/bafed498/samples/jupyter-notebooks/Deep
 Learning Image Classification.ipynb
--
diff --git a/samples/jupyter-notebooks/Deep Learning Image Classification.ipynb 
b/samples/jupyter-notebooks/Deep Learning Image Classification.ipynb
new file mode 100644
index 000..61617fc
--- /dev/null
+++ b/samples/jupyter-notebooks/Deep Learning Image Classification.ipynb
@@ -0,0 +1,408 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"# Deep Learning Image Classification\n",
+"\n",
+"This notebook shows SystemML Deep Learning functionality to map images of 
single digit numbers to their corresponding numeric representations. See 
[Getting Started with Deep Learning and 
Python](http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/)
 for an explanation of the used deep learning concepts and assumptions.\n",
+"\n",
+"The downloaded MNIST dataset contains labeled images of handwritten 
digits, where each example is a 28x28 pixel image of grayscale values in the 
range [0,255] stretched out as 784 pixels, and each label is one of 10 possible 
digits in [0,9]. We download 60,000 training examples, and 10,000 test 
examples, where the format is \"label, pixel_1, pixel_2, ..., pixel_n\". We 
train a SystemML LeNet model. The results of the learning algorithms have an 
accuracy of 98 percent.\n",
+"\n",
+"1. [Install and load SystemML and other libraries](#load_systemml)\n",
+"1. [Download and Access MNIST data](#access_data)\n",
+"1. [Train a CNN classifier for MNIST handwritten digits](#train)\n",
+"1. [Detect handwritten Digits](#predict)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+"collapsed": true
+   },
+   "source": [
+"\n",
+"![Image of Image to 
Digit](https://www.wolfram.com/mathematica/new-in-10/enhanced-image-processing/HTMLImages.en/handwritten-digits-classification/smallthumb_10.gif)\n",
+"Mapping images of numbers to numbers\n",
+""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"\n",
+"## Install and load SystemML and other libraries"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {
+"scrolled": true
+   },
+   "source": [
+"!pip uninstall systemml --y\n",
+"!pip install --user 
https://repository.apache.org/content/groups/snapshots/org/apache/systemml/systemml/1.0.0-SNAPSHOT/systemml-1.0.0-20171201.070207-23-python.tar.gz;
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+"collapsed": false
+   },
+   "outputs": [],
+   "source": [
+"from systemml import MLContext, dml\n",
+"\n",
+"ml = MLContext(sc)\n",
+"\n",
+"print \"Spark Version:\", sc.version\n",
+"print \"SystemML Version:\", ml.version()\n",
+"print \"SystemML Built-Time:\", ml.buildTime()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+"collapsed": true
+   },
+   "outputs": [],
+   "source": [
+"from sklearn import datasets\n",
+"from sklearn.cross_validation import train_test_split\n",
+"from sklearn.metrics import classification_report\n",
+"import pandas as pd\n&qu

svn commit: r23478 - /dev/systemml/1.0.0-rc2/

[reinwald]

Author: reinwald
Date: Fri Dec  8 19:12:18 2017
New Revision: 23478

Log:
Apache SystemML 1.0.0-rc2

Added:
dev/systemml/1.0.0-rc2/
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-python.tar.gz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.tgz.sha512
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip   (with props)
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.asc
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.md5
dev/systemml/1.0.0-rc2/systemml-1.0.0-src.zip.sha512

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz
==
Binary file - no diff available.

Propchange: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.asc Fri Dec  8 19:12:18 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJaKuPMAAoJEGOG/rIi7k//1z0P/Ak6x2XGHkrJZ9VJfAH1zVyH
+FsGydRoHRUwmLNE4impbvI49tBD/XcZRnl/kO3LgB/q8D5kStMIFinMPjWE/6NPw
+DUOijcpE5zfBT18obaallsFQJXV27Gj9s2x8Iy85bbsDwdI1EnzU4w0/hlXe7GSm
+EtgXjoD2X9Arkuxu8hk5VKs6oXwgLnJyLiDwSYZEIXglDdWmtHNaCogu/qvamBIP
+CpQB0VhxIDE+bwJP6whxp1uWUKjTGnLrV++SHeGyWlDr9q6oW1SI/uh40iFyWLCU
+sj9dL7UfvNlS/qtLwv7XqPceIOwmynyaqxWFYXepyam7Ji1zFnINe6NK1YStGtdI
+Phwl+3sjTio4XQwXNTbEzrnXM7aVN51z6hR7DQKUarBYhtgkYaf7II4F3iPQKbY6
+ZI/DITaVIiXFv6tljun1UTFj/x1vRku7lk0Ycwd5JnVzFVYMbDW2u5LYkkW89s3s
+EdglKRA/P8MVh/PZ72jOzOOWsCsGLMVd+1c7fhJ/gBk177pGN0c17itJhR1ayqLG
+LnTbkVce6M5xBVFoW/JoFaX/0chhQl5+aaKOP+z0DzjuKP6pfxaof4WhuLocv8Rx
+dL2YD4x0BfNaWVcvJ584t745uqHF4VfTZvl26Rs+K4UGTbtta1xM4c81+DZiZGkZ
+5YC4G2V0Sc9FLq4IOOY4
+=u7Ut
+-END PGP SIGNATURE-

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5 (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.md5 Fri Dec  8 19:12:18 2017
@@ -0,0 +1 @@
+67c5f733c0d392d8671b9020b64468a7 *systemml-1.0.0-bin.tgz

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512 (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.tgz.sha512 Fri Dec  8 19:12:18 
2017
@@ -0,0 +1 @@
+4e92ee43886c516d565b4371f7721fde4bc0621fc55e4384b8b6792943a47187b50db06c725b202925bc52f2aa52ba8ac1bd497f7e154c5655cf928ee6d6fcde
  systemml-1.0.0-bin.tgz

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip
==
Binary file - no diff available.

Propchange: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc
==
--- dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc (added)
+++ dev/systemml/1.0.0-rc2/systemml-1.0.0-bin.zip.asc Fri Dec  8 19:12:18 2017
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJaKuPMAAoJEGOG/rIi7k//DdAP/RNLaKErwbzUQALe073lVo90
+VPzoi89DVUSZydDGmLPazBx7PHI5xb5l0JgShd49XCevmAH9C5dymHGi19sn7Mj1
+tFNx9cskiKuNEgLOYtIpnuAAe7iyvj9NWdSB8BsJNfRc1Nbih7S5nXuTA4D7TQ8F
+XujAoxqDZeKUKg/hw5NEBYKTZmLOLVtkC3GiNJ8QuYHJv0w4vRADUM5WyAf9AbKn
+szrgcH5PRiJCK3+XTwcPfmnxToxdAISFk0FJWxTbHM7vOml8G/8M0YlbzDnKZyJc
+2KmB8f2+QugbVxUgLIh5koZ3I20IDhJoDTe7sme2RBFSWLV/VKZ/T34s/Nedm76k
+QqfAzrZQRZ6NBGnOOAVhnRpbfItH7MX/FGLcPPVOVn8J9Eeb9rsZTW7eH4tDif/Q
+6/Cjao1Go1FMUSYrESE73GBlXnSvd4QY2yg4BwdQrna5eA2AfxRgnRvXc2sUB9c0
+9B0JhFEVCnzpzaKjx/tKXArQGQLn+zFD+jgZ+zww6et/4Pb3+N7JeL/iUAVELoSd
+ns8778rZQ1h/SK/cQbRtGL9zLuanAUNGm+KE6r1F9slJWQ8A/jKExpyglZbQAHf5

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.0.0-rc2 [created] 0fe49723c

systemml git commit: [maven-release-plugin] prepare for next development iteration

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 6519f3383 -> 7b44380e4


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/7b44380e
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/7b44380e
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/7b44380e

Branch: refs/heads/master
Commit: 7b44380e49425ac20912c2d42de8603a06fa55a1
Parents: 6519f33
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Fri Dec 8 11:09:11 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Fri Dec 8 11:09:11 2017 -0800

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/7b44380e/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8f042b6..78b0428 100644
--- a/pom.xml
+++ b/pom.xml
@@ -18,7 +18,7 @@
18

org.apache.systemml
-   1.0.0
+   1.1.0-SNAPSHOT
systemml
jar
SystemML
@@ -34,7 +34,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   v1.0.0-rc2
+   HEAD


JIRA

systemml git commit: [MINOR] switch build/test server to travis-ci

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 73f9d417d -> f7078c292


[MINOR] switch build/test server to travis-ci

Closes #765.


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/f7078c29
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/f7078c29
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/f7078c29

Branch: refs/heads/master
Commit: f7078c292fe46331a91e1b18394e5579c25cf324
Parents: 73f9d41
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Mon May 21 13:08:31 2018 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Mon May 21 13:08:31 2018 -0700

--
 .travis.yml|  52 +
 conf/log4j-silent.properties   | 282 
 pom.xml|  76 +---
 src/test/scripts/installDependencies.R |   2 +-
 4 files changed, 387 insertions(+), 25 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/f7078c29/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
new file mode 100644
index 000..a0c308b
--- /dev/null
+++ b/.travis.yml
@@ -0,0 +1,52 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+dist: trusty
+
+language: java
+
+jdk:
+  - oraclejdk8
+
+addons:
+  apt:
+sources:
+- r-packages-trusty
+packages:
+- r-base-dev
+
+cache:
+  apt: true
+  directories:
+# caching .m2 causes an error loading hadoop-yarn-common-2.6.0.jar. Not sure 
why.
+#- ${HOME}/.m2
+- ${HOME}/R
+- /usr/local/lib/R/site-library
+
+install:
+  - sudo Rscript ./src/test/scripts/installDependencies.R
+
+before_script:
+#  this is not needed anymore since adding authentication object in code for 
running hadoop/spark local
+#  - chmod -R 755 *
+
+script:
+#  - mvn clean verify jacoco:report coveralls:report
+  - mvn clean verify
+
+after_success:
+#  -  mvn test jacoco:report coveralls:report
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/systemml/blob/f7078c29/conf/log4j-silent.properties
--
diff --git a/conf/log4j-silent.properties b/conf/log4j-silent.properties
new file mode 100644
index 000..7e7adda
--- /dev/null
+++ b/conf/log4j-silent.properties
@@ -0,0 +1,282 @@
+#-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+# 
+#   http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-
+
+# Define some default values that can be overridden by system properties
+hadoop.root.logger=INFO,console
+hadoop.log.dir=.
+hadoop.log.file=hadoop.log
+hadoop.security.logger=OFF
+
+# Security appender
+log4j.appender.RFAS=org.apache.log4j.RollingFileAppender 
+log4j.appender.RFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}
+log4j.appender.RFAS.layout=org.apache.log4j.PatternLayout
+log4j.appender.RFAS.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+log4j.appender.RFAS.MaxFileSize=${hadoop.security.log.maxfilesize}
+log4j.appender.RFAS.MaxBackupIndex=${hadoop.security.log.maxbackupindex}
+
+log4j.logger.org.apache.spark=ERROR
+
+#
+# Job Summary Appender 
+#
+# Use fo

systemml git commit: [MINOR] add 2 new DML examples to Jupyter notebook

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master d44b3280f -> 8084dc127


[MINOR] add 2 new DML examples to Jupyter notebook


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/8084dc12
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/8084dc12
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/8084dc12

Branch: refs/heads/master
Commit: 8084dc1276663c7dfaa8983b13efb0ec11e6ee1e
Parents: d44b328
Author: Berthold Reinwald 
Authored: Mon Jun 4 16:54:14 2018 -0700
Committer: Berthold Reinwald 
Committed: Mon Jun 4 16:54:14 2018 -0700

--
 ...DML Tips and Tricks (aka Fun With DML).ipynb | 691 ---
 1 file changed, 576 insertions(+), 115 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/8084dc12/samples/jupyter-notebooks/DML
 Tips and Tricks (aka Fun With DML).ipynb
--
diff --git a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With 
DML).ipynb
index b2d2fad..6dd096c 100644
--- a/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
+++ b/samples/jupyter-notebooks/DML Tips and Tricks (aka Fun With DML).ipynb
@@ -4,13 +4,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
-"1. [Replace NaN with mode](#NaN2Mode)\n",
+"1. [Create all value pairs for v1 and v2](#AllValuePairs)\n",
+"* [Replace NaN with mode](#NaN2Mode)\n",
 "* [Use sample builtin function to create sample from matrix](#sample)\n",
 "* [Count of Matching Values in two Matrices/Vectors](#MatchinRows)\n",
 "* [Cross Validation](#CrossValidation)\n",
 "* [Value-based join of two Matrices](#JoinMatrices)\n",
 "* [Filter Matrix to include only Frequent Column 
Values](#FilterMatrix)\n",
-"* [Construct (sparse) Matrix from (rowIndex, colIndex, values) 
triplets](#Construct_sparse_Matrix)\n",
+"* [(Sparse) Matrix to/from (rowIndex, colIndex, values) conversions 
(i,j,v)](#Construct_sparse_Matrix)\n",
 "* [Find and remove duplicates in columns or 
rows](#Find_and_remove_duplicates)\n",
 "* [Set based Indexing](#Set_based_Indexing)\n",
 "* [Group by Aggregate using Linear Algebra](#Multi_column_Sorting)\n",
@@ -20,25 +21,95 @@
   },
   {
"cell_type": "code",
-   "execution_count": 15,
+   "execution_count": null,
"metadata": {
 "collapsed": false,
 "scrolled": false
},
+   "outputs": [],
+   "source": [
+"from systemml import MLContext, dml\n",
+"ml = MLContext(sc)\n",
+"\n",
+"print (ml.buildTime())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+"## Create all value pairs for v1 and v2"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 50,
+   "metadata": {
+"collapsed": false
+   },
"outputs": [
 {
  "name": "stdout",
  "output_type": "stream",
  "text": [
-  "2017-09-22 07:57:57 UTC\n"
+  "2.000\n",
+  "1.000\n",
+  "8.000\n",
+  "3.000\n",
+  "5.000\n",
+  "6.000\n",
+  "7.000\n",
+  "\n",
+  "80.000\n",
+  "20.000\n",
+  "50.000\n",
+  "\n",
+  "2.000 80.000\n",
+  "2.000 20.000\n",
+  "2.000 50.000\n",
+  "1.000 80.000\n",
+  "1.000 20.000\n",
+  "1.000 50.000\n",
+  "8.000 80.000\n",
+  "8.000 20.000\n",
+  "8.000 50.000\n",
+  "3.000 80.000\n",
+  "3.000 20.000\n",
+  "3.000 50.000\n",
+  "5.000 80.000\n",
+  "5.000 20.000\n",
+  "5.000 50.000\n",
+  "6.000 80.000\n",
+  "6.000 20.000\n",
+  "6.000 50.000\n",
+  "7.000 80.000\n",
+  "7.000 20.000\n",
+  "7.000 50.000\n",
+  "\n",
+  "SystemML Statistics:\n",
+  "Total execution time:\t\t0.000 sec.\n",
+  "Number of executed Spark inst:\t0.\n",
+  "\n",
+  "\n"
  

svn commit: r1818859 - in /systemml/site: ./ release-notes/

[reinwald]

Author: reinwald
Date: Wed Dec 20 22:35:12 2017
New Revision: 1818859

URL: http://svn.apache.org/viewvc?rev=1818859=rev
Log:
Add 1.0.0 link to documentation page

Modified:
systemml/site/community.html
systemml/site/documentation.html
systemml/site/download.html
systemml/site/get-started.html
systemml/site/index.html
systemml/site/install-systemml.html
systemml/site/privacy-policy.html
systemml/site/release-notes/systemml-release-notes-0.10.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.11.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.12.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.13.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.14.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.15.0.html
systemml/site/release-notes/systemml-release-notes-0.9.0-incubating.html
systemml/site/roadmap.html
systemml/site/security.html
systemml/site/tutorial-kdd2017.html
systemml/site/tutorials.html

Modified: systemml/site/community.html
URL: 
http://svn.apache.org/viewvc/systemml/site/community.html?rev=1818859=1818858=1818859=diff
==
--- systemml/site/community.html (original)
+++ systemml/site/community.html Wed Dec 20 22:35:12 2017
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
   
   
+  
   Privacy Policy
   
   
+  
   Security
   
 

Modified: systemml/site/documentation.html
URL: 
http://svn.apache.org/viewvc/systemml/site/documentation.html?rev=1818859=1818858=1818859=diff
==
--- systemml/site/documentation.html (original)
+++ systemml/site/documentation.html Wed Dec 20 22:35:12 2017
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
   
   
+  
   Privacy Policy
   

systemml-website git commit: Add 1.0.0 link to documentation page

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master 6fc5bada5 -> 733cbd741


Add 1.0.0 link to documentation page


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/733cbd74
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/733cbd74
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/733cbd74

Branch: refs/heads/master
Commit: 733cbd741d5bf823971e3409cc741a3b59aa3c75
Parents: 6fc5bad
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Wed Dec 20 14:16:40 2017 -0800
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Wed Dec 20 14:16:40 2017 -0800

--
 _src/_data/project.yml  | 2 +-
 _src/documentation.html | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/733cbd74/_src/_data/project.yml
--
diff --git a/_src/_data/project.yml b/_src/_data/project.yml
index 65b84b9..391ed1a 100644
--- a/_src/_data/project.yml
+++ b/_src/_data/project.yml
@@ -50,7 +50,7 @@ issues_list_archive_markmail:
 jira: SYSTEMML
 
 release_name: systemml
-release_version: 0.15.0
+release_version: 1.0.0
 
 source_repository: https://git-wip-us.apache.org/repos/asf/systemml.git
 source_repository_mirror: https://github.com/apache/systemml

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/733cbd74/_src/documentation.html
--
diff --git a/_src/documentation.html b/_src/documentation.html
index ea4e948..5f3963e 100644
--- a/_src/documentation.html
+++ b/_src/documentation.html
@@ -39,6 +39,7 @@ limitations under the License.
   Documentation such as programming guides, reference guides, and API 
docs is available online for each version of SystemML below:
   
 http://apache.github.io/systemml/;>Latest
+1.0.0
 0.15.0
 0.14.0
 0.13.0

svn commit: r1818973 - /systemml/site/release-notes/systemml-release-notes-1.0.0.html

[reinwald]

Author: reinwald
Date: Thu Dec 21 19:55:01 2017
New Revision: 1818973

URL: http://svn.apache.org/viewvc?rev=1818973=rev
Log:
Add missing 1.0.0 release notes

Added:
systemml/site/release-notes/systemml-release-notes-1.0.0.html

Added: systemml/site/release-notes/systemml-release-notes-1.0.0.html
URL: 
http://svn.apache.org/viewvc/systemml/site/release-notes/systemml-release-notes-1.0.0.html?rev=1818973=auto
==
--- systemml/site/release-notes/systemml-release-notes-1.0.0.html (added)
+++ systemml/site/release-notes/systemml-release-notes-1.0.0.html Thu Dec 21 
19:55:01 2017
@@ -0,0 +1,293 @@
+
+
+
+
+  
+
+  
+  
+
+  SystemML 1.0.0 Release Notes
+  
+  
+  
+  
+
+  
+  
+  
+
+  
+  https://systemml.apache.org/; />
+  
+  
+  
+  
+
+  
+  
+
+  
+  
+  
+
+  
+  
+
+ 
+  
+
+
+  Apache SystemML
+  
+
+  
+  
+
+
+
+Download
+
+  
+  
+  
+
+
+
+Get Started
+
+  
+  
+  
+
+
+
+Docs
+
+  
+  
+  
+
+
+
+Roadmap
+
+  
+  
+  
+
+Community 
+
+  
+  
+  
+  Get Involved
+  
+  
+  
+  https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
+  
+  
+  
+  https://github.com/apache/systemml; 
target="_blank">Source Code
+  
+  
+  
+  https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
+  
+  
+  
+  Roadmap
+  
+
+
+  
+  
+  
+
+Apache 
+
+  
+  
+  
+  http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
+  
+  
+  
+  http://www.apache.org/licenses/; target="_blank">Apache 
License
+  
+  
+  
+  http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
+  
+  
+  
+  http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
+  
+  
+  
+  Privacy Policy
+  
+  
+  
+  Security
+  
+
+
+  
+  
+
+  
+
+ 
+
+
+
+  
+
+
+  
+
+  Apache SystemML 1.0.0 Release 
Notes
+
+  
+
+
+
+  
+
+
+  The Apache SystemML 1.0.0 release was approved on December 12, 2017. 
The release includes enhancements, features, and additions as listed below.
+
+  API Enhancements
+  
+JMLC/MLContext parameter passing w/ and w/o meta data.
+  
+
+  Deep Learning
+  
+Improved CPU convolution function performance, including 
sparsity.
+Caffe2DML.
+Enhanced/additional NN layers, e.g. conv2d transpose and depthwise 
convolution
+NN architecture summary table.
+[Experimental] Keras2DML.
+  
+
+  New Scripts / Algorithms
+  
+Autoencoder.
+Enhanced PCA.
+  
+
+  Features  new Functions
+  
+sinh, cosh, and tanh.
+transformcolmap.
+n-ary rbind/cbind.
+order with multiple order-by columns.
+  
+
+  Compiler
+  
+Improved rewrites, e.g. merging of statement block sequences.
+IPA effectiveness.
+Automatic vectorization of indexing pairs.
+Instruction generation for memory efficiency.
+Instruction code organization.
+[Experimental] Code generation, code gen optimizer, and 
multi-threaded codegen operators.
+  
+
+  Robustness  Performance
+  
+ParFor/HOP Memory budgets for Spark cluster configurations.
+JMLC prepared scripts (thread affinity for outputs and configs, 
script cloning, configuration management).
+Sparse-dense binary cell wise operations.
+  
+
+  [Experimental] GPU
+  
+GPU conv2d and memory estimates.
+Additional kernels,e.g. right indexing.
+Single precision backend.
+  
+
+  Additional Packages
+  
+SystemML Lite artifact, a minimumm-size uber JAR for embeddability 
(w/o Hadoop or Spark dependencies).
+  
+
+  Environment
+  
+Compression on by default.
+Exploitation of native BLAS libraries.
+JDK 8 (dropped JDK 7).
+jCUDA for Windows  Linux (x86_64, ppc64le) included.
+Fine-grained runtime statistics.
+Refactored configurations parameters to have sysml prefix
+Performance test suite.
+  
+
+  Other
+  
+Refreshed/new examples and notebooks (DML examples

svn commit: r28790 - /dev/systemml/1.2.0-rc1/

[reinwald]

Author: reinwald
Date: Fri Aug 17 06:06:42 2018
New Revision: 28790

Log:
Apache SystemML 1.2.0-rc1

Added:
dev/systemml/1.2.0-rc1/
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.sha512
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.sha512
dev/systemml/1.2.0-rc1/systemml-1.2.0-lite.jar   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-lite.jar.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-lite.jar.sha512
dev/systemml/1.2.0-rc1/systemml-1.2.0-python.tar.gz   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-python.tar.gz.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-python.tar.gz.sha512
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.tgz   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.tgz.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.tgz.sha512
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.zip   (with props)
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.zip.asc
dev/systemml/1.2.0-rc1/systemml-1.2.0-src.zip.sha512

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz
==
Binary file - no diff available.

Propchange: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.asc
==
--- dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.asc (added)
+++ dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.asc Fri Aug 17 06:06:42 2018
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJbdmUlAAoJEGOG/rIi7k//0Q4QALz0fejp9ljthkObXdQKM6nL
+paQBWW7IhMiS2URB10GIFOMlXurnwaq2k0jjJ2uN0621I2myRy6hjo6PL1gVPTjf
+KzVAHS6LzBS1IiIJgThad/RtK8u2D//XuYUU7/+C+b9/iMUWgSHMH2AMvNY2rSnf
+FYPmlb7wEKcmQm9WfBYKylxVJveBLkJ8YtoX4Ti1aGS2uHMlpa3YY1GZ3FShKFbF
+8M5B5ZFAhclVKb53YXwRGkXqCYwlPN3EUisRaKqxReFWU32NFGzCG44MmFxolpv/
+z2w13Sdr3U8M/AvTkGlaaOReC+ZLyzkNgLsgbBjofQCEXU3BRCmlmYhLkrgcMf59
++YKJdARTKzS35v223kHMMqEtFELulNE/eDisXRnlIJ8sYw7fayuheiwhTTT6igfz
+fdRCjHwn0MdZOKK0RzvzVjsAabo38gLV1jLANZV44iBFwuI9M/dbGAVi9vn9n4eP
+pqDxyF9ejZD/M7u8NTVob2gS780gx7ev51ZRpJiEZ0RpWsk7zpLdhL1fUgUPQCO4
+mM6vggybNdRuwAjRA3E2Sv4ZimHvkCGximrrs3EmFORQLMRw8k864dpA8bUONGcA
+hVI3H/4JsGEp9t+c7Tep6WpGdzjwIvWAdP/sEfu+d9uiFwxDXEDUBCGkHKuN29GA
+tCvX/WhcHPD+tI2kXr3W
+=lULh
+-END PGP SIGNATURE-

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.sha512
==
--- dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.sha512 (added)
+++ dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.tgz.sha512 Fri Aug 17 06:06:42 
2018
@@ -0,0 +1 @@
+4262fa40a6383525fd47b4afc6f9f17573eb6587b544be6d8a77f0a76f9a707f9159db4be4d2fc75bed1bff530728b480f116eaa41d836c1c56a66570296c043
  systemml-1.2.0-bin.tgz

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip
==
Binary file - no diff available.

Propchange: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip
--
svn:mime-type = application/octet-stream

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.asc
==
--- dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.asc (added)
+++ dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.asc Fri Aug 17 06:06:42 2018
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJbdmUlAAoJEGOG/rIi7k//3JEQAKFjxesXIc/dpX94Cot4kAw9
+lMa0EmzZOGgfPQG3+jI10FYa5UFcBSOpdRcyWNmbbC0FOuF/xipyaQOvYxf2ds67
+9T0KOkgJ8pVHQSf8GQsgL7EJI9Db9O/KhR610DotcW3s84hzj3hquZ5CPF2kX8Y4
+exd6MdrYDaYdt2PyGZMwGrTjU4kIef3V4HiI9qN5/evbmJEvLMEa2VBO59M4mVEq
+JLiWRpKydMxQCj+7FWs8pGcGKvppeSzul72FTyDrZL0V0M3kHaa3aDqQmYo/nhtM
+c0KIhXQU4MYc2mnGI9uNrS4GEk1g7Zx0rGL7GHg/rvoGp/t50TByWCLHF9FZAQRx
+TO+X09EAAaKEkQCFxb1lVOjAWjVBFaxHhPBwX1RWSWcSha5vxZlLLTAmWlhMZHG2
+mLG/Aq9vu6tRd47Bo/nrDZuOxEj7CV+FzzojtQs9VukxLvbSd/hqFkyOuaGydonj
+SAXVu0xUf9SBsmjhPvzPmhsDwuGvlWWcYrKISG2PQQWBs8s8Twj2dXE5IC4eNCZU
+yAuRFWmea5POuf2Er/2TEukrk9HvGxswwGU51wh1Cptu4dkwmtM1XXtNRM6j9J4K
+zJlS64DhACK2CCdGZbAFU7KN30ACZ4R3lGRbNASbcxAhCR1dmfdR9kXJmchj5vqC
+ZXc8aK+++wkzoBIW0NLo
+=cK1T
+-END PGP SIGNATURE-

Added: dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.sha512
==
--- dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.sha512 (added)
+++ dev/systemml/1.2.0-rc1/systemml-1.2.0-bin.zip.sha512 Fri Aug 17 06:06:42 
2018
@@ -0,0 +1

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Tags:  refs/tags/v1.2.0-rc1 [created] f229a9bda

systemml git commit: [maven-release-plugin] prepare for next development iteration

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master a1a05e29f -> 53cdc590c


[maven-release-plugin] prepare for next development iteration


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/53cdc590
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/53cdc590
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/53cdc590

Branch: refs/heads/master
Commit: 53cdc590c5b3398135707d0f18fa80cf42c5225d
Parents: a1a05e2
Author: Berthold Reinwald 
Authored: Thu Aug 16 22:58:28 2018 -0700
Committer: Berthold Reinwald 
Committed: Thu Aug 16 22:58:28 2018 -0700

--
 pom.xml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/53cdc590/pom.xml
--
diff --git a/pom.xml b/pom.xml
index d69307e..e3d2785 100644
--- a/pom.xml
+++ b/pom.xml
@@ -25,7 +25,7 @@
18

org.apache.systemml
-   1.2.0
+   1.3.0-SNAPSHOT
systemml
jar
SystemML
@@ -41,7 +41,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   v1.2.0-rc1
+   HEAD


JIRA

systemml git commit: [maven-release-plugin] prepare release v1.2.0-rc1

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/master 709a5c5b4 -> a1a05e29f


[maven-release-plugin] prepare release v1.2.0-rc1


Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/a1a05e29
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/a1a05e29
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/a1a05e29

Branch: refs/heads/master
Commit: a1a05e29f6ee78f3c33fea355f62c78ce21766ee
Parents: 709a5c5
Author: Berthold Reinwald 
Authored: Thu Aug 16 22:58:13 2018 -0700
Committer: Berthold Reinwald 
Committed: Thu Aug 16 22:58:13 2018 -0700

--
 pom.xml | 11 ---
 1 file changed, 4 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml/blob/a1a05e29/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 8a25b9a..d69307e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -17,8 +17,7 @@
  * specific language governing permissions and limitations
  * under the License.
 -->
-http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
-   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; 
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0

org.apache
@@ -26,7 +25,7 @@
18

org.apache.systemml
-   1.2.0-SNAPSHOT
+   1.2.0
systemml
jar
SystemML
@@ -42,7 +41,7 @@
scm:git:g...@github.com:apache/systemml

scm:git:https://git-wip-us.apache.org/repos/asf/systemml

https://git-wip-us.apache.org/repos/asf?p=systemml.git
-   HEAD
+   v1.2.0-rc1


JIRA
@@ -602,9 +601,7 @@
package


-   
+   

svn commit: r1838996 - /systemml/site/documentation.html

[reinwald]

Author: reinwald
Date: Sat Aug 25 00:25:03 2018
New Revision: 1838996

URL: http://svn.apache.org/viewvc?rev=1838996=rev
Log:
add 1.2.0 link to documentation page

Modified:
systemml/site/documentation.html

Modified: systemml/site/documentation.html
URL: 
http://svn.apache.org/viewvc/systemml/site/documentation.html?rev=1838996=1838995=1838996=diff
==
--- systemml/site/documentation.html (original)
+++ systemml/site/documentation.html Sat Aug 25 00:25:03 2018
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
   
   
+  
   Privacy Policy
   
   
+  
   Security
   
 
@@ -163,6 +178,7 @@
   Documentation such as programming guides, reference guides, and API 
docs is available online for each version of SystemML below:
   
 http://apache.github.io/systemml/;>Latest
+1.2.0
 1.1.0
 1.0.0
 0.15.0

svn commit: r1838994 - in /systemml/site/docs/1.2.0: ./ api/ api/java/ api/java/apidocs/ api/java/apidocs/org/ api/java/apidocs/org/apache/ api/java/apidocs/org/apache/sysml/ api/java/apidocs/org/apac

[reinwald]

Author: reinwald
Date: Fri Aug 24 23:59:52 2018
New Revision: 1838994

URL: http://svn.apache.org/viewvc?rev=1838994=rev
Log:
Add 1.2.0 docs to website


[This commit notification would consist of 260 parts, 
which exceeds the limit of 50 ones, so it was shortened to the summary.]

systemml-website git commit: Add 1.2.0 link to documentation page

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master 239a86e50 -> 5f0f7508a


Add 1.2.0 link to documentation page


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/5f0f7508
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/5f0f7508
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/5f0f7508

Branch: refs/heads/master
Commit: 5f0f7508a6aa63b10985548fc5e713bc05f7d56d
Parents: 239a86e
Author: Berthold Reinwald 
Authored: Fri Aug 24 17:07:13 2018 -0700
Committer: Berthold Reinwald 
Committed: Fri Aug 24 17:07:13 2018 -0700

--
 _src/_data/project.yml  | 2 +-
 _src/documentation.html | 1 +
 package.json| 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/5f0f7508/_src/_data/project.yml
--
diff --git a/_src/_data/project.yml b/_src/_data/project.yml
index c8282bb..e141033 100644
--- a/_src/_data/project.yml
+++ b/_src/_data/project.yml
@@ -50,7 +50,7 @@ issues_list_archive_markmail:
 jira: SYSTEMML
 
 release_name: systemml
-release_version: 1.1.0
+release_version: 1.2.0
 
 source_repository: https://git-wip-us.apache.org/repos/asf/systemml.git
 source_repository_mirror: https://github.com/apache/systemml

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/5f0f7508/_src/documentation.html
--
diff --git a/_src/documentation.html b/_src/documentation.html
index ba0664f..6c19173 100644
--- a/_src/documentation.html
+++ b/_src/documentation.html
@@ -39,6 +39,7 @@ limitations under the License.
   Documentation such as programming guides, reference guides, and API 
docs is available online for each version of SystemML below:
   
 http://apache.github.io/systemml/;>Latest
+1.2.0
 1.1.0
 1.0.0
 0.15.0

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/5f0f7508/package.json
--
diff --git a/package.json b/package.json
index 58ae998..a1a0495 100644
--- a/package.json
+++ b/package.json
@@ -5,7 +5,7 @@
   "main": "index.js",
   "dependencies": {},
   "devDependencies": {
-"browser-sync": "^2.23.6",
+"browser-sync": "^2.24.6",
 "gulp": "^3.9.1",
 "gulp-concat": "^2.6.1",
 "gulp-if": "^2.0.1",
@@ -14,7 +14,7 @@
 "gulp-uglify": "^2.1.2",
 "gulp-useref": "^3.1.2",
 "gulp-util": "^3.0.7",
-"susy": "^2.2.12"
+"susy": "^2.2.14"
   },
   "scripts": {
 "test": "echo \"Error: no test specified\" && exit 1"

svn commit: r28947 - /dev/systemml/1.2.0-rc1/ /release/systemml/1.2.0/

[reinwald]

Author: reinwald
Date: Fri Aug 24 21:51:42 2018
New Revision: 28947

Log:
Move SystemML 1.2.0 from dev to release

Added:
release/systemml/1.2.0/
  - copied from r28946, dev/systemml/1.2.0-rc1/
Removed:
dev/systemml/1.2.0-rc1/

svn commit: r1839017 - /systemml/site/release-notes/systemml-release-notes-1.2.0.html

[reinwald]

Author: reinwald
Date: Sat Aug 25 05:21:31 2018
New Revision: 1839017

URL: http://svn.apache.org/viewvc?rev=1839017=rev
Log:
additional roadmap file

Added:
systemml/site/release-notes/systemml-release-notes-1.2.0.html

Added: systemml/site/release-notes/systemml-release-notes-1.2.0.html
URL: 
http://svn.apache.org/viewvc/systemml/site/release-notes/systemml-release-notes-1.2.0.html?rev=1839017=auto
==
--- systemml/site/release-notes/systemml-release-notes-1.2.0.html (added)
+++ systemml/site/release-notes/systemml-release-notes-1.2.0.html Sat Aug 25 
05:21:31 2018
@@ -0,0 +1,295 @@
+
+
+
+
+  
+
+  
+  
+
+  SystemML 1.2.0 Release Notes
+  
+  
+  
+  
+
+  
+  
+  
+
+  
+  https://systemml.apache.org/; />
+  
+  
+  
+  
+
+  
+  
+
+  
+  
+  
+
+  
+  
+
+ 
+  
+
+
+  Apache SystemML
+  
+
+  
+  
+
+
+
+Download
+
+  
+  
+  
+
+
+
+Get Started
+
+  
+  
+  
+
+
+
+Docs
+
+  
+  
+  
+
+
+
+Roadmap
+
+  
+  
+  
+
+Community 
+
+  
+  
+  
+  Get Involved
+  
+  
+  
+  https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
+  
+  
+  
+  https://github.com/apache/systemml; 
target="_blank">Source Code
+  
+  
+  
+  https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
+  
+  
+  
+  Roadmap
+  
+
+
+  
+  
+  
+
+Apache 
+
+  
+  
+  
+  http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
+  
+  
+  
+  http://www.apache.org/licenses/; target="_blank">Apache 
License
+  
+  
+  
+  http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
+  
+  
+  
+  http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
+  
+  
+  
+  Privacy Policy
+  
+  
+  
+  Security
+  
+
+
+  
+  
+
+  
+
+ 
+
+
+
+  
+
+
+  
+
+  Apache SystemML 1.2.0 Release 
Notes
+
+  
+
+
+
+  
+
+
+  The Apache SystemML 1.2.0 release was approved on Aug. 24th, 2018. 
The release includes enhancements, features, and additions as listed below.
+
+  New Builtin 
Functions/Operations/Scripts/Features
+  
+Factorization Machines
+Support for functions with default parameters
+exists() for checking of existing variables
+Triangular matrix functions: lower.tri() and 
upper.tri()
+New nary min/max operations
+as.matrix() over list of scalars
+Function calls with named function arguments
+Convolution operations (forward/backward)
+Maxpooling operations (forward)
+Support for bias_add and bias_mult
+Global constants
+Added support for rowProd/colProd
+DML eval function
+New data type list for lists and structs
+  
+
+  API
+  
+JMLC API extension for passing multiple dml scripts
+  
+
+  Compiler  Runtime
+  
+Code motion framework
+Global subexpression elimination
+Optional rewrite for hoisting loop-invariant operations
+Improved IPA constant propagation and replacement
+ParFor Data Partitioning Rewrite on Hops instead of Statements
+New rewrites for chains of comparisons
+Extended rewrite framework for codegen plans
+Improved parfor optimizer rewrite for in-place-update
+Rework function block recompilation
+Support rowMeans in codegen row templates
+  
+
+  Performance Improvements
+  
+Improved multi-threading of unary aggregates
+Reuse of fair scheduler pools in local parfor workers
+Performance issues Spark ctable(X,Y) w/ large num distinct
+Performance instruction generation
+Performance sample operations
+Performance ultra-sparse block operations
+New native tsmm operator and its integration
+Multi-threaded unary operations (e.g., exp, log, sigmoid)
+  
+
+  Bug Fixes
+  
+Memory leak buffer pool due to missing variable cleanup
+Missing buffer pool serialization of compressed matrices
+Compilation failure on inferring size of reshapes w/ zero 
rows/columns
+Incorrect result for min/ma

systemml-website git commit: Add release notes 1.2.0 and updated road map

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master 5f0f7508a -> bd61b2065


Add release notes 1.2.0 and updated road map


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/bd61b206
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/bd61b206
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/bd61b206

Branch: refs/heads/master
Commit: bd61b2065247ee9d6b953721395583dfa589b4c7
Parents: 5f0f750
Author: Berthold Reinwald 
Authored: Fri Aug 24 21:53:51 2018 -0700
Committer: Berthold Reinwald 
Committed: Fri Aug 24 21:53:51 2018 -0700

--
 .../systemml-release-notes-1.2.0.md | 105 +++
 _src/roadmap.html   |  68 ++--
 2 files changed, 141 insertions(+), 32 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/bd61b206/_src/release-notes/systemml-release-notes-1.2.0.md
--
diff --git a/_src/release-notes/systemml-release-notes-1.2.0.md 
b/_src/release-notes/systemml-release-notes-1.2.0.md
new file mode 100644
index 000..fc8c2a7
--- /dev/null
+++ b/_src/release-notes/systemml-release-notes-1.2.0.md
@@ -0,0 +1,105 @@
+---
+layout: page
+title: SystemML 1.2.0 Release Notes
+description: Project Release Notes
+group: nav-right
+---
+
+
+
+  
+
+  {{ site.data.project.name }} 1.2.0 Release Notes
+
+  
+
+
+
+  
+
+
+The Apache SystemML 1.2.0 release was approved on Aug. 24th, 2018. The release 
includes enhancements, features, and additions as listed below.
+
+
+### New Builtin Functions/Operations/Scripts/Features
+- Factorization Machines
+- Support for functions with default parameters
+- `exists()` for checking of existing variables
+- Triangular matrix functions: `lower.tri()` and `upper.tri()`
+- New nary min/max operations
+- `as.matrix()` over list of scalars
+- Function calls with named function arguments
+- Convolution operations (forward/backward)
+- Maxpooling operations (forward)
+- Support for bias_add and bias_mult
+- Global constants
+- Added support for rowProd/colProd
+- DML eval function
+- New data type list for lists and structs
+
+### API
+- JMLC API extension for passing multiple dml scripts
+
+### Compiler & Runtime
+- Code motion framework
+- Global subexpression elimination
+- Optional rewrite for hoisting loop-invariant operations
+- Improved IPA constant propagation and replacement
+- ParFor Data Partitioning Rewrite on Hops instead of Statements
+- New rewrites for chains of comparisons
+- Extended rewrite framework for codegen plans
+- Improved parfor optimizer rewrite for in-place-update
+- Rework function block recompilation
+- Support rowMeans in codegen row templates
+
+### Performance Improvements
+- Improved multi-threading of unary aggregates
+- Reuse of fair scheduler pools in local parfor workers
+- Performance issues Spark ctable(X,Y) w/ large num distinct
+- Performance instruction generation
+- Performance sample operations
+- Performance ultra-sparse block operations
+- New native tsmm operator and its integration
+- Multi-threaded unary operations (e.g., exp, log, sigmoid)
+
+### Bug Fixes 
+- Memory leak buffer pool due to missing variable cleanup
+- Missing buffer pool serialization of compressed matrices
+- Compilation failure on inferring size of reshapes w/ zero rows/columns
+- Incorrect result for min/max over matrices with NaNs
+- Missing support for external functions with variable number of outputs
+- Reblock ultra-sparse matrix fails with index out of bounds
+- Performance issue CSE on DAGs w/ many root nodes (e.g., resnet200)
+- Non-fused bias_add builtin creates incorrect results over sparse inputs
+- Inconsistent namespace naming depending on OS
+- Codegen failing on three-way multi-aggregate
+- Codegen optimizer failing for MLogreg special cases
+- Failing matrix market to binary reblock with zero rows/columns
+
+### Deprecated/Removed/Cleanup
+- Opt level 4
+- File-based removeEmpty()
+- Cleanup exception handling apis/compiler/runtime
+
+### Experimental 
+- Parameter server: local backend and distributed
+- Sparsity estimators
+
+### [JIRA release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342959=12319522)

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/bd61b206/_src/roadmap.html
--
diff --git a/_src/roadmap.html b/_src/roadmap.html
index 9dc3044..f247051 100644
--- a/_src/roadmap.html
+++ b/_src/roadmap.html
@@ -36,15 +36,17 @@ limitations under the License.
 
   
 
-  Planned for Future SystemML 1.2
+  Planned for Future Releases
   
+Completion of Prior Experimental F

svn commit: r1839016 - in /systemml/site: ./ release-notes/

[reinwald]

Author: reinwald
Date: Sat Aug 25 05:19:43 2018
New Revision: 1839016

URL: http://svn.apache.org/viewvc?rev=1839016=rev
Log:
Add 1.2.0 release notes and updated roadmap

Modified:
systemml/site/community.html
systemml/site/download.html
systemml/site/get-started.html
systemml/site/index.html
systemml/site/install-systemml.html
systemml/site/privacy-policy.html
systemml/site/release-notes/systemml-release-notes-0.10.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.11.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.12.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.13.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.14.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.15.0.html
systemml/site/release-notes/systemml-release-notes-0.9.0-incubating.html
systemml/site/release-notes/systemml-release-notes-1.0.0.html
systemml/site/release-notes/systemml-release-notes-1.1.0.html
systemml/site/roadmap.html
systemml/site/security.html
systemml/site/tutorial-kdd2017.html
systemml/site/tutorials.html

Modified: systemml/site/community.html
URL: 
http://svn.apache.org/viewvc/systemml/site/community.html?rev=1839016=1839015=1839016=diff
==
--- systemml/site/community.html (original)
+++ systemml/site/community.html Sat Aug 25 05:19:43 2018
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
   
   
+  
   Privacy Policy
   
   
+  
   Security
   
 

Modified: systemml/site/download.html
URL: 
http://svn.apache.org/viewvc/systemml/site/download.html?rev=1839016=1839015=1839016=diff
==
--- systemml/site/download.html (original)
+++ systemml/site/download.html Sat Aug 25 05:19:43 2018
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks

[systemml] Git Push Summary

[reinwald]

Repository: systemml
Updated Branches:
  refs/heads/branch-1.2.0 [created] a1a05e29f

svn commit: r26016 - /dev/systemml/1.1.0-rc2/ /release/systemml/1.1.0/

[reinwald]

Author: reinwald
Date: Wed Mar 28 17:16:54 2018
New Revision: 26016

Log:
Move SystemML 1.1.0 from dev to release

Added:
release/systemml/1.1.0/
  - copied from r26015, dev/systemml/1.1.0-rc2/
Removed:
dev/systemml/1.1.0-rc2/

systemml-website git commit: Add 1.1.0 link to documentation page

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master 24affb7a7 -> ebc2ac593


Add 1.1.0 link to documentation page


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/ebc2ac59
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/ebc2ac59
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/ebc2ac59

Branch: refs/heads/master
Commit: ebc2ac59389d9b2d8c8946254ebfe6c07f6268a2
Parents: 24affb7
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Mar 29 13:04:10 2018 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Mar 29 13:04:10 2018 -0700

--
 _src/_data/project.yml  |  2 +-
 _src/documentation.html |  1 +
 gulpfile.js |  2 +-
 package.json| 12 +---
 4 files changed, 8 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/ebc2ac59/_src/_data/project.yml
--
diff --git a/_src/_data/project.yml b/_src/_data/project.yml
index 391ed1a..c8282bb 100644
--- a/_src/_data/project.yml
+++ b/_src/_data/project.yml
@@ -50,7 +50,7 @@ issues_list_archive_markmail:
 jira: SYSTEMML
 
 release_name: systemml
-release_version: 1.0.0
+release_version: 1.1.0
 
 source_repository: https://git-wip-us.apache.org/repos/asf/systemml.git
 source_repository_mirror: https://github.com/apache/systemml

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/ebc2ac59/_src/documentation.html
--
diff --git a/_src/documentation.html b/_src/documentation.html
index 5f3963e..ba0664f 100644
--- a/_src/documentation.html
+++ b/_src/documentation.html
@@ -39,6 +39,7 @@ limitations under the License.
   Documentation such as programming guides, reference guides, and API 
docs is available online for each version of SystemML below:
   
 http://apache.github.io/systemml/;>Latest
+1.1.0
 1.0.0
 0.15.0
 0.14.0

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/ebc2ac59/gulpfile.js
--
diff --git a/gulpfile.js b/gulpfile.js
index 99aa00a..c9161a9 100644
--- a/gulpfile.js
+++ b/gulpfile.js
@@ -49,7 +49,7 @@ gulp.task('js', () => {
 
 gulp.task('jekyll:build', (done) => {
   browserSync.notify(config.messages.jekyllBuild);
-  const jekyll = child.spawn('jekyll', ['build', '--incremental', '--drafts'], 
{stdio: 'inherit'})
+  const jekyll = child.spawn('jekyll', ['build', '--drafts'], {stdio: 
'inherit'})
 .on('close', done);
 });
 

http://git-wip-us.apache.org/repos/asf/systemml-website/blob/ebc2ac59/package.json
--
diff --git a/package.json b/package.json
index 203fa91..58ae998 100644
--- a/package.json
+++ b/package.json
@@ -3,17 +3,15 @@
   "version": "0.1.0",
   "description": "",
   "main": "index.js",
-  "dependencies": {
-"gulp": "^3.9.1"
-  },
+  "dependencies": {},
   "devDependencies": {
-"browser-sync": "^2.17.0",
+"browser-sync": "^2.23.6",
 "gulp": "^3.9.1",
-"gulp-concat": "^2.6.0",
+"gulp-concat": "^2.6.1",
 "gulp-if": "^2.0.1",
 "gulp-sass": "^2.3.2",
-"gulp-sourcemaps": "^1.9.1",
-"gulp-uglify": "^2.0.0",
+"gulp-sourcemaps": "^1.12.1",
+"gulp-uglify": "^2.1.2",
 "gulp-useref": "^3.1.2",
 "gulp-util": "^3.0.7",
 "susy": "^2.2.12"

systemml-website git commit: Add 1.1.0 release notes, updated roadmap, and download page

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master ebc2ac593 -> 244d3a91f


Add 1.1.0 release notes, updated roadmap, and download page


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/244d3a91
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/244d3a91
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/244d3a91

Branch: refs/heads/master
Commit: 244d3a91f6383b48f3630895b683e0c105e045e8
Parents: ebc2ac5
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Mar 29 15:55:36 2018 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Mar 29 15:55:36 2018 -0700

--
 _src/download.html | 14 +++---
 _src/roadmap.html  | 49 +++--
 2 files changed, 34 insertions(+), 29 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/244d3a91/_src/download.html
--
diff --git a/_src/download.html b/_src/download.html
index b29ff11..58004e6 100644
--- a/_src/download.html
+++ b/_src/download.html
@@ -47,32 +47,32 @@ limitations under the License.
   
   
 Name
-MD5
+SHA512
 Signature
   
   
 http://www.apache.org/dyn/closer.lua/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.tgz" target="_blank"> {{ 
site.data.project.release_name }}-{{ site.data.project.release_version }} 
(Binary tgz)
-https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.tgz.md5">MD5 
+https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.tgz.sha512">SHA512 
 https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.tgz.asc">ASC
   
   
 http://www.apache.org/dyn/closer.lua/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.zip" target="_blank">{{ 
site.data.project.release_name }}-{{ site.data.project.release_version }} 
(Binary zip)
-https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.zip.md5">MD5
+https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.zip.sha512">SHA512
 https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-bin.zip.asc">ASC
   
   
 http://www.apache.org/dyn/closer.lua/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.tgz" target="_blank">{{ 
site.data.project.release_name }}-{{ site.data.project.release_version }} 
(Source tgz)
-https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.tgz.md5">MD5
+https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.tgz.sha512">SHA512
 https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.tgz.asc">ASC
   
   
 http://www.apache.org/dyn/closer.lua/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.zip" target="_blank">{{ 
site.data.project.release_name }}-{{ site.data.project.release_version }} 
(Source zip)
-https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.zip.md5">MD5
+https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.zip.sha512">SHA512
 https://www.apache.org/dist/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-src.zip.asc">ASC
   
   
 http://www.apache.org/dyn/closer.lua/systemml/{{ 
site.data.project.release_version }}/systemml-{{ 
site.data.project.release_version }}-python.tgz" target="_blank">

systemml-website git commit: add 1.1.0 release notes

[reinwald]

Repository: systemml-website
Updated Branches:
  refs/heads/master 244d3a91f -> 80b5a5f78


add 1.1.0 release notes


Project: http://git-wip-us.apache.org/repos/asf/systemml-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml-website/commit/80b5a5f7
Tree: http://git-wip-us.apache.org/repos/asf/systemml-website/tree/80b5a5f7
Diff: http://git-wip-us.apache.org/repos/asf/systemml-website/diff/80b5a5f7

Branch: refs/heads/master
Commit: 80b5a5f789a8c994cc335a0adbc1dad074f7e33f
Parents: 244d3a9
Author: Berthold Reinwald <reinw...@us.ibm.com>
Authored: Thu Mar 29 16:23:43 2018 -0700
Committer: Berthold Reinwald <reinw...@us.ibm.com>
Committed: Thu Mar 29 16:23:43 2018 -0700

--
 .../systemml-release-notes-1.1.0.md | 78 
 1 file changed, 78 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/systemml-website/blob/80b5a5f7/_src/release-notes/systemml-release-notes-1.1.0.md
--
diff --git a/_src/release-notes/systemml-release-notes-1.1.0.md 
b/_src/release-notes/systemml-release-notes-1.1.0.md
new file mode 100644
index 000..6aff3bd
--- /dev/null
+++ b/_src/release-notes/systemml-release-notes-1.1.0.md
@@ -0,0 +1,78 @@
+---
+layout: page
+title: SystemML 1.1.0 Release Notes
+description: Project Release Notes
+group: nav-right
+---
+
+
+
+  
+
+  {{ site.data.project.name }} 1.1.0 Release Notes
+
+  
+
+
+
+  
+
+
+The Apache SystemML 1.1.0 release was approved on March 28, 2018. The release 
includes enhancements, features, and additions as listed below.
+
+### New Builtin Functions/Operations
+- `ifelse`
+- `assert` 
+- Deep learning builtin functions: `avg_pool`, `avg_pool_backward`
+- Second-order `eval` 
+- Accumulator operator += 
+- Bitwise operators not, and, or, xor, & LShift, Rshift
+- Logical operator support over matrices AND/OR/NOT/XOR
+
+### Additional Layers in the NN library
+- Average pooling
+- Upsampling
+- Low-rank fully connected
+
+### New Capabilities/Features
+- Dense matrix blocks >16GB, and operations
+- Enable additional ParFor result aggregation operations
+- UDFs callable in expressions
+- Zero rows/columns matrices and updated operations such as removeEmpty
+- Matrix-matrix multiplication over compressed matrices
+- Extended Caffe2DML and Keras2DML APIs
+
+### Compiler & Runtime
+- Use common thread pool
+- Single-precision support for native conv2d and mm operations
+- Improved nnz maintenance, runtime propogation and memory management
+- Robustness for matrices with larger than int dimensions
+- [Experimental] Codegen extensions: operation support, extended optimizer, 
see SYSTEMML-2065
+
+### Performance Improvements
+- sparse left indexing, sparse reshape, ultra-sparse transpose, ultra-sparse 
rand, binary in-place operations, sparse relu backward, maxpooling, sparse 
im2col, ultra-sparse conv2d, read of short-wide sparse matrices, avoid 
unnecessary evictions, lock-free statistics maintenance, spark cpmm, spark 
aggregates, spark reshape, spark binary ops, etc.
+
+### Bug Fixes 
+- in APIs, performance, optimizer, runtime, GPU backend, Spark backend
+
+### Deprecate
+- Support for Spark 2.1 / 2.2 (make switch to newer ANTLR version)
+
+### [JIRA release 
notes](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319522=12342282)

svn commit: r1828040 - in /systemml/site: ./ release-notes/

[reinwald]

Author: reinwald
Date: Thu Mar 29 23:06:33 2018
New Revision: 1828040

URL: http://svn.apache.org/viewvc?rev=1828040=rev
Log:
Add 1.1.0 link to documentation and release notes, and update roadmap and 
download page.

Modified:
systemml/site/community.html
systemml/site/documentation.html
systemml/site/download.html
systemml/site/get-started.html
systemml/site/index.html
systemml/site/install-systemml.html
systemml/site/privacy-policy.html
systemml/site/release-notes/systemml-release-notes-0.10.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.11.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.12.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.13.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.14.0-incubating.html
systemml/site/release-notes/systemml-release-notes-0.15.0.html
systemml/site/release-notes/systemml-release-notes-0.9.0-incubating.html
systemml/site/release-notes/systemml-release-notes-1.0.0.html
systemml/site/roadmap.html
systemml/site/security.html
systemml/site/tutorial-kdd2017.html
systemml/site/tutorials.html

Modified: systemml/site/community.html
URL: 
http://svn.apache.org/viewvc/systemml/site/community.html?rev=1828040=1828039=1828040=diff
==
--- systemml/site/community.html (original)
+++ systemml/site/community.html Thu Mar 29 23:06:33 2018
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
target="_blank">Thanks
   
   
+  
   Privacy Policy
   
   
+  
   Security
   
 

Modified: systemml/site/documentation.html
URL: 
http://svn.apache.org/viewvc/systemml/site/documentation.html?rev=1828040=1828039=1828040=diff
==
--- systemml/site/documentation.html (original)
+++ systemml/site/documentation.html Thu Mar 29 23:06:33 2018
@@ -56,6 +56,7 @@
   
 
 
+
 Download
 
   
@@ -63,6 +64,7 @@
   
 
 
+
 Get Started
 
   
@@ -70,6 +72,7 @@
   
 
 
+
 Docs
 
   
@@ -77,6 +80,7 @@
   
 
 
+
 Roadmap
 
   
@@ -87,18 +91,23 @@
 
   
   
+  
   Get Involved
   
   
+  
   https://issues.apache.org/jira/browse/SYSTEMML; 
target="_blank">Issue Tracker
   
   
+  
   https://github.com/apache/systemml; 
target="_blank">Source Code
   
   
+  
   https://github.com/apache/systemml-website; 
target="_blank">Website Source Code
   
   
+  
   Roadmap
   
 
@@ -111,21 +120,27 @@
 
   
   
+  
   http://www.apache.org/foundation/how-it-works.html; 
target="_blank">Apache Software Foundation
   
   
+  
   http://www.apache.org/licenses/; target="_blank">Apache 
License
   
   
+  
   http://www.apache.org/foundation/sponsorship; 
target="_blank">Sponsorship
   
   
+  
   http://www.apache.org/foundation/thanks.html; 
tar

svn commit: r25971 - in /dev/systemml/1.1.0-rc2: systemml-1.1.0-lite.jar.asc systemml-1.1.0-lite.jar.sha512

[reinwald]

Author: reinwald
Date: Mon Mar 26 17:02:46 2018
New Revision: 25971

Log:
added missing systemml-1.1.0-lite.jar.asc and ystemml-1.1.0-lite.jar.sha512

Added:
dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.asc
dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.sha512

Added: dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.asc
==
--- dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.asc (added)
+++ dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.asc Mon Mar 26 17:02:46 2018
@@ -0,0 +1,17 @@
+-BEGIN PGP SIGNATURE-
+Comment: GPGTools - https://gpgtools.org
+
+iQIcBAABCgAGBQJauSb7AAoJEGOG/rIi7k//losQAIl1dG1wXvRJ9c4zPIEeiiUX
+Fftbdj789hMzKhT/WmTW64f7EODG1wOjLWtAreH5YdjACtIs4GpatnE0PGYEL4M2
+23E8LKK58QPE+OyU6zzWgjov4qHnrY3yDGYfZeOhw4diK6JJTF3GgcTeXtBythDD
+5xOyZbUvNuxWywzzupvuiNsqze5bkQhPON5UDE/PMiiFUQuaiJWkxW2+cI5/ajn5
+5Er6ACQvkTrDk1oRSlLhZV6ClXKnoXSzY6g9Ho4ug+4Jcm7fs97acwIiJ2MaDziW
+ol6b/jD3zyz7ogLFtQxB5yUow/MFiEqkqVOL5mr6wXX9ezIYeea0OUGQfpF7pwk9
++mciP6irQBOKvtaDOKz5r3sJHXGEUC5mkOdfeeSae49TcjUdY9MRPavyMsiyWAAt
+VaMmHsiHfIR9v+l17RnIWGbrO/y4vvHCBqN4rRU6I+iDvI2rDVt5+pFqg1RKLFjV
+OvV96iQWEoiqTSxpg+Jg9kTeZ/Q5HAIrjmjWxWauHFVkN7LChe0HxcrP+PLlDEoM
+7Bo+gupcqXrZVd5OKrLMfi8ugVXOdBUfCPoo6ZcSUieMibkXV+EWjxTa7d7t1y6d
+XxekbxW2zwhMEzAPpHheKE/4RoLJux7+fPq58PNvsGsFdUiKqKnIdpGjb6PTjr15
+6DTbigeXRosL8LzMv6z+
+=68l2
+-END PGP SIGNATURE-

Added: dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.sha512
==
--- dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.sha512 (added)
+++ dev/systemml/1.1.0-rc2/systemml-1.1.0-lite.jar.sha512 Mon Mar 26 17:02:46 
2018
@@ -0,0 +1 @@
+e49772b74c9af6c4f9c035666df6105727234ce34f870f85f15dbe3abcc59140a61eb60279f06684da691072270fe0fefa8ea28155a8efa7ec5c09baf485875b
  systemml-1.1.0-lite.jar

svn commit: r1828046 [11/20] - /systemml/site/docs/1.1.0/

[reinwald]

Added: systemml/site/docs/1.1.0/developer-tools-systemml.html
URL: 
http://svn.apache.org/viewvc/systemml/site/docs/1.1.0/developer-tools-systemml.html?rev=1828046=auto
==
--- systemml/site/docs/1.1.0/developer-tools-systemml.html (added)
+++ systemml/site/docs/1.1.0/developer-tools-systemml.html Fri Mar 30 04:31:05 
2018
@@ -0,0 +1,253 @@
+
+
+
+
+  
+
+SystemML Developer Tools - SystemML 1.1.0
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+http://systemml.apache.org/;>
+
+
+http://systemml.apache.org/;>Apache 
SystemML™
+1.1.0
+
+
+Toggle navigation
+
+
+
+
+
+
+
+Overview
+https://github.com/apache/systemml;>GitHub
+
+Documentation
+
+Running SystemML:
+https://github.com/apache/systemml;>SystemML GitHub README
+Spark MLContext
+Spark 
Batch Mode
+Hadoop 
Batch Mode
+Standalone 
Guide
+Java Machine Learning 
Connector (JMLC)
+
+Language Guides:
+DML 
Language Reference
+Beginner's Guide to DML and 
PyDML
+Beginner's Guide for Python Users
+Reference 
Guide for Python Users
+
+ML Algorithms:
+Algorithms Reference
+
+Tools:
+Debugger 
Guide
+IDE Guide
+
+Other:
+Contributing to SystemML
+Engine 
Developer Guide
+Troubleshooting Guide
+Release 
Process
+
+
+
+
+API Docs
+
+Java
+Python
+
+
+
+
+Issues
+
+JIRA:
+https://issues.apache.org/jira/browse/SYSTEMML;>SystemML JIRA
+
+
+
+
+
+
+
+
+
+  
+SystemML Developer Tools
+  
+
+  
+
+Useful Tools for Developing SystemML:
+
+
+  IntelliJ
+  Import SystemML project 
to IntelliJ
+
+  
+  Eclipse
+  Eclipse with Scala
+  Eclipse Java Only (How to 
skip Scala)
+
+  
+  Troubleshooting
+  Invalid cross-compiled 
libraries error
+  Incompatible Scala version 
error
+  Not found type error
+  Marketplace not 
found error for Eclipse Luna
+
+  
+
+
+IntelliJ
+
+IntelliJ can be used since it provides great support for mixed Java and 
Scala projects as described https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ;>here.
+
+Import SystemML project to 
IntelliJ
+
+
+  Download IntelliJ and install the Scala plug-in for IntelliJ.
+  Go to File - Import Project, locate the systemml 
source directory, and select Maven Project.
+  In the Import wizard, its fine to leave settings at their 
default. However it is usually useful to enable Import Maven projects 
automatically, since changes to the project structure will automatically 
update the IntelliJ project.
+
+
+Eclipse
+
+Eclipse https://eclipse.org/downloads/packages/release/luna/sr2;>Luna SR2 can 
be used for an integrated development environment with SystemML code.  Maven 
integration is required which is included in the https://eclipse.org/downloads/packages/eclipse-ide-java-developers/lunasr2;>Eclipse
 IDE for Java Developers package.
+
+To get started in Eclipse, import SystemMLs pom.xml file as an 
existing Maven