(systemds-website) branch asf-site updated: [MINOR] update contributors (#143) (#144)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/systemds-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 4e98e32b [MINOR] update contributors (#143) (#144) 4e98e32b is described below commit 4e98e32b293f21521047c99b26c0b85fd136d9b8 Author: Sebastian Baunsgaard AuthorDate: Sat Jun 8 21:24:52 2024 +0200 [MINOR] update contributors (#143) (#144) --- content/community.html | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/community.html b/content/community.html index 7dd4f44f..c0b0005d 100644 --- a/content/community.html +++ b/content/community.html @@ -218,7 +218,7 @@ PMC Member -TU Graz +TU Berlin @@ -518,7 +518,7 @@ PMC Member, Chair -TU Graz, previously IBM +TU Berlin, previously IBM @@ -578,7 +578,7 @@ PMC Member -TU Graz +ETH Zürich @@ -638,7 +638,7 @@ PMC Member -TU Graz +TU Berlin @@ -667,7 +667,7 @@ http://github.com/Shafaq-Siddiqi;>Shafaq Siddiqi -Committer +PMC Member TU Graz
(systemds-website) branch update-website deleted (was d39460fa)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch update-website in repository https://gitbox.apache.org/repos/asf/systemds-website.git was d39460fa [DOC] add contributor details The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(systemds-website) branch update-website created (now d39460fa)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch update-website in repository https://gitbox.apache.org/repos/asf/systemds-website.git at d39460fa [DOC] add contributor details No new revisions were added by this update.
(systemds-website) branch main updated (fb604df3 -> 68114e8f)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds-website.git from fb604df3 [DOC] add new member to community page add 68114e8f [MINOR] Update contributors No new revisions were added by this update. Summary of changes: _src/_data/contributors.yml | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-)
(systemds-website) branch asf-staging updated: [MINOR] update contributors (#143)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch asf-staging in repository https://gitbox.apache.org/repos/asf/systemds-website.git The following commit(s) were added to refs/heads/asf-staging by this push: new 8003cd1d [MINOR] update contributors (#143) 8003cd1d is described below commit 8003cd1d5b3f0f549ce7d18bbbd0403d850d0534 Author: Sebastian Baunsgaard AuthorDate: Sat Jun 8 21:16:54 2024 +0200 [MINOR] update contributors (#143) --- content/community.html | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/community.html b/content/community.html index 7dd4f44f..c0b0005d 100644 --- a/content/community.html +++ b/content/community.html @@ -218,7 +218,7 @@ PMC Member -TU Graz +TU Berlin @@ -518,7 +518,7 @@ PMC Member, Chair -TU Graz, previously IBM +TU Berlin, previously IBM @@ -578,7 +578,7 @@ PMC Member -TU Graz +ETH Zürich @@ -638,7 +638,7 @@ PMC Member -TU Graz +TU Berlin @@ -667,7 +667,7 @@ http://github.com/Shafaq-Siddiqi;>Shafaq Siddiqi -Committer +PMC Member TU Graz
(systemds) branch main updated: [MINOR ]Update CITATION
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 60ea6b378c [MINOR ]Update CITATION
60ea6b378c is described below
commit 60ea6b378c6358d9370178730f2ec0648853d4df
Author: Sebastian Baunsgaard
AuthorDate: Thu May 9 23:47:15 2024 +0200
[MINOR ]Update CITATION
There was an error in the citation file, with an extra space in the
reference name.
---
CITATION | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CITATION b/CITATION
index 34011b607d..57cb0f4d17 100644
--- a/CITATION
+++ b/CITATION
@@ -1,4 +1,4 @@
-@software{Apache SystemDS,
+@software{ApacheSystemDS,
author= {Apache SystemDS Development Team},
title = {{Apache SystemDS: An open source ML system for the end-to-end
data science lifecycle}},
url = {https://github.com/apache/systemds},
(systemds) branch main updated: [MINOR] Double Buffering longer than buffer arrays
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 08ce6bc1f5 [MINOR] Double Buffering longer than buffer arrays
08ce6bc1f5 is described below
commit 08ce6bc1f5da755b7c0d1bb6dce347ba28711263
Author: Sebastian Baunsgaard
AuthorDate: Tue Apr 16 10:51:51 2024 +0200
[MINOR] Double Buffering longer than buffer arrays
This commit fixes the double buffering of byte arrays
to handle cases where the byte arrays given are larger than the sizes
of the buffer.
Previous to this commit these arrays made the buffer crash, while
this commit fixes it to forward the buffers.
Also contained is a bit of documentation in the FastBufferedDataOutput.
Closes 2019
---
.../runtime/util/DoubleBufferingOutputStream.java | 67 ++
.../runtime/util/FastBufferedDataOutputStream.java | 32 +++
.../apache/sysds/runtime/util/LocalFileUtils.java | 21 +++
3 files changed, 62 insertions(+), 58 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/util/DoubleBufferingOutputStream.java
b/src/main/java/org/apache/sysds/runtime/util/DoubleBufferingOutputStream.java
index 16504e64ee..8d3dd7e994 100644
---
a/src/main/java/org/apache/sysds/runtime/util/DoubleBufferingOutputStream.java
+++
b/src/main/java/org/apache/sysds/runtime/util/DoubleBufferingOutputStream.java
@@ -16,13 +16,12 @@
* specific language governing permissions and limitations
* under the License.
*/
-
+
package org.apache.sysds.runtime.util;
import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
-import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
@@ -34,7 +33,7 @@ public class DoubleBufferingOutputStream extends
FilterOutputStream {
protected Future[] _locks;
protected byte[][] _buff;
private int _pos;
-
+
public DoubleBufferingOutputStream(OutputStream out) {
this(out, 2, 8192);
}
@@ -43,42 +42,52 @@ public class DoubleBufferingOutputStream extends
FilterOutputStream {
super(out);
if(size <= 0)
throw new IllegalArgumentException("Buffer size <= 0.");
- if( size%8 != 0 )
+ if(size % 8 != 0)
throw new IllegalArgumentException("Buffer size not a
multiple of 8.");
_buff = new byte[num][size];
_locks = new Future[num];
- for(int i=0; i= len) {
+ // copy the block into the buffer.
+ System.arraycopy(b, off, b_pos, 0, len);
+ // submit write request guaranteed to
be sequential since it is using a single thread.
+ _locks[_pos] = _pool.submit(() ->
writeBuffer(b_pos, 0, len));
+ // copy for asynchronous write because
b is reused higher up
+ }
+ else {
+ // The given byte array is longer than
the buffer.
+ // This means that the async buffer
would overflow and therefore not work.
+ // To avoid this we simply write the
given byte array without a buffer.
+ // This approach only works if the
caller adhere to not modify the byte array given
+ _locks[_pos] = _pool.submit(() ->
writeBuffer(b, off, len));
+ // get the task to reduce the risk (
and at least block the current thread)
+ // to avoid race conditions from
callers.
+ _locks[_pos].get();
+ }
+ _pos = (_pos + 1) % _buff.length;
}
}
catch(Exception ex) {
throw new IOException(ex);
}
}
-
- public void writeBuffer(byte[] b, int off, int len) {
+
+ private void writeBuffer(byte[] b, int off, int len) {
try {
out.write(b, off, len);
}
@@ -91,14 +100,14 @@ public class DoubleBufferingOutputStream extends
FilterOutputStream {
public void flush() throws IOException {
try {
synchronized(_buff) {
- for(int i=0; i<_buff.length; i++)
+
(systemds) branch main updated: [MINOR] Add a custom LongInt hashmap
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 5173aa072a [MINOR] Add a custom LongInt hashmap
5173aa072a is described below
commit 5173aa072a7fd2ebae6ef3ba1260801140da264c
Author: Sebastian Baunsgaard
AuthorDate: Tue Apr 16 13:49:08 2024 +0200
[MINOR] Add a custom LongInt hashmap
This commit adds a new longint hash map for efficient combining of
column groups. The commit does not enable the HashMap, but separate
it into a smaller self standing and tested commit.
Closes #2020
---
.../runtime/compress/utils/HashMapLongInt.java | 221 +
.../compress/util/HashMapLongIntTest.java | 84
2 files changed, 305 insertions(+)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/utils/HashMapLongInt.java
b/src/main/java/org/apache/sysds/runtime/compress/utils/HashMapLongInt.java
new file mode 100644
index 00..8379a06698
--- /dev/null
+++ b/src/main/java/org/apache/sysds/runtime/compress/utils/HashMapLongInt.java
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.runtime.compress.utils;
+
+import java.util.Arrays;
+import java.util.Iterator;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.sysds.runtime.compress.utils.HashMapLongInt.KV;
+
+public class HashMapLongInt implements Iterable {
+ protected static final Log LOG =
LogFactory.getLog(HashMapLongInt.class.getName());
+
+ protected long[][] keys;
+ protected int[][] values;
+ protected int size = 0;
+
+ public HashMapLongInt(int arrSize) {
+ keys = createKeys(arrSize);
+ values = createValues(arrSize);
+ }
+
+ public int size() {
+ return size;
+ }
+
+ /**
+* return -1 if there was no such key.
+*
+* @param key the key to add
+* @param value The value for that key.
+* @return -1 if there was no such key, otherwise the value
+*/
+ public int putIfAbsent(long key, int value) {
+ final int ix = hash(key);
+ if(keys[ix] == null)
+ return createBucket(ix, key, value);
+ else
+ return addToBucket(ix, key, value);
+ }
+
+ public int get(long key) {
+ final int ix = hash(key);
+ final long[] bucketKeys = keys[ix];
+ if(bucketKeys != null) {
+ for(int i = 0; i < bucketKeys.length; i++) {
+ if(bucketKeys[i] == key)
+ return values[ix][i];
+ }
+ }
+ return -1;
+ }
+
+ private int addToBucket(int ix, long key, int value) {
+ final long[] bucketKeys = keys[ix];
+ for(int i = 0; i < bucketKeys.length; i++) {
+ if(bucketKeys[i] == key)
+ return values[ix][i];
+ else if(bucketKeys[i] == -1) {
+ bucketKeys[i] = key;
+ values[ix][i] = value;
+ size++;
+ return -1;
+ }
+ }
+ return reallocateBucket(ix, key, value);
+ }
+
+ private int reallocateBucket(int ix, long key, int value) {
+ final long[] bucketKeys = keys[ix];
+ final int len = bucketKeys.length;
+
+ // there was no match in the bucket
+ // reallocate bucket.
+ long[] newBucketKeys = new long[len * 2];
+ int[] newBucketValues = new int[len * 2];
+ System.arraycopy(bucketKeys, 0, newBucketKeys, 0, len);
+ System.arraycopy(values[ix], 0, newBucketValues, 0, len);
+ Arrays.fill(newBucketKeys, len + 1,
(systemds) branch main updated: [SYSTEMDS-3426] Python NN Builtin (Affine,Relu)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new c61e54e924 [SYSTEMDS-3426] Python NN Builtin (Affine,Relu)
c61e54e924 is described below
commit c61e54e92429a1a138ae1221cec940ee95ecad08
Author: Duc Thai Vu
AuthorDate: Mon Apr 15 11:57:39 2024 +0200
[SYSTEMDS-3426] Python NN Builtin (Affine,Relu)
This commit adds the new interface for easy usage of our neural network
in python. The design take inspiration from other neural network frameworks.
This specific commit contains the building blocks of Affine and Relu.
Closes #1848
Closes #1929
Co-authored-by: Duc Thai Vu
Co-authored-by: Rahul Joshi
---
.../operator/algorithm/builtin/pageRank.py | 9 +-
src/main/python/systemds/operator/nn/__init__.py | 20 +++
src/main/python/systemds/operator/nn/affine.py | 114 ++
src/main/python/systemds/operator/nn/relu.py | 68 +
src/main/python/systemds/operator/nodes/source.py | 17 ++-
src/main/python/systemds/utils/helpers.py | 20 ++-
src/main/python/tests/nn/__init__.py | 20 +++
src/main/python/tests/nn/neural_network.py | 89 +++
src/main/python/tests/nn/test_affine.py| 163 +
src/main/python/tests/nn/test_neural_network.py| 94
src/main/python/tests/nn/test_relu.py | 105 +
11 files changed, 710 insertions(+), 9 deletions(-)
diff --git a/src/main/python/systemds/operator/algorithm/builtin/pageRank.py
b/src/main/python/systemds/operator/algorithm/builtin/pageRank.py
index 5e03e9dd93..d1f037b935 100644
--- a/src/main/python/systemds/operator/algorithm/builtin/pageRank.py
+++ b/src/main/python/systemds/operator/algorithm/builtin/pageRank.py
@@ -30,9 +30,6 @@ from systemds.utils.consts import VALID_INPUT_TYPES
def pageRank(G: Matrix,
- p: Matrix,
- e: Matrix,
- u: Matrix,
**kwargs: Dict[str, VALID_INPUT_TYPES]):
"""
DML builtin method for PageRank algorithm (power iterations)
@@ -41,14 +38,16 @@ def pageRank(G: Matrix,
:param G: Input Matrix
:param p: initial page rank vector (number of nodes), e.g., rand intialized
+default rand initialized with seed
:param e: additional customization, default vector of ones
-:param u: personalization vector (number of nodes)
+:param u: personalization vector (number of nodes), default vector of ones
:param alpha: teleport probability
:param max_iter: maximum number of iterations
+:param seed: seed for default rand initialization of page rank vector
:return: computed pagerank
"""
-params_dict = {'G': G, 'p': p, 'e': e, 'u': u}
+params_dict = {'G': G}
params_dict.update(kwargs)
return Matrix(G.sds_context,
'pageRank',
diff --git a/src/main/python/systemds/operator/nn/__init__.py
b/src/main/python/systemds/operator/nn/__init__.py
new file mode 100644
index 00..e66abb4646
--- /dev/null
+++ b/src/main/python/systemds/operator/nn/__init__.py
@@ -0,0 +1,20 @@
+# -
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# -
diff --git a/src/main/python/systemds/operator/nn/affine.py
b/src/main/python/systemds/operator/nn/affine.py
new file mode 100644
index 00..44c67d1eda
--- /dev/null
+++ b/src/main/python/systemds/operator/nn/affine.py
@@ -0,0 +1,114 @@
+# -
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the Li
(systemds) branch main updated: [MINOR] Update cocode algorithms for CLA
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 48de384bb3 [MINOR] Update cocode algorithms for CLA
48de384bb3 is described below
commit 48de384bb3dca3e63f35b654e907e9ecaf5d747c
Author: Sebastian Baunsgaard
AuthorDate: Tue Apr 9 20:16:50 2024 +0200
[MINOR] Update cocode algorithms for CLA
This commit adds a new memorizer that rely on an array in
the size of number of columns to compress, instead of a hashmap with all.
The memory footprint is the same, but the performance is very much
improved because it allows constant time deletion of all memorized
column groups that contains a combination with the given specific columns.
The technique first allocate an array in size number of columns
each index get its own hashmap. containing the columngroup associated with
it.
then when combining columnsgroups, the lowest index of all columns combined
determine which array index hash map to add the combined index into.
Once a combination is chosen, the buckets of the lowest index of each
column group combined is reset, and the combined columngroup is inserted.
The result is constant time O(1) deletion and insertion in the memorizer
---
.../runtime/compress/cocode/AColumnCoCoder.java| 7 +-
.../runtime/compress/cocode/CoCodeGreedy.java | 36 +++---
.../runtime/compress/cocode/CoCodeHybrid.java | 33 ++
.../runtime/compress/cocode/CoCodePriorityQue.java | 43 ++--
.../runtime/compress/cocode/CoCoderFactory.java| 23 +--
.../sysds/runtime/compress/cocode/ColIndexes.java | 4 +-
.../sysds/runtime/compress/cocode/Memorizer.java | 13 ++--
.../cocode/{Memorizer.java => MemorizerV2.java}| 53 ---
.../sysds/runtime/compress/estim/AComEst.java | 76 +-
.../compress/estim/CompressedSizeInfoColGroup.java | 21 ++
10 files changed, 196 insertions(+), 113 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/cocode/AColumnCoCoder.java
b/src/main/java/org/apache/sysds/runtime/compress/cocode/AColumnCoCoder.java
index fc13e16f65..cfe1b1b55e 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/cocode/AColumnCoCoder.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/cocode/AColumnCoCoder.java
@@ -26,6 +26,10 @@ import org.apache.sysds.runtime.compress.cost.ACostEstimate;
import org.apache.sysds.runtime.compress.estim.AComEst;
import org.apache.sysds.runtime.compress.estim.CompressedSizeInfo;
+/**
+ * Main abstract class for the co-coding of columns to combine different
compression statistics and calculate the
+ * combinations of columns
+ */
public abstract class AColumnCoCoder {
protected static final Log LOG =
LogFactory.getLog(AColumnCoCoder.class.getName());
@@ -34,8 +38,7 @@ public abstract class AColumnCoCoder {
protected final ACostEstimate _cest;
protected final CompressionSettings _cs;
- protected AColumnCoCoder(AComEst sizeEstimator, ACostEstimate
costEstimator,
- CompressionSettings cs) {
+ protected AColumnCoCoder(AComEst sizeEstimator, ACostEstimate
costEstimator, CompressionSettings cs) {
_sest = sizeEstimator;
_cest = costEstimator;
_cs = cs;
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/cocode/CoCodeGreedy.java
b/src/main/java/org/apache/sysds/runtime/compress/cocode/CoCodeGreedy.java
index d5d6c6936e..45f5654ab2 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/cocode/CoCodeGreedy.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/cocode/CoCodeGreedy.java
@@ -37,14 +37,14 @@ import org.apache.sysds.runtime.util.CommonThreadPool;
public class CoCodeGreedy extends AColumnCoCoder {
- private final Memorizer mem;
+ private final MemorizerV2 mem;
protected CoCodeGreedy(AComEst sizeEstimator, ACostEstimate
costEstimator, CompressionSettings cs) {
super(sizeEstimator, costEstimator, cs);
- mem = new Memorizer(sizeEstimator);
+ mem = new MemorizerV2(sizeEstimator,
sizeEstimator.getNumColumns());
}
- protected CoCodeGreedy(AComEst sizeEstimator, ACostEstimate
costEstimator, CompressionSettings cs, Memorizer mem) {
+ protected CoCodeGreedy(AComEst sizeEstimator, ACostEstimate
costEstimator, CompressionSettings cs, MemorizerV2 mem) {
super(sizeEstimator, costEstimator, cs);
this.mem = mem;
}
@@ -93,16 +93,22 @@ public class CoCodeGreedy extends AColumnCoCoder {
for(int j = i + 1; j < workSet.size();
j++) {
final ColIndexes c1 =
workSet
(systemds) branch main updated (91834886fe -> 34492851f5)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from 91834886fe [MINOR] CLA update Map To indexes add 34492851f5 [SYSTEMDS-3572] Thread pool ParFor name threads No new revisions were added by this update. Summary of changes: .../apache/sysds/conf/ConfigurationManager.java| 21 +- .../sysds/runtime/codegen/SpoofCellwise.java | 12 +- .../sysds/runtime/codegen/SpoofMultiAggregate.java | 6 +- .../sysds/runtime/codegen/SpoofOuterProduct.java | 22 ++- .../apache/sysds/runtime/codegen/SpoofRowwise.java | 11 +- .../compress/CompressedMatrixBlockFactory.java | 39 ++-- .../runtime/compress/cocode/CoCodeGreedy.java | 200 +-- .../runtime/compress/cocode/CoCodePriorityQue.java | 5 +- .../runtime/compress/colgroup/ColGroupFactory.java | 2 +- .../colgroup/scheme/CompressionScheme.java | 2 - .../sysds/runtime/compress/estim/AComEst.java | 7 +- .../runtime/compress/io/WriterCompressed.java | 2 +- .../runtime/compress/lib/CLALibBinaryCellOp.java | 20 +- .../sysds/runtime/compress/lib/CLALibCompAgg.java | 62 +++--- .../runtime/compress/lib/CLALibDecompress.java | 9 +- .../runtime/compress/lib/CLALibLeftMultBy.java | 38 ++-- .../sysds/runtime/compress/lib/CLALibScalar.java | 7 +- .../sysds/runtime/compress/lib/CLALibSlice.java| 5 +- .../sysds/runtime/compress/lib/CLALibTSMM.java | 22 +-- .../runtime/controlprogram/ParForProgramBlock.java | 35 ++-- .../context/SparkExecutionContext.java | 8 +- .../controlprogram/paramserv/LocalPSWorker.java| 35 ++-- .../runtime/controlprogram/paramserv/PSWorker.java | 6 - .../controlprogram/paramserv/SparkPSWorker.java| 3 - .../apache/sysds/runtime/data/LibTensorAgg.java| 6 +- .../frame/data/lib/FrameFromMatrixBlock.java | 7 +- .../frame/data/lib/FrameLibApplySchema.java| 1 - .../frame/data/lib/FrameLibDetectSchema.java | 6 +- .../frame/data/lib/MatrixBlockFromFrame.java | 1 - .../sysds/runtime/functionobjects/CTable.java | 55 +++--- .../runtime/io/FrameReaderBinaryBlockParallel.java | 15 +- .../sysds/runtime/io/FrameReaderJSONLParallel.java | 10 +- .../runtime/io/FrameReaderTextCSVParallel.java | 17 +- .../runtime/io/FrameReaderTextCellParallel.java| 11 +- .../runtime/io/FrameWriterBinaryBlockParallel.java | 16 +- .../sysds/runtime/io/FrameWriterJSONLParallel.java | 15 +- .../runtime/io/FrameWriterTextCSVParallel.java | 16 +- .../runtime/io/FrameWriterTextCellParallel.java| 16 +- .../sysds/runtime/io/ReaderHDF5Parallel.java | 31 ++- .../sysds/runtime/io/ReaderTextCSVParallel.java| 94 - .../sysds/runtime/io/ReaderTextCellParallel.java | 14 +- .../sysds/runtime/io/ReaderTextLIBSVMParallel.java | 20 +- .../io/TensorReaderBinaryBlockParallel.java| 15 +- .../runtime/io/TensorReaderTextCellParallel.java | 12 +- .../io/TensorWriterBinaryBlockParallel.java| 25 ++- .../runtime/io/TensorWriterTextCellParallel.java | 24 ++- .../sysds/runtime/io/WriterHDF5Parallel.java | 25 ++- .../runtime/io/WriterMatrixMarketParallel.java | 16 +- .../sysds/runtime/io/WriterTextCSVParallel.java| 16 +- .../sysds/runtime/io/WriterTextCellParallel.java | 17 +- .../sysds/runtime/io/WriterTextLIBSVMParallel.java | 16 +- .../sysds/runtime/iogen/FormatIdentifyer.java | 32 ++-- .../apache/sysds/runtime/iogen/ReaderMapping.java | 30 ++- .../sysds/runtime/iogen/ReaderMappingIndex.java| 30 ++- .../template/FrameGenerateReaderParallel.java | 20 +- .../template/MatrixGenerateReaderParallel.java | 8 +- .../sysds/runtime/matrix/data/LibMatrixAgg.java| 37 ++-- .../runtime/matrix/data/LibMatrixBincell.java | 18 +- .../sysds/runtime/matrix/data/LibMatrixDNN.java| 11 +- .../runtime/matrix/data/LibMatrixDatagen.java | 12 +- .../runtime/matrix/data/LibMatrixFourier.java | 8 +- .../sysds/runtime/matrix/data/LibMatrixMult.java | 83 .../sysds/runtime/matrix/data/LibMatrixReorg.java | 212 +++-- .../runtime/matrix/data/LibMatrixTercell.java | 6 +- .../sysds/runtime/matrix/data/MatrixBlock.java | 23 ++- .../transform/encode/MultiColumnEncoder.java | 17 +- .../runtime/transform/tokenize/Tokenizer.java | 14 +- .../sysds/runtime/util/CommonThreadPool.java | 141 ++ .../apache/sysds/runtime/util/LocalFileUtils.java | 2 + .../sysds/performance/micro/InformationLoss.java | 47 +++-- .../org/apache/sysds/test/AutomatedTestBase.java | 19 +- .../test/component/compress/AsyncCompressTest.java | 19 +- .../sysds/test/component/misc/ThreadPool.java | 94 + .../jmlc/JMLCClonedPreparedScriptTest.java | 6 +- .../sysds/test/util
(systemds) branch main updated (8bda7c92a0 -> 91834886fe)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from 8bda7c92a0 [MINOR] Optimize contains any Single Index add 91834886fe [MINOR] CLA update Map To indexes No new revisions were added by this update. Summary of changes: .../compress/colgroup/mapping/AMapToData.java | 52 +- .../compress/colgroup/mapping/MapToBit.java| 12 .../compress/colgroup/mapping/MapToByte.java | 63 + .../compress/colgroup/mapping/MapToChar.java | 63 - .../compress/colgroup/mapping/MapToCharPByte.java | 64 +- .../compress/colgroup/mapping/MapToInt.java| 28 +- .../compress/colgroup/mapping/MapToUByte.java | 39 + 7 files changed, 316 insertions(+), 5 deletions(-)
(systemds) branch main updated: [MINOR] Optimize contains any Single Index
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 8bda7c92a0 [MINOR] Optimize contains any Single Index
8bda7c92a0 is described below
commit 8bda7c92a06f288bb2fe32583b23ac5633d2f61d
Author: Sebastian Baunsgaard
AuthorDate: Sat Apr 6 17:32:29 2024 +0200
[MINOR] Optimize contains any Single Index
---
.../sysds/runtime/compress/colgroup/indexes/ArrayIndex.java| 10 ++
.../sysds/runtime/compress/colgroup/indexes/SingleIndex.java | 9 +
2 files changed, 19 insertions(+)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ArrayIndex.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ArrayIndex.java
index 0c0693d53c..57cd08fb01 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ArrayIndex.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ArrayIndex.java
@@ -45,6 +45,16 @@ public class ArrayIndex extends AColIndex {
return cols[i];
}
+ /**
+* For performance reasons we can extract the array. Be careful when
you do.
+*
+* @return The internal array.
+*/
+ public int[] getArray() {
+ // For performance reasons available
+ return cols;
+ }
+
@Override
public IColIndex shift(int i) {
int[] ret = new int[cols.length];
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/SingleIndex.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/SingleIndex.java
index 2b14ecc3e7..3c149512fe 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/SingleIndex.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/SingleIndex.java
@@ -138,6 +138,15 @@ public class SingleIndex extends AColIndex {
return idx;
}
+
+ @Override
+ public boolean containsAny(IColIndex idx) {
+ if(idx instanceof SingleIndex)
+ return this.idx == idx.get(0);
+ else// turn around the logic.
+ return idx.contains(this.idx);
+ }
+
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
(systemds) branch main updated: [MINOR] Fix SYSDS_QUIET
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 69d55cd9d7 [MINOR] Fix SYSDS_QUIET 69d55cd9d7 is described below commit 69d55cd9d73884303ce983d29606eae574f2964e Author: Sebastian Baunsgaard AuthorDate: Fri Apr 5 23:19:21 2024 +0200 [MINOR] Fix SYSDS_QUIET --- bin/systemds | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bin/systemds b/bin/systemds index ffad4b42c6..2e8e629495 100755 --- a/bin/systemds +++ b/bin/systemds @@ -397,7 +397,7 @@ if [ $PRINT_SYSDS_HELP == 1 ]; then exit 1 fi -if [ $SYSDS_QUIET != 0 ]; then +if [ $SYSDS_QUIET == 0 ]; then print_out "###" print_out "# SYSTEMDS_ROOT= $SYSTEMDS_ROOT" print_out "# SYSTEMDS_JAR_FILE= $SYSTEMDS_JAR_FILE" @@ -449,7 +449,7 @@ else $*" fi -if [ $SYSDS_QUIET != 0 ]; then +if [ $SYSDS_QUIET == 0 ]; then print_out "# Executing command: $CMD" print_out "###" fi
(systemds) branch main updated: [SYSTEMDS-3676] Relative path remove in bin
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new ecb53edea0 [SYSTEMDS-3676] Relative path remove in bin
ecb53edea0 is described below
commit ecb53edea03a8175f5077b0758503979f2921646
Author: Sebastian Baunsgaard
AuthorDate: Fri Apr 5 16:07:41 2024 +0200
[SYSTEMDS-3676] Relative path remove in bin
This commit fixes the MacOS startup using the systemds bin script.
In the process of fixing it, the commit also imrpove the startup
overhead from 75 ms average to 20 ms average when using the
bin/systemds script.
The speedup comes from skipping to search for jar file, configuration, and
logging files if they are located in default positions inside conf, target,
root or .
Closes #2012
---
bin/systemds | 148 ++-
1 file changed, 56 insertions(+), 92 deletions(-)
diff --git a/bin/systemds b/bin/systemds
index 35ff10ab26..ffad4b42c6 100755
--- a/bin/systemds
+++ b/bin/systemds
@@ -20,14 +20,6 @@
#
#-
-##
-# This script is part of the SystemDS binary release. It is
-# meant to work out of the box when unzipping the
-# systemds-.zip (or tbz) file.
-#
-# Make configuration changes here:
-##
-
# If not set by env, set to 1 to run spark-submit instead of local java
# This should be used to run with spark-submit instead of java
if [[ -z "$SYSDS_DISTRIBUTED" ]]; then
@@ -56,11 +48,8 @@ print_out()
}
if [[ -z $SYSTEMDS_ROOT ]] ; then
- SYSTEMDS_ROOT=.
+ SYSTEMDS_ROOT=$(pwd)
print_out "SYSTEMDS_ROOT not set defaulting to current dir $(pwd)"
-else
- # construct a relative path
- SYSTEMDS_ROOT=$(realpath --relative-to=. ${SYSTEMDS_ROOT})
fi;
# when using find, look in the directories in this order
@@ -95,24 +84,21 @@ fi
# check if log4j config file exists, otherwise unset
# to run with a non fatal complaint by SystemDS
if [ -z "$LOG4JPROP" ] ; then
- LOG4JPROP=$(ordered_find "log4j*properties")
-
- if [ -z "${LOG4JPROP}" ]; then
-LOG4JPROP=""
- else
-LOG4JPROPFULL="-Dlog4j.configuration=file:$LOG4JPROP"
- fi
-else
- # L4J was set by env var. Unset if that setting is wrong
- LOG4JPROP2=$(find "$LOG4JPROP")
- if [ -z "${LOG4JPROP2}" ]; then
-LOG4JPROP=""
- else
-LOG4JPROP=$LOG4JPROP
-LOG4JPROPFULL="-Dlog4j.configuration=file:$LOG4JPROP2"
+ # before wild card search look obvious places.
+ if [ -f "$SYSTEMDS_ROOT/conf/log4j.properties" ]; then
+LOG4JPROP="$SYSTEMDS_ROOT/conf/log4j.properties"
+ elif [ -f "$SYSTEMDS_ROOT/log4j.properties" ]; then
+LOG4JPROP="$SYSTEMDS_ROOT/log4j.properties"
+ else # wildcard search
+LOG4JPROP=$(ordered_find "log4j*properties")
fi
fi
+# If the LOG4J variable is declared or found.
+if [ -f "${LOG4JPROP}" ]; then
+ LOG4JPROPFULL="-Dlog4j.configuration=file:$LOG4JPROP"
+fi
+
if [ -n "${SYSTEMDS_DISTRIBUTED_OPTS}" ]; then
print_out "Overriding SYSTEMDS_DISTRIBUTED_OPTS with env var
$SYSTEMDS_DISTRIBUTED_OPTS"
else
@@ -132,17 +118,7 @@ else
fi
-##
-# No need to touch the content below. These commands launch
-# SystemDS based on the settings above.
-##
-
-
-#-
-# some helper functions
-
# error help print
-PRINT_SYSDS_HELP=0
function printUsage {
cat << EOF
@@ -180,9 +156,6 @@ local java Set SYSDS_QUIET=1 to omit extra information
printed by this run
script.
EOF
-if [ ${PRINT_SYSDS_HELP} -eq 0 ]; then
- exit 0
-fi
}
# print an error if no argument is supplied.
@@ -190,16 +163,18 @@ if [ -z "$1" ] ; then
echo "Wrong Usage. Add -help for additional parameters.";
echo ""
printUsage;
+exit -1
fi
#This loop handles the parameters to the run-script, not the ones passed to
SystemDS.
#To not confuse getopts with SystemDS parameters, only the first two params
are considered
#here. If more run-script params are needed, adjust the next line accordingly
+PRINT_SYSDS_HELP=0
while getopts ":hr:f:" options "$1$2"; do
case $options in
h ) echo "Help requested. Will exit after extended usage message!"
-PRINT_SYSDS_HELP=1
printUsage
+PRINT_SYSDS_HELP=1
break
;;
\? ) echo "Unknown parameter -$OPTARG"
(systemds) branch main updated: [MINOR] Frame Shallow Update
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 1fa2ebc7ba [MINOR] Frame Shallow Update
1fa2ebc7ba is described below
commit 1fa2ebc7bad9e6bb8006f70c8ae01a00cde74d5d
Author: Sebastian Baunsgaard
AuthorDate: Fri Apr 5 17:01:47 2024 +0200
[MINOR] Frame Shallow Update
This commit make minor modifications to the
shallow handling of Frames.
One instance is fast abort of isShallowSerialize.
Closes #2013
---
.../sysds/runtime/frame/data/FrameBlock.java | 61 --
1 file changed, 32 insertions(+), 29 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java
b/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java
index 3efafbb30b..312f88ca7d 100644
--- a/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java
@@ -106,6 +106,7 @@ public class FrameBlock implements CacheBlock,
Externalizable {
/** Locks on the columns not tied to the columns objects. */
private SoftReference _columnLocks = null;
+ /** Materialized number of rows in this FrameBlock */
private int _nRow = 0;
/** Cached size in memory to avoid repeated scans of string columns */
@@ -756,7 +757,8 @@ public class FrameBlock implements CacheBlock,
Externalizable {
public void write(DataOutput out) throws IOException {
final boolean isDefaultMeta = isColNamesDefault() &&
isColumnMetadataDefault();
// write header (rows, cols, default)
- out.writeInt(getNumRows());
+ final int nRow = getNumRows();
+ out.writeInt(nRow);
out.writeInt(getNumColumns());
out.writeBoolean(isDefaultMeta);
// write columns (value type, data)
@@ -767,7 +769,7 @@ public class FrameBlock implements CacheBlock,
Externalizable {
out.writeUTF(getColumnName(j));
_colmeta[j].write(out);
}
- if(type >= 0) // if allocated write column data
+ if(type >= 0 && nRow > 0) // if allocated write column
data
_coldata[j].write(out);
}
}
@@ -796,6 +798,8 @@ public class FrameBlock implements CacheBlock,
Externalizable {
isDefaultMeta ? null : new String[numCols]; // if meta
is default allocate on demand
_colmeta = (_colmeta != null && _colmeta.length == numCols) ?
_colmeta : new ColumnMetadata[numCols];
_coldata = (_coldata != null && _coldata.length == numCols) ?
_coldata : new Array[numCols];
+ if(_nRow == 0)
+ _coldata = null;
// read columns (value type, meta, data)
for(int j = 0; j < numCols; j++) {
byte type = in.readByte();
@@ -807,7 +811,7 @@ public class FrameBlock implements CacheBlock,
Externalizable {
else
_colmeta[j] = new ColumnMetadata(); // must be
allocated.
- if(type >= 0) // if in allocated column data then read
it
+ if(type >= 0 && _nRow > 0) // if in allocated column
data then read it
_coldata[j] = ArrayFactory.read(in, _nRow);
}
_msize = -1;
@@ -815,30 +819,12 @@ public class FrameBlock implements
CacheBlock, Externalizable {
@Override
public void writeExternal(ObjectOutput out) throws IOException {
-
- // if((out instanceof ObjectOutputStream)){
- // ObjectOutputStream oos = (ObjectOutputStream)out;
- // FastBufferedDataOutputStream fos = new
FastBufferedDataOutputStream(oos);
- // write(fos); //note: cannot close fos as this would
close oos
- // fos.flush();
- // }
- // else{
- write(out);
- // }
+ write(out);
}
@Override
public void readExternal(ObjectInput in) throws IOException {
- // if(in instanceof ObjectInputStream) {
- // // fast deserialize of dense/sparse blocks
- // ObjectInputStream ois = (ObjectInputStream) in;
- // FastBufferedDataInputStream fis = new
FastBufferedDataInputStream(ois);
- // readFields(fis); // note: cannot close fos as this
would close oos
- // }
- // else {
-
(systemds) branch main updated: [MINOR] Fix compression statistic logging for frames
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 6b23ea4227 [MINOR] Fix compression statistic logging for frames
6b23ea4227 is described below
commit 6b23ea4227127dd8bb9f071453de59ddf518b226
Author: Sebastian Baunsgaard
AuthorDate: Fri Apr 5 17:07:54 2024 +0200
[MINOR] Fix compression statistic logging for frames
Logging of frames statistics for compression is misleading when
samples are used to estimate the number of elements.
Therefore this commit change the logging message to reflect the
approximate nature of distinct counts
---
.../frame/data/compress/ArrayCompressionStatistics.java| 10 --
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/compress/ArrayCompressionStatistics.java
b/src/main/java/org/apache/sysds/runtime/frame/data/compress/ArrayCompressionStatistics.java
index 8323060f81..c9d5dc71e8 100644
---
a/src/main/java/org/apache/sysds/runtime/frame/data/compress/ArrayCompressionStatistics.java
+++
b/src/main/java/org/apache/sysds/runtime/frame/data/compress/ArrayCompressionStatistics.java
@@ -20,6 +20,8 @@
package org.apache.sysds.runtime.frame.data.compress;
import org.apache.sysds.common.Types.ValueType;
+import org.apache.sysds.conf.ConfigurationManager;
+import org.apache.sysds.conf.DMLConfig;
import org.apache.sysds.runtime.frame.data.columns.ArrayFactory.FrameArrayType;
public class ArrayCompressionStatistics {
@@ -48,8 +50,12 @@ public class ArrayCompressionStatistics {
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
- sb.append(String.format("Compressed Stats: size:%8d->%8d,
Use:%10s, Unique:%6d, ValueType:%7s", originalSize,
- compressedSizeEstimate, bestType == null ? "None" :
bestType.toString(), nUnique, valueType));
+
if(ConfigurationManager.getDMLConfig().getDoubleValue(DMLConfig.COMPRESSED_SAMPLING_RATIO)
< 1)
+ sb.append(String.format("Compressed Stats:
size:%8d->%8d, Use:%10s, EstUnique:%6d, ValueType:%7s",
+ originalSize, compressedSizeEstimate, bestType
== null ? "None" : bestType.toString(), nUnique, valueType));
+ else
+ sb.append(String.format("Compressed Stats:
size:%8d->%8d, Use:%10s, Unique:%6d, ValueType:%7s", originalSize,
+ compressedSizeEstimate, bestType == null ?
"None" : bestType.toString(), nUnique, valueType));
return sb.toString();
}
}
(systemds) branch main updated: [MINOR] Add general contains and specific contains nan on DenseBlock
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 3e6e462854 [MINOR] Add general contains and specific contains nan on
DenseBlock
3e6e462854 is described below
commit 3e6e462854b1818893e86443aae858ae1cfc1088
Author: Sebastian Baunsgaard
AuthorDate: Fri Apr 5 16:58:37 2024 +0200
[MINOR] Add general contains and specific contains nan on DenseBlock
---
.../org/apache/sysds/runtime/data/DenseBlock.java | 34 --
.../apache/sysds/runtime/data/DenseBlockBool.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockFP32.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockFP64.java | 2 +-
.../sysds/runtime/data/DenseBlockFP64DEDUP.java| 2 +-
.../apache/sysds/runtime/data/DenseBlockInt32.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockInt64.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockLBool.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockLFP32.java | 2 +-
.../apache/sysds/runtime/data/DenseBlockLFP64.java | 2 +-
.../sysds/runtime/data/DenseBlockLFP64DEDUP.java | 2 +-
.../sysds/runtime/data/DenseBlockLInt32.java | 2 +-
.../sysds/runtime/data/DenseBlockLInt64.java | 2 +-
.../sysds/runtime/data/DenseBlockLString.java | 2 +-
.../sysds/runtime/data/DenseBlockString.java | 2 +-
15 files changed, 46 insertions(+), 16 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
b/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
index 0a30d79250..0baf881936 100644
--- a/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
@@ -67,6 +67,8 @@ public abstract class DenseBlock implements Serializable,
Block
/**
* Get the ith dimensions size of the dense block.
+*
+* 0 is rows , 1 is cols, etc.
*
* @param i the number of dimension to get
* @return the size of the dimension
@@ -414,7 +416,7 @@ public abstract class DenseBlock implements Serializable,
Block
* @param toIndex ending index in block (exclusive)
* @param v value
*/
- protected abstract void fillBlock(int bix, int fromIndex, int toIndex,
double v);
+ public abstract void fillBlock(int bix, int fromIndex, int toIndex,
double v);
/**
* Set a value at a position given by block index and index in that
block.
@@ -669,14 +671,42 @@ public abstract class DenseBlock implements Serializable,
Block
* @param ru row upper bound (exclusive)
* @return true if pattern appears at least once, otherwise false
*/
+
public boolean contains(double pattern, int rl, int ru) {
boolean NaNpattern = Double.isNaN(pattern);
int clen = _odims[0];
for(int i=rl; i
(systemds) branch main updated: [MINOR] Overwrite toString on Timing objects
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new e383dbf130 [MINOR] Overwrite toString on Timing objects
e383dbf130 is described below
commit e383dbf130211ac41203e3b048874f9a511fbb7d
Author: Sebastian Baunsgaard
AuthorDate: Fri Apr 5 16:54:16 2024 +0200
[MINOR] Overwrite toString on Timing objects
For ease of use, overwrite the timing object to print the time
observed if printed.
---
.../org/apache/sysds/runtime/controlprogram/parfor/stat/Timing.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/stat/Timing.java
b/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/stat/Timing.java
index ae971e3e4e..6b38a98334 100644
---
a/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/stat/Timing.java
+++
b/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/stat/Timing.java
@@ -76,4 +76,9 @@ public class Timing {
double tmp = stop();
System.out.println("PARFOR: time = " + tmp + "ms");
}
+
+ @Override
+ public String toString(){
+ return "Timing: " + stop();
+ }
}
(systemds) branch main updated (91005840bd -> 387b6c1c8e)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
from 91005840bd [MINOR] Github Actions Isolate Flaky Runs
add 387b6c1c8e [SYSTEMDS-3687] Python API startup fixes
No new revisions were added by this update.
Summary of changes:
.gitignore | 1 -
src/main/python/.gitignore | 7 +++
{conf => src/main/python/conf}/log4j.properties| 0
src/main/python/pre_setup.py | 10
.../python/systemds/context/systemds_context.py| 61 +++---
5 files changed, 59 insertions(+), 20 deletions(-)
copy {conf => src/main/python/conf}/log4j.properties (100%)
(systemds) branch main updated: [MINOR] Github Actions Isolate Flaky Runs
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 91005840bd [MINOR] Github Actions Isolate Flaky Runs 91005840bd is described below commit 91005840bdba484224c2066e434bc01642d33513 Author: Sebastian Baunsgaard AuthorDate: Thu Apr 4 18:55:45 2024 +0200 [MINOR] Github Actions Isolate Flaky Runs --- .github/workflows/javaTests.yml | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/javaTests.yml b/.github/workflows/javaTests.yml index 5589269a99..db2511d1a6 100644 --- a/.github/workflows/javaTests.yml +++ b/.github/workflows/javaTests.yml @@ -76,12 +76,14 @@ jobs: "**.functions.frame.**,**.functions.indexing.**,**.functions.io.**,**.functions.iogen.**", "**.functions.dnn.**", "**.functions.paramserv.**", - "**.functions.recompile.**,**.functions.misc.**,**.functions.mlcontext.**", + "**.functions.recompile.**,**.functions.misc.**", + "**.functions.mlcontext.**", "**.functions.nary.**,**.functions.quaternary.**", "**.functions.parfor.**,**.functions.pipelines.**", "**.functions.homomorphicEncryption.**", "**.functions.unary.scalar.**,**.functions.updateinplace.**,**.functions.vect.**", - "**.functions.reorg.**,**.functions.rewrite.**,**.functions.ternary.**,**.functions.transform.**", + "**.functions.reorg.**,**.functions.rewrite.**,**.functions.ternary.**", + "**.functions.transform.**", "**.functions.unary.matrix.**,**.functions.linearization.**,**.functions.jmlc.**" ] java: [11]
(systemds) 04/05: [SYSTEMDS-3685] FFT parallel, including other builtin functioncalls
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 3f166c03bcebce7c95a0d4fb82c0f526939f4fc1
Author: Sebastian Baunsgaard
AuthorDate: Thu Apr 4 17:18:24 2024 +0200
[SYSTEMDS-3685] FFT parallel, including other builtin functioncalls
This commit enable the compile time propatation of the parallelization
degree to the new FFT instructions.
---
.../java/org/apache/sysds/hops/FunctionOp.java | 21 +++-
src/main/java/org/apache/sysds/hops/Hop.java | 8 +-
src/main/java/org/apache/sysds/hops/UnaryOp.java | 2 +-
.../java/org/apache/sysds/lops/Compression.java| 10 +-
.../java/org/apache/sysds/lops/FunctionCallCP.java | 48 +---
.../cp/AggregateUnaryCPInstruction.java| 6 +-
.../instructions/cp/CompressionCPInstruction.java | 45 ---
.../runtime/instructions/cp/DnnCPInstruction.java | 13 +-
.../cp/MultiReturnBuiltinCPInstruction.java| 68 ++-
...ltiReturnComplexMatrixBuiltinCPInstruction.java | 44 ---
.../sysds/runtime/matrix/data/LibCommonsMath.java | 133 +++--
.../runtime/matrix/data/LibMatrixFourier.java | 100 +++-
.../python/systemds/operator/algorithm/__init__.py | 2 +
.../operator/algorithm/builtin/pageRank.py | 55 +
src/main/python/tests/lineage/test_lineagetrace.py | 36 --
.../applications/ScalableDecompositionTest.java| 4 +-
.../sysds/test/component/matrix/FourierTest.java | 94 +--
.../scripts/functions/builtin/GridSearchLMCV.dml | 3 +-
18 files changed, 449 insertions(+), 243 deletions(-)
diff --git a/src/main/java/org/apache/sysds/hops/FunctionOp.java
b/src/main/java/org/apache/sysds/hops/FunctionOp.java
index 95b5411500..7f424d36d0 100644
--- a/src/main/java/org/apache/sysds/hops/FunctionOp.java
+++ b/src/main/java/org/apache/sysds/hops/FunctionOp.java
@@ -42,7 +42,7 @@ import org.apache.sysds.runtime.meta.DataCharacteristics;
* Note: Currently, we support expressions in function arguments along with
function calls
* in expressions with single outputs, leaving multiple outputs handling as it
is.
*/
-public class FunctionOp extends Hop
+public class FunctionOp extends MultiThreadedHop
{
public enum FunctionType{
DML,
@@ -342,7 +342,14 @@ public class FunctionOp extends Hop
tmp.add( in.constructLops() );
//construct function call
- FunctionCallCP fcall = new FunctionCallCP(tmp, _fnamespace,
_fname, _inputNames, _outputNames, _outputHops, _opt, et);
+ final FunctionCallCP fcall;
+ if(isMultiThreadedOpType()) {
+ fcall = new FunctionCallCP(tmp, _fnamespace, _fname,
_inputNames, _outputNames, _outputHops, _opt, et,
+
OptimizerUtils.getConstrainedNumThreads(_maxNumThreads));
+ }
+ else {
+ fcall = new FunctionCallCP(tmp, _fnamespace, _fname,
_inputNames, _outputNames, _outputHops, _opt, et);
+ }
setLineNumbers(fcall);
setLops(fcall);
@@ -358,13 +365,14 @@ public class FunctionOp extends Hop
// Lop matrixOut = lop.getFunctionOutputs().get(0);
Lop compressionInstruction = null;
+ final int k =
OptimizerUtils.getConstrainedNumThreads(_maxNumThreads);
if(_compressedWorkloadTree != null) {
SingletonLookupHashMap m =
SingletonLookupHashMap.getMap();
int singletonID =
m.put(_compressedWorkloadTree);
- compressionInstruction = new
Compression(getLops(), DataType.MATRIX, ValueType.FP64, et, singletonID);
+ compressionInstruction = new
Compression(getLops(), DataType.MATRIX, ValueType.FP64, et, singletonID, k);
}
else
- compressionInstruction = new
Compression(getLops(), DataType.MATRIX, ValueType.FP64, et, 0);
+ compressionInstruction = new
Compression(getLops(), DataType.MATRIX, ValueType.FP64, et, 0, k);
setOutputDimensions( compressionInstruction );
@@ -427,6 +435,11 @@ public class FunctionOp extends Hop
public void refreshSizeInformation() {
//do nothing
}
+
+ @Override
+ public boolean isMultiThreadedOpType() {
+ return isBuiltinFunction();
+ }
@Override
@SuppressWarnings("unchecked")
diff --git a/src/main/java/org/apache/sysds/hops/Hop.java
b/src/main/java/org/apache/sysds/hops/Hop.java
index 127fe7e145..93501ef
(systemds) 05/05: [MINOR] refine the selection of jar file for bin script
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 44fbc5af83fa835a0b89f688d3874568231a8ea0
Author: Sebastian Baunsgaard
AuthorDate: Thu Apr 4 17:18:31 2024 +0200
[MINOR] refine the selection of jar file for bin script
---
bin/systemds | 36 +-
src/main/python/tests/lineage/test_lineagetrace.py | 3 --
2 files changed, 21 insertions(+), 18 deletions(-)
diff --git a/bin/systemds b/bin/systemds
index 65f2a82867..35ff10ab26 100755
--- a/bin/systemds
+++ b/bin/systemds
@@ -64,7 +64,7 @@ else
fi;
# when using find, look in the directories in this order
-DIR_SEARCH_ORDER=". $SYSTEMDS_ROOT $SYSTEMDS_ROOT/conf $SYSTEMDS_ROOT/lib
$SYSTEMDS_ROOT/src $SYSTEMDS_ROOT/target"
+DIR_SEARCH_ORDER="$SYSTEMDS_ROOT/target . $SYSTEMDS_ROOT $SYSTEMDS_ROOT/conf
$SYSTEMDS_ROOT/lib $SYSTEMDS_ROOT/src"
ordered_find() {
result=""
for dir in $(echo "$DIR_SEARCH_ORDER" | tr ' ' '\n') ; do
@@ -292,17 +292,30 @@ if [ -z "$FEDMONITORING" ] ; then
FEDMONITORING=0
fi
-# find me a SystemDS jar file to run
-if [ -z "$SYSTEMDS_JAR_FILE" ];then
+# find a SystemDS jar file to run
+if [ -z ${SYSTEMDS_JAR_FILE+x} ]; then # If it is not found yet.
+ if [ ! -z ${SYSTEMDS_ROOT+x} ]; then # Check currently set SYSETMDS_ROOT
+# Current SYSTEMDS_ROOT is set and is a directory.
+if [ -d "$SYSTEMDS_ROOT/target" ] && [ -d "$SYSTEMDS_ROOT/.git" ]; then
+ # Current path is most likely a build directory of systemds
+ SYSTEMDS_JAR_FILE=$(ordered_find "systemds-?.?.?-SNAPSHOT.jar")
+fi
+ fi
+fi
+
+# If no jar file is found, start searching
+if [ -z ${SYSTEMDS_JAR_FILE+x} ]; then
SYSTEMDS_JAR_FILE=$(ordered_find "systemds.jar")
- if [ -z "$SYSTEMDS_JAR_FILE" ];then
+ if [ -z ${SYSTEMDS_JAR_FILE+x} ]; then
SYSTEMDS_JAR_FILE=$(ordered_find "systemds-?.?.?.jar")
-if [ -z "$SYSTEMDS_JAR_FILE" ];then
+if [ -z ${SYSTEMDS_JAR_FILE+x} ]; then
SYSTEMDS_JAR_FILE=$(ordered_find "systemds-?.?.?-SNAPSHOT.jar")
+ if [ -z ${SYSTEMDS_JAR_FILE+x} ]; then
+echo "wARNING: Unable to find SystemDS jar file to launch"
+exit -1
+ fi
fi
fi
-else
- print_out "Using user supplied systemds jar file $SYSTEMDS_JAR_FILE"
fi
if [[ "$*" == *-config* ]]; then
@@ -402,17 +415,11 @@
NATIVE_LIBS="$SYSTEMDS_ROOT${DIR_SEP}target${DIR_SEP}classes${DIR_SEP}lib"
export PATH=${HADOOP_REL}${DIR_SEP}bin${PATH_SEP}${PATH}${PATH_SEP}$NATIVE_LIBS
export LD_LIBRARY_PATH=${HADOOP_REL}${DIR_SEP}bin${PATH_SEP}${LD_LIBRARY_PATH}
-# set java class path
-CLASSPATH="${SYSTEMDS_JAR_FILE}${PATH_SEP} \
- ${SYSTEMDS_ROOT}${DIR_SEP}lib${DIR_SEP}*${PATH_SEP} \
- ${SYSTEMDS_ROOT}${DIR_SEP}target${DIR_SEP}lib${DIR_SEP}*"
-# trim whitespace (introduced by the line breaks above)
-CLASSPATH=$(echo "${CLASSPATH}" | tr -d '[:space:]')
if [ $PRINT_SYSDS_HELP == 1 ]; then
echo "--"
echo "Further help on SystemDS arguments:"
- java -cp "$CLASSPATH" org.apache.sysds.api.DMLScript -help
+ java -jar $SYSTEMDS_JAR_FILE org.apache.sysds.api.DMLScript -help
exit 1
fi
@@ -422,7 +429,6 @@ print_out "# SYSTEMDS_JAR_FILE= $SYSTEMDS_JAR_FILE"
print_out "# SYSDS_EXEC_MODE= $SYSDS_EXEC_MODE"
print_out "# CONFIG_FILE= $CONFIG_FILE"
print_out "# LOG4JPROP= $LOG4JPROP"
-print_out "# CLASSPATH= $CLASSPATH"
print_out "# HADOOP_HOME= $HADOOP_HOME"
#build the command to run
diff --git a/src/main/python/tests/lineage/test_lineagetrace.py
b/src/main/python/tests/lineage/test_lineagetrace.py
index 7e4e4bb3b1..d8c325d8f3 100644
--- a/src/main/python/tests/lineage/test_lineagetrace.py
+++ b/src/main/python/tests/lineage/test_lineagetrace.py
@@ -75,8 +75,6 @@ class TestLineageTrace(unittest.TestCase):
# Call SYSDS!
result_file_name = temp_dir + "/tmp_res.txt"
-os.environ["SYSDS_QUIET"] = "0"
-os.system("which systemds")
command = "systemds " + script + \
" > " + result_file_name + " 2> /dev/null"
status = os.system(command)
@@ -89,7 +87,6 @@ def parse_trace(path: str):
data = []
with open(path, "r") as log:
for line in log:
-print(line)
if "°" in line:
data.append(line.strip().split("°"))
(systemds) 01/05: [SYSTEMDS-3685] DML Integration of FFT and IFFT
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 52639b2e57683cf8c9ab6ecd26c5a961100f2d79
Author: Jessica Eva Sophie Priebe
AuthorDate: Thu Apr 4 17:17:07 2024 +0200
[SYSTEMDS-3685] DML Integration of FFT and IFFT
This commits integrate 4 new builtin functions:
FFT
FFT_LINEARIZED
IFFT
IFFT_LINEARIZED
The functions implement fast fourier transformations and inverces.
The linearized functions are for performing the transformations
equivalently on each row in a matrix, while the normal ones are able
to perform 2-d ffts if given a matrix.
The return of a FFC is a normal and complex matrix pair.
LDE 23/24 project
Co-authored-by: Mufan Wang
Co-authored-by: Frederic Caspar Zoepffel
Co-authored-by: Jessica Eva Sophie Priebe
Closes #1995
---
.../java/org/apache/sysds/common/Builtins.java | 4 +
.../java/org/apache/sysds/hops/FunctionOp.java | 36 ++
.../sysds/parser/BuiltinFunctionExpression.java| 141 ++
.../org/apache/sysds/parser/DMLTranslator.java | 4 +
.../runtime/instructions/CPInstructionParser.java | 10 +-
.../runtime/instructions/cp/CPInstruction.java | 2 +-
.../cp/MultiReturnBuiltinCPInstruction.java| 34 ++
...tiReturnComplexMatrixBuiltinCPInstruction.java} | 117 +++--
.../sysds/runtime/matrix/data/LibCommonsMath.java | 167 ++-
.../runtime/matrix/data/LibMatrixFourier.java | 479 +
.../sysds/test/component/matrix/FourierTest.java | 344 +++
11 files changed, 1287 insertions(+), 51 deletions(-)
diff --git a/src/main/java/org/apache/sysds/common/Builtins.java
b/src/main/java/org/apache/sysds/common/Builtins.java
index 4d0e13791f..7e83984e47 100644
--- a/src/main/java/org/apache/sysds/common/Builtins.java
+++ b/src/main/java/org/apache/sysds/common/Builtins.java
@@ -133,6 +133,8 @@ public enum Builtins {
FIT_PIPELINE("fit_pipeline", true),
FIX_INVALID_LENGTHS("fixInvalidLengths", true),
FIX_INVALID_LENGTHS_APPLY("fixInvalidLengthsApply", true),
+ FFT("fft", false, ReturnType.MULTI_RETURN),
+ FFT_LINEARIZED("fft_linearized", false, ReturnType.MULTI_RETURN),
FF_TRAIN("ffTrain", true),
FF_PREDICT("ffPredict", true),
FLOOR("floor", false),
@@ -154,6 +156,8 @@ public enum Builtins {
HOSPITAL_RESIDENCY_MATCH("hospitalResidencyMatch", true),
HYPERBAND("hyperband", true),
IFELSE("ifelse", false),
+ IFFT("ifft", false, ReturnType.MULTI_RETURN),
+ IFFT_LINEARIZED("ifft_linearized", false, ReturnType.MULTI_RETURN),
IMG_MIRROR("img_mirror", true),
IMG_MIRROR_LINEARIZED("img_mirror_linearized", true),
IMG_BRIGHTNESS("img_brightness", true),
diff --git a/src/main/java/org/apache/sysds/hops/FunctionOp.java
b/src/main/java/org/apache/sysds/hops/FunctionOp.java
index 28cd6eeafb..ffc12c30ee 100644
--- a/src/main/java/org/apache/sysds/hops/FunctionOp.java
+++ b/src/main/java/org/apache/sysds/hops/FunctionOp.java
@@ -201,6 +201,26 @@ public class FunctionOp extends Hop
long outputValues =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(1).getDim1(), 1, 1.0);
return outputVectors+outputValues;
}
+ else if ( getFunctionName().equalsIgnoreCase("fft") ) {
+ long outputRe =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(0).getDim1(),
getOutputs().get(0).getDim2(), 1.0);
+ long outputIm =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(1).getDim1(),
getOutputs().get(1).getDim2(), 1.0);
+ return outputRe+outputIm;
+ }
+ else if ( getFunctionName().equalsIgnoreCase("ifft") ) {
+ long outputRe =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(0).getDim1(),
getOutputs().get(0).getDim2(), 1.0);
+ long outputIm =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(1).getDim1(),
getOutputs().get(1).getDim2(), 1.0);
+ return outputRe+outputIm;
+ }
+ else if (
getFunctionName().equalsIgnoreCase("fft_linearized") ) {
+ long outputRe =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(0).getDim1(),
getOutputs().get(0).getDim2(), 1.0);
+ long outputIm =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().
(systemds) 03/05: [SYSTEMDS-3686] STFT
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 35b8e03cbb62d9ea26f0417abfd100dbdef2e002
Author: Mufan Wang
AuthorDate: Thu Apr 4 17:18:09 2024 +0200
[SYSTEMDS-3686] STFT
This commit adds a short time fourier transformation to the system.
this applies fast fourier transformations on windows of different
stride and widths, enabeling applications such as sound classification.
LDE 23/24 project
Co-authored-by: Mufan Wang
Co-authored-by: Frederic Caspar Zoepffel
Co-authored-by: Jessica Eva Sophie Priebe
Closes #2000
---
.../java/org/apache/sysds/common/Builtins.java | 1 +
.../java/org/apache/sysds/hops/FunctionOp.java | 9 +
.../sysds/parser/BuiltinFunctionExpression.java| 66
.../org/apache/sysds/parser/DMLTranslator.java | 1 +
.../runtime/instructions/CPInstructionParser.java | 1 +
.../instructions/cp/ComputationCPInstruction.java | 18 +-
.../cp/MultiReturnBuiltinCPInstruction.java| 8 +
...ltiReturnComplexMatrixBuiltinCPInstruction.java | 65 +++-
.../sysds/runtime/matrix/data/LibCommonsMath.java | 53 ++
.../sysds/runtime/matrix/data/LibMatrixSTFT.java | 121 ++
.../test/component/matrix/EigenDecompTest.java | 3 +
.../sysds/test/component/matrix/STFTTest.java | 182 +
12 files changed, 525 insertions(+), 3 deletions(-)
diff --git a/src/main/java/org/apache/sysds/common/Builtins.java
b/src/main/java/org/apache/sysds/common/Builtins.java
index 7e83984e47..8f113c092f 100644
--- a/src/main/java/org/apache/sysds/common/Builtins.java
+++ b/src/main/java/org/apache/sysds/common/Builtins.java
@@ -310,6 +310,7 @@ public enum Builtins {
STATSNA("statsNA", true),
STRATSTATS("stratstats", true),
STEPLM("steplm",true, ReturnType.MULTI_RETURN),
+ STFT("stft", false, ReturnType.MULTI_RETURN),
SQRT("sqrt", false),
SUM("sum", false),
SVD("svd", false, ReturnType.MULTI_RETURN),
diff --git a/src/main/java/org/apache/sysds/hops/FunctionOp.java
b/src/main/java/org/apache/sysds/hops/FunctionOp.java
index ffc12c30ee..95b5411500 100644
--- a/src/main/java/org/apache/sysds/hops/FunctionOp.java
+++ b/src/main/java/org/apache/sysds/hops/FunctionOp.java
@@ -221,6 +221,11 @@ public class FunctionOp extends Hop
long outputIm =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(1).getDim1(),
getOutputs().get(1).getDim2(), 1.0);
return outputRe+outputIm;
}
+ else if ( getFunctionName().equalsIgnoreCase("stft") ) {
+ long outputRe =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(0).getDim1(),
getOutputs().get(0).getDim2(), 1.0);
+ long outputIm =
OptimizerUtils.estimateSizeExactSparsity(getOutputs().get(1).getDim1(),
getOutputs().get(1).getDim2(), 1.0);
+ return outputRe+outputIm;
+ }
else if ( getFunctionName().equalsIgnoreCase("lstm") ||
getFunctionName().equalsIgnoreCase("lstm_backward") ) {
// TODO: To allow for initial version to always
run on the GPU
return 0;
@@ -286,6 +291,10 @@ public class FunctionOp extends Hop
// 2 matrices of size same as the input
return
2*OptimizerUtils.estimateSizeExactSparsity(getInput().get(0).getDim1(),
getInput().get(0).getDim2(), 1.0);
}
+ else if ( getFunctionName().equalsIgnoreCase("stft") ) {
+ // 2 matrices of size same as the input
+ return
2*OptimizerUtils.estimateSizeExactSparsity(getInput().get(0).getDim1(),
getInput().get(0).getDim2(), 1.0);
+ }
else if
(getFunctionName().equalsIgnoreCase("batch_norm2d") ||
getFunctionName().equalsIgnoreCase("batch_norm2d_backward") ||
getFunctionName().equalsIgnoreCase("batch_norm2d_train") ||
getFunctionName().equalsIgnoreCase("batch_norm2d_test")) {
return 0;
diff --git
a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
index 4b3c8e82f7..c3f1026627 100644
--- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
+++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
@@ -589,6 +5
(systemds) 02/05: [SYSTEMDS-3685] Python FFT
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit d22ffeccd7b10af366574f7fe03d637be9db49d5
Author: Frederic Caspar Zoepffel
AuthorDate: Thu Apr 4 17:17:44 2024 +0200
[SYSTEMDS-3685] Python FFT
This commit adds support in the Python API for fft and ifft.
Future work is to add the linearized versions of the commands.
LDE 23/24 project
Co-authored-by: Mufan Wang
Co-authored-by: Frederic Caspar Zoepffel
Co-authored-by: Jessica Eva Sophie Priebe
Closes #1983
---
.../sysds/parser/BuiltinFunctionExpression.java| 148 ++---
.../python/systemds/context/systemds_context.py| 37 ++-
src/main/python/tests/matrix/test_fft.py | 333 +
3 files changed, 479 insertions(+), 39 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
index 5e86a2fd8e..4b3c8e82f7 100644
--- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
+++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java
@@ -381,20 +381,41 @@ public class BuiltinFunctionExpression extends
DataIdentifier {
break;
}
case FFT: {
+
+ Expression expressionOne = getFirstExpr();
+ Expression expressionTwo = getSecondExpr();
+
+ if(expressionOne == null) {
+ raiseValidateError("The first argument to " +
_opcode + " cannot be null.", false,
+ LanguageErrorCodes.INVALID_PARAMETERS);
+ }
+ else if(expressionOne.getOutput() == null ||
expressionOne.getOutput().getDim1() == 0 ||
+ expressionOne.getOutput().getDim2() == 0) {
+ raiseValidateError("The first argument to " +
_opcode + " cannot be an empty matrix.", false,
+ LanguageErrorCodes.INVALID_PARAMETERS);
+ }
+ else if(expressionTwo != null) {
+ raiseValidateError("Too many arguments. This
FFT implementation is only defined for real inputs.", false,
+ LanguageErrorCodes.INVALID_PARAMETERS);
+ }
+ else
if(!isPowerOfTwo(expressionOne.getOutput().getDim1()) ||
+
!isPowerOfTwo(expressionOne.getOutput().getDim2())) {
+ raiseValidateError(
+ "This FFT implementation is only
defined for matrices with dimensions that are powers of 2.", false,
+ LanguageErrorCodes.INVALID_PARAMETERS);
+ }
+
checkNumParameters(1);
- checkMatrixParam(getFirstExpr());
+ checkMatrixParam(expressionOne);
- // setup output properties
DataIdentifier fftOut1 = (DataIdentifier)
getOutputs()[0];
DataIdentifier fftOut2 = (DataIdentifier)
getOutputs()[1];
- // Output1 - FFT Values
fftOut1.setDataType(DataType.MATRIX);
fftOut1.setValueType(ValueType.FP64);
fftOut1.setDimensions(getFirstExpr().getOutput().getDim1(),
getFirstExpr().getOutput().getDim2());
fftOut1.setBlocksize(getFirstExpr().getOutput().getBlocksize());
- // Output2 - FFT Vectors
fftOut2.setDataType(DataType.MATRIX);
fftOut2.setValueType(ValueType.FP64);
fftOut2.setDimensions(getFirstExpr().getOutput().getDim1(),
getFirstExpr().getOutput().getDim2());
@@ -405,16 +426,53 @@ public class BuiltinFunctionExpression extends
DataIdentifier {
}
case IFFT: {
Expression expressionTwo = getSecondExpr();
- checkNumParameters(getSecondExpr() != null ? 2 : 1);
- checkMatrixParam(getFirstExpr());
- if (expressionTwo != null)
- checkMatrixParam(getSecondExpr());
+ Expression expressionOne = getFirstExpr();
+
+ if(expressionOne == null) {
+ raiseValidateError("The first argument to " +
_opcode + " cannot be null.", false,
+ LanguageErrorCodes.INVALID_PARAMETERS);
+
(systemds) branch main updated (8bae559bcb -> 44fbc5af83)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
from 8bae559bcb [MINOR] gitignore venv directories from python venv
new 52639b2e57 [SYSTEMDS-3685] DML Integration of FFT and IFFT
new d22ffeccd7 [SYSTEMDS-3685] Python FFT
new 35b8e03cbb [SYSTEMDS-3686] STFT
new 3f166c03bc [SYSTEMDS-3685] FFT parallel, including other builtin
functioncalls
new 44fbc5af83 [MINOR] refine the selection of jar file for bin script
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
bin/systemds | 36 +-
.../java/org/apache/sysds/common/Builtins.java | 5 +
.../java/org/apache/sysds/hops/FunctionOp.java | 66 ++-
src/main/java/org/apache/sysds/hops/Hop.java | 8 +-
src/main/java/org/apache/sysds/hops/UnaryOp.java | 2 +-
.../java/org/apache/sysds/lops/Compression.java| 10 +-
.../java/org/apache/sysds/lops/FunctionCallCP.java | 48 +-
.../sysds/parser/BuiltinFunctionExpression.java| 279 +++
.../org/apache/sysds/parser/DMLTranslator.java | 5 +
.../runtime/instructions/CPInstructionParser.java | 11 +-
.../cp/AggregateUnaryCPInstruction.java| 6 +-
.../runtime/instructions/cp/CPInstruction.java | 2 +-
.../instructions/cp/CompressionCPInstruction.java | 45 +-
.../instructions/cp/ComputationCPInstruction.java | 18 +-
.../runtime/instructions/cp/DnnCPInstruction.java | 13 +-
.../cp/MultiReturnBuiltinCPInstruction.java| 70 ++-
...ltiReturnComplexMatrixBuiltinCPInstruction.java | 240 ++
.../sysds/runtime/matrix/data/LibCommonsMath.java | 243 +-
.../runtime/matrix/data/LibMatrixFourier.java | 515 +
.../sysds/runtime/matrix/data/LibMatrixSTFT.java | 121 +
.../python/systemds/context/systemds_context.py| 37 +-
.../python/systemds/operator/algorithm/__init__.py | 2 +
.../algorithm/builtin/{deepWalk.py => pageRank.py} | 34 +-
src/main/python/tests/lineage/test_lineagetrace.py | 33 +-
src/main/python/tests/matrix/test_fft.py | 333 +
.../applications/ScalableDecompositionTest.java| 4 +-
.../test/component/matrix/EigenDecompTest.java | 3 +
.../sysds/test/component/matrix/FourierTest.java | 366 +++
.../sysds/test/component/matrix/STFTTest.java | 182
.../scripts/functions/builtin/GridSearchLMCV.dml | 3 +-
30 files changed, 2617 insertions(+), 123 deletions(-)
create mode 100644
src/main/java/org/apache/sysds/runtime/instructions/cp/MultiReturnComplexMatrixBuiltinCPInstruction.java
create mode 100644
src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixFourier.java
create mode 100644
src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixSTFT.java
copy src/main/python/systemds/operator/algorithm/builtin/{deepWalk.py =>
pageRank.py} (65%)
create mode 100644 src/main/python/tests/matrix/test_fft.py
create mode 100644
src/test/java/org/apache/sysds/test/component/matrix/FourierTest.java
create mode 100644
src/test/java/org/apache/sysds/test/component/matrix/STFTTest.java
(systemds) branch main updated: [MINOR] gitignore venv directories from python venv
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 8bae559bcb [MINOR] gitignore venv directories from python venv 8bae559bcb is described below commit 8bae559bcb4f52408efeeaec29be295aa5207ccb Author: Sebastian Baunsgaard AuthorDate: Thu Apr 4 18:28:49 2024 +0200 [MINOR] gitignore venv directories from python venv --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 1a83a3a80e..6357b16e20 100644 --- a/.gitignore +++ b/.gitignore @@ -144,3 +144,6 @@ scripts/perftest/fed/temp src/test/scripts/functions/iogen/*.raw src/test/scripts/functions/pipelines/intermediates/regression/* src/test/scripts/functions/pipelines/intermediates/classification/* + +venv +venv/*
(systemds) branch main updated: [MINOR] Add missing license
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new a22a85915b [MINOR] Add missing license a22a85915b is described below commit a22a85915b7c981d185f75f9b92b1a570acbb2d9 Author: Sebastian Baunsgaard AuthorDate: Thu Apr 4 18:24:21 2024 +0200 [MINOR] Add missing license --- .../apache/sysds/performance/matrix/SparseAppend.java | 19 +++ 1 file changed, 19 insertions(+) diff --git a/src/test/java/org/apache/sysds/performance/matrix/SparseAppend.java b/src/test/java/org/apache/sysds/performance/matrix/SparseAppend.java index 7930fd3275..73db34ce12 100644 --- a/src/test/java/org/apache/sysds/performance/matrix/SparseAppend.java +++ b/src/test/java/org/apache/sysds/performance/matrix/SparseAppend.java @@ -1,3 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + package org.apache.sysds.performance.matrix; import java.util.Random;
(systemds) branch main updated: [MINOR] Append perf
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new bbdbb1a781 [MINOR] Append perf
bbdbb1a781 is described below
commit bbdbb1a7814d58da15cdd0bd35a08811a635d869
Author: Sebastian Baunsgaard
AuthorDate: Thu Apr 4 18:08:27 2024 +0200
[MINOR] Append perf
This commit adds a perf script for MCSR appending.
It is manly ment as an example of how to execute a perf script.
Me:~/github/systemds$ java -jar target/systemds-3.3.0-SNAPSHOT-perf.jar
1004 1000 10
Appending rep: 1000 of 10 distinct append calls (including random and
allocations)
Append all dense: 4.262+- 0.164 ms
Append all zero on empty: 0.198+- 0.004 ms
Append all zero on Scalar:0.197+- 0.004 ms
Append all zero on Array: 0.203+- 0.013 ms
Append half zero on Array:4.422+- 0.170 ms
```
Closes #2011
---
.../apache/sysds/runtime/data/SparseBlockMCSR.java | 5 +-
.../java/org/apache/sysds/performance/Main.java| 8 ++
.../java/org/apache/sysds/performance/README.md| 10 +--
.../org/apache/sysds/performance/TimingUtils.java | 14
.../performance/matrix/MatrixMulPerformance.java | 4 +-
.../sysds/performance/matrix/SparseAppend.java | 89 ++
6 files changed, 121 insertions(+), 9 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
index 025da10394..52b5d2e338 100644
--- a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
+++ b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
@@ -378,12 +378,13 @@ public class SparseBlockMCSR extends SparseBlock
@Override
public final void append(final int r, final int c, final double v) {
+ // Perf verified in java -jar
target/systemds-3.3.0-SNAPSHOT-perf.jar 1004 1000 10
if(v == 0)
return;
else if(_rows[r] == null)
_rows[r] = new SparseRowScalar(c, v);
- else
- _rows[r] = _rows[r].append(c, v);
+ else
+ _rows[r] = _rows[r].append(c, v);
}
@Override
diff --git a/src/test/java/org/apache/sysds/performance/Main.java
b/src/test/java/org/apache/sysds/performance/Main.java
index 2fed0d7144..9959192188 100644
--- a/src/test/java/org/apache/sysds/performance/Main.java
+++ b/src/test/java/org/apache/sysds/performance/Main.java
@@ -32,6 +32,7 @@ import org.apache.sysds.performance.generators.IGenerate;
import org.apache.sysds.performance.generators.MatrixFile;
import org.apache.sysds.performance.matrix.MatrixMulPerformance;
import org.apache.sysds.performance.matrix.MatrixStorage;
+import org.apache.sysds.performance.matrix.SparseAppend;
import org.apache.sysds.runtime.data.SparseBlock;
import org.apache.sysds.runtime.frame.data.FrameBlock;
import org.apache.sysds.runtime.matrix.data.MatrixBlock;
@@ -115,6 +116,9 @@ public class Main {
case 1003:
run1003(args);
break;
+ case 1004:
+ run1004(args);
+ break;
default:
break;
}
@@ -319,6 +323,10 @@ public class Main {
ms.testBalancedDims(SparseBlock.Type.DCSR, sparsity,
numEntries, resolution, maxRowColRatio, repetitions);
}
+ private static void run1004(String[] args){
+ new SparseAppend(args);
+ }
+
public static void main(String[] args) {
try {
exec(Integer.parseInt(args[0]), args);
diff --git a/src/test/java/org/apache/sysds/performance/README.md
b/src/test/java/org/apache/sysds/performance/README.md
index 7e7edbb805..7129757f34 100644
--- a/src/test/java/org/apache/sysds/performance/README.md
+++ b/src/test/java/org/apache/sysds/performance/README.md
@@ -28,7 +28,7 @@ mvn package
Example of running it:
```bash
-java -jar target/systemds-3.2.0-SNAPSHOT-perf.jar 1
+java -jar target/systemds-3.3.0-SNAPSHOT-perf.jar 1
```
example result of the above job:
@@ -49,24 +49,24 @@ Running Steam Compression Test
With profiler:
```bash
-java -jar
-agentpath:$HOME/Programs/profiler/lib/libasyncProfiler.so=start,event=cpu,file=temp/log.html
target/systemds-3.2.0-SNAPSHOT-perf.jar 12 1 100 4 1.0 16 1000 -1
+java -jar
-agentpath:$HOME/Programs/profiler/lib/libasyncProfiler.so=start,event=cpu,file=temp/log.html
target/systemds-3.3.0-SNAPSHOT-perf.jar 12 1 100 4 1.0 16 1000 -1
```
Take a Matrix and perform serialization
(systemds) branch main updated: [SYSTEMDS-3684] Startup SYSTEMDS_STANDALONE_OPTS Regression Fix
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new e94df56c8b [SYSTEMDS-3684] Startup SYSTEMDS_STANDALONE_OPTS Regression Fix e94df56c8b is described below commit e94df56c8b81ed8c3d4cf2bb25aab5eec795cd1e Author: Sebastian Baunsgaard AuthorDate: Tue Mar 26 18:09:22 2024 +0100 [SYSTEMDS-3684] Startup SYSTEMDS_STANDALONE_OPTS Regression Fix This commit fixes a bug I introduced a couple of weeks ago, where I erroneously forgot to include the java arguments from SYSTEMDS_STANDALONE_OPTS in the bin/systemds file, when I changed the launching from using -cp to -jar for performance gains in startup time of SystemDS. Thank you to Louis Le Page for finding the regression. Closes #2007 --- bin/systemds | 9 ++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/bin/systemds b/bin/systemds index bbe581c907..65f2a82867 100755 --- a/bin/systemds +++ b/bin/systemds @@ -431,7 +431,8 @@ if [ $WORKER == 1 ]; then print_out "# starting Federated worker on port $PORT" print_out "###" CMD=" \ - java $LOG4JPROPFULL \ + java $SYSTEMDS_STANDALONE_OPTS \ + $LOG4JPROPFULL \ -jar $SYSTEMDS_JAR_FILE \ -w $PORT \ $CONFIG_FILE \ @@ -444,7 +445,8 @@ elif [ "$FEDMONITORING" == 1 ]; then print_out "# starting Federated backend monitoring on port $PORT" print_out "###" CMD=" \ - java $LOG4JPROPFULL \ + java $SYSTEMDS_STANDALONE_OPTS \ + $LOG4JPROPFULL \ -jar $SYSTEMDS_JAR_FILE \ -fedMonitoring $PORT \ $CONFIG_FILE \ @@ -457,7 +459,8 @@ elif [ $SYSDS_DISTRIBUTED == 0 ]; then print_out "# Running script $SCRIPT_FILE locally with opts: $*" print_out "###" CMD=" \ - java $LOG4JPROPFULL \ + java $SYSTEMDS_STANDALONE_OPTS \ + $LOG4JPROPFULL \ -jar $SYSTEMDS_JAR_FILE \ -f $SCRIPT_FILE \ -exec $SYSDS_EXEC_MODE \
(systemds) branch main updated (0d6236454b -> 9d5002eb0a)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from 0d6236454b [MINOR] Add warnings logging for hadoop and systemds for releases add 9d5002eb0a [MINOR] Change 'binary' systemds to use Manifest No new revisions were added by this update. Summary of changes: bin/systemds | 18 ++ pom.xml | 3 --- 2 files changed, 6 insertions(+), 15 deletions(-)
(systemds) branch main updated: [MINOR] Add warnings logging for hadoop and systemds for releases
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 0d6236454b [MINOR] Add warnings logging for hadoop and systemds for releases 0d6236454b is described below commit 0d6236454bd0757e15218854327574a48583177c Author: Sebastian Baunsgaard AuthorDate: Tue Mar 5 15:40:16 2024 +0100 [MINOR] Add warnings logging for hadoop and systemds for releases --- conf/log4j.properties.template | 2 ++ 1 file changed, 2 insertions(+) diff --git a/conf/log4j.properties.template b/conf/log4j.properties.template index 9b751b57ca..9a381e00aa 100644 --- a/conf/log4j.properties.template +++ b/conf/log4j.properties.template @@ -22,8 +22,10 @@ log4j.rootLogger=ERROR,console log4j.logger.org.apache.sysds=ERROR +log4j.logger.org.apache.sysds.utils.SettingsChecker=WARN log4j.logger.org.apache.spark=ERROR log4j.logger.org.apache.hadoop=OFF +log4j.logger.org.apache.hadoop.util.NativeCodeLoader=INFO log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err
(systemds) branch main updated (eb29b2d548 -> 42ed9e7951)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from eb29b2d548 [SYSTEMDS-2926] AWS scripts update for EMR-7.0.0 (#2003) add 68abe0daa2 [SYSTEMDS-3673] log4j and slf4j update to latest version add 42ed9e7951 [SYSTEMDS-3673] slf4j apache logging ignore subpackage No new revisions were added by this update. Summary of changes: pom.xml | 23 +-- 1 file changed, 21 insertions(+), 2 deletions(-)
(systemds) branch main updated: [MINOR] Generate Python tSNE builtin
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 03ccaee6af [MINOR] Generate Python tSNE builtin 03ccaee6af is described below commit 03ccaee6afc016d83c307734f6e0115f8ea22edf Author: Sebastian Baunsgaard AuthorDate: Mon Feb 19 21:51:04 2024 +0100 [MINOR] Generate Python tSNE builtin --- src/main/python/systemds/operator/algorithm/builtin/tSNE.py | 13 + 1 file changed, 13 insertions(+) diff --git a/src/main/python/systemds/operator/algorithm/builtin/tSNE.py b/src/main/python/systemds/operator/algorithm/builtin/tSNE.py index 3c659160c6..491a3a 100644 --- a/src/main/python/systemds/operator/algorithm/builtin/tSNE.py +++ b/src/main/python/systemds/operator/algorithm/builtin/tSNE.py @@ -35,6 +35,16 @@ def tSNE(X: Matrix, This function performs dimensionality reduction using tSNE algorithm based on the paper: Visualizing Data using t-SNE, Maaten et. al. + There exists a variant of t-SNE, implemented in sklearn, that first reduces the + dimenisonality of the data using PCA to reduce noise and then applies t-SNE for + further dimensionality reduction. A script of this can be found in the tutorials + folder: scripts/tutorials/tsne/pca-tsne.dml + + For direct reference and tips on choosing the dimension for the PCA pre-processing, + you can visit: + https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/manifold/_t_sne.py + https://lvdmaaten.github.io/tsne/ + :param X: Data Matrix of shape @@ -44,9 +54,12 @@ def tSNE(X: Matrix, :param lr: Learning rate :param momentum: Momentum Parameter :param max_iter: Number of iterations +:param tol: Tolerance for early stopping in gradient descent :param seed: The seed used for initial values. If set to -1 random seeds are selected. :param is_verbose: Print debug information +:param print_iter: Intervals of printing out the L1 norm values. Parameter not relevant if +is_verbose = FALSE. :return: Data Matrix of shape (number of data points, reduced_dims) """
(systemds) branch main updated: [SYSTEMDS-3670] TSNE PCA preprocessing
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 610222cbca [SYSTEMDS-3670] TSNE PCA preprocessing 610222cbca is described below commit 610222cbca25b76c327cb5ace780c3d0ead9e1bf Author: Sebastian Baunsgaard AuthorDate: Tue Jan 30 19:34:33 2024 +0100 [SYSTEMDS-3670] TSNE PCA preprocessing This commit adds a comment and example script of TSNE with PCA preprocessing According to Scikit Learn then PCA preprocessing reduces the dimensions TSNE has to work with and, therefore, improve performance. LDE Project Part 1 WS 2023/2024 Closes #1991 --- scripts/builtin/tSNE.dml| 10 ++ scripts/tutorials/tsne/pca-tsne.dml | 38 + 2 files changed, 48 insertions(+) diff --git a/scripts/builtin/tSNE.dml b/scripts/builtin/tSNE.dml index 131ab1013c..a28a1c1a0a 100644 --- a/scripts/builtin/tSNE.dml +++ b/scripts/builtin/tSNE.dml @@ -22,6 +22,16 @@ # This function performs dimensionality reduction using tSNE algorithm based on # the paper: Visualizing Data using t-SNE, Maaten et. al. # +# There exists a variant of t-SNE, implemented in sklearn, that first reduces the +# dimenisonality of the data using PCA to reduce noise and then applies t-SNE for +# further dimensionality reduction. A script of this can be found in the tutorials +# folder: scripts/tutorials/tsne/pca-tsne.dml +# +# For direct reference and tips on choosing the dimension for the PCA pre-processing, +# you can visit: +# https://github.com/scikit-learn/scikit-learn/blob/main/sklearn/manifold/_t_sne.py +# https://lvdmaaten.github.io/tsne/ +# # INPUT: # --- # X Data Matrix of shape diff --git a/scripts/tutorials/tsne/pca-tsne.dml b/scripts/tutorials/tsne/pca-tsne.dml new file mode 100644 index 00..eb159f68e4 --- /dev/null +++ b/scripts/tutorials/tsne/pca-tsne.dml @@ -0,0 +1,38 @@ +#- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#- + +# +# tSNE dimensional reduction technique with PCA pre-processing, +# inspired from the sklearn implementation of tSNE: +# https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html + + +# Load data +data = read($X) + +# Pre-process data with PCA +[PCA, components, centering, scalefactor] = pca(X=data, K=$k) + +# Do tSNE with PCA output +Y = tSNE(X=PCA) + +# Save reduced dimensions +write(Y, $Y)
(systemds) branch main updated: [MINOR] Fix edge case tests to reflect new changes
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 2cd782f72a [MINOR] Fix edge case tests to reflect new changes
2cd782f72a is described below
commit 2cd782f72a1f767e67c14022384fa50d7161b540
Author: Sebastian Baunsgaard
AuthorDate: Tue Jan 30 19:34:33 2024 +0100
[MINOR] Fix edge case tests to reflect new changes
---
.../java/org/apache/sysds/test/component/frame/FrameCustomTest.java | 5 ++---
.../sysds/test/functions/transform/TransformEncodeDecodeTest.java| 2 +-
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git
a/src/test/java/org/apache/sysds/test/component/frame/FrameCustomTest.java
b/src/test/java/org/apache/sysds/test/component/frame/FrameCustomTest.java
index 9d6c7aa482..3387db56ab 100644
--- a/src/test/java/org/apache/sysds/test/component/frame/FrameCustomTest.java
+++ b/src/test/java/org/apache/sysds/test/component/frame/FrameCustomTest.java
@@ -35,7 +35,7 @@ public class FrameCustomTest {
double maxp1 = Integer.MAX_VALUE + 1.0;
MatrixBlock mb = TestUtils.generateTestMatrixBlock(100, 100,
maxp1, maxp1, 1.0, 23);
FrameBlock f = DataConverter.convertToFrameBlock(mb);
- assertTrue(f.getSchema()[0] == ValueType.INT64);
+ assertTrue(f.getSchema()[0] == ValueType.FP64);
}
@Test
@@ -50,8 +50,7 @@ public class FrameCustomTest {
public void castErrorValue() {
MatrixBlock mb = new MatrixBlock(10, 10,
Double.parseDouble("2.572306572E9"));
FrameBlock f = DataConverter.convertToFrameBlock(mb);
- assertTrue(f.getSchema()[0] == ValueType.INT64);
-
+ assertTrue(f.getSchema()[0] == ValueType.FP64);
}
@Test
diff --git
a/src/test/java/org/apache/sysds/test/functions/transform/TransformEncodeDecodeTest.java
b/src/test/java/org/apache/sysds/test/functions/transform/TransformEncodeDecodeTest.java
index 089fd78349..762167625d 100644
---
a/src/test/java/org/apache/sysds/test/functions/transform/TransformEncodeDecodeTest.java
+++
b/src/test/java/org/apache/sysds/test/functions/transform/TransformEncodeDecodeTest.java
@@ -114,7 +114,7 @@ public class TransformEncodeDecodeTest extends
AutomatedTestBase {
SCRIPT_DIR + TEST_DIR + SPEC, output("FO")};
// run test
- LOG.error(runTest(null));
+ runTest(null);
// compare matrices (values recoded to identical codes)
FrameReader reader =
FrameReaderFactory.createFrameReader(FileFormat.safeValueOf(fmt));
(systemds) branch main updated: [MINOR] Test SliceLine as.frame
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 64174bdf28 [MINOR] Test SliceLine as.frame
64174bdf28 is described below
commit 64174bdf284bd96431d1ce18ef383378e86b3192
Author: Sebastian Baunsgaard
AuthorDate: Tue Jan 30 14:36:18 2024 +0100
[MINOR] Test SliceLine as.frame
Adds the test missing from commit:
75cf454e282100be722a3dc9805d941dc16ee770
---
.../frame/FrameMatrixCastingSliceLineTest.java | 66 ++
.../scripts/functions/frame/SliceLineFailCase.dml | 31 ++
2 files changed, 97 insertions(+)
diff --git
a/src/test/java/org/apache/sysds/test/functions/frame/FrameMatrixCastingSliceLineTest.java
b/src/test/java/org/apache/sysds/test/functions/frame/FrameMatrixCastingSliceLineTest.java
new file mode 100644
index 00..addb0313d2
--- /dev/null
+++
b/src/test/java/org/apache/sysds/test/functions/frame/FrameMatrixCastingSliceLineTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.sysds.test.functions.frame;
+
+import org.apache.sysds.common.Types.ExecMode;
+import org.apache.sysds.test.AutomatedTestBase;
+import org.apache.sysds.test.TestConfiguration;
+import org.apache.sysds.test.TestUtils;
+import org.junit.Test;
+
+public class FrameMatrixCastingSliceLineTest extends AutomatedTestBase {
+ private final static String TEST_DIR = "functions/frame/";
+ private final static String TEST_NAME1 = "SliceLineFailCase";
+ private final static String TEST_CLASS_DIR = TEST_DIR +
FrameMatrixCastingTest.class.getSimpleName() + "/";
+
+ @Override
+ public void setUp() {
+ TestUtils.clearAssertionInformation();
+ addTestConfiguration(TEST_NAME1, new
TestConfiguration(TEST_CLASS_DIR, TEST_NAME1, new String[] {"B"}));
+ }
+
+ @Test
+ public void runFrameCastingTest() {
+
+ ExecMode platformOld = rtplatform;
+ setOutputBuffering(true);
+ try {
+
+ TestConfiguration config =
getTestConfiguration(TEST_NAME1);
+ loadTestConfiguration(config);
+
+ String HOME = SCRIPT_DIR + TEST_DIR;
+ fullDMLScriptName = HOME + TEST_NAME1 + ".dml";
+ programArgs = new String[] {};
+
+ // should not fail
+ // this test does not verify behavior
+ runTest(null);
+
+ }
+ catch(Exception ex) {
+ throw new RuntimeException(ex);
+ }
+ finally {
+ rtplatform = platformOld;
+ }
+ }
+
+}
diff --git a/src/test/scripts/functions/frame/SliceLineFailCase.dml
b/src/test/scripts/functions/frame/SliceLineFailCase.dml
new file mode 100644
index 00..54d5d987a0
--- /dev/null
+++ b/src/test/scripts/functions/frame/SliceLineFailCase.dml
@@ -0,0 +1,31 @@
+
+#-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-
+
+k = ifdef
(systemds) branch main updated: [MINOR] Add extra safety checks for as.frame
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 13d4db7694 [MINOR] Add extra safety checks for as.frame
13d4db7694 is described below
commit 13d4db76948b141fad82a120d2068c1ca4560993
Author: Sebastian Baunsgaard
AuthorDate: Tue Jan 30 14:27:47 2024 +0100
[MINOR] Add extra safety checks for as.frame
---
.../apache/sysds/runtime/frame/data/lib/FrameFromMatrixBlock.java | 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/lib/FrameFromMatrixBlock.java
b/src/main/java/org/apache/sysds/runtime/frame/data/lib/FrameFromMatrixBlock.java
index eeac27e2e1..001e4f7a47 100644
---
a/src/main/java/org/apache/sysds/runtime/frame/data/lib/FrameFromMatrixBlock.java
+++
b/src/main/java/org/apache/sysds/runtime/frame/data/lib/FrameFromMatrixBlock.java
@@ -89,10 +89,16 @@ public class FrameFromMatrixBlock {
for(int c = 0; c < nCol; c++){
for(int r = 0; r < nRow; r++){
switch(schema[c]){
+ case INT64:
+ // keep the type as FP64 if
long is detected
+ schema[c] = ValueType.FP64;
case FP64:
break;
default:
- schema[c] =
FrameUtil.isType(mb.quickGetValue(r, c), schema[c]);
+ final double v =
mb.quickGetValue(r, c);
+ if(v > Integer.MAX_VALUE)
+ schema[c] =
ValueType.FP64; // handle Integer overflow.
+ schema[c] = FrameUtil.isType(v,
schema[c]);
}
}
}
(systemds) branch main updated: [MINOR] Remove exception in cast as IntArray
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 75cf454e28 [MINOR] Remove exception in cast as IntArray
75cf454e28 is described below
commit 75cf454e282100be722a3dc9805d941dc16ee770
Author: Sebastian Baunsgaard
AuthorDate: Tue Jan 30 14:01:45 2024 +0100
[MINOR] Remove exception in cast as IntArray
This commit removes the exception in cast as IntArray from DoubleArray.
We encounter an issue in this conversion for large numbers of double
values, that does not cast perfectly to the same double values when
casting the integer values back to doubles.
The script that reproduce the bug is:
```
k = ifdef($k, 5)
paq = ifdef($paq, 1)
X = round(rand(rows = 50, cols = 10, min=1, max=10))
y = X %*% rand(rows = ncol(X), cols = 1)
w = lm(X = X, y = y)
yhat = X %*% w
ress = slicefinder(X = X, e = abs(y - yhat), k = k, maxL = 0, minSup =
1, alpha = 1, selFeat = TRUE, verbose = TRUE)
print(toString(ress))
```
A subsequent commits add a test case that ensure this bug does not
happen again.
---
.../sysds/runtime/frame/data/columns/DoubleArray.java | 15 +++
1 file changed, 3 insertions(+), 12 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/columns/DoubleArray.java
b/src/main/java/org/apache/sysds/runtime/frame/data/columns/DoubleArray.java
index 68672c5d73..8835b7c21c 100644
--- a/src/main/java/org/apache/sysds/runtime/frame/data/columns/DoubleArray.java
+++ b/src/main/java/org/apache/sysds/runtime/frame/data/columns/DoubleArray.java
@@ -293,33 +293,24 @@ public class DoubleArray extends Array {
@Override
protected Array changeTypeInteger() {
int[] ret = new int[size()];
- for(int i = 0; i < size(); i++) {
- if(_data[i] != (int) _data[i])
- throw new DMLRuntimeException("Unable to change
to Integer from Double array because of value:" + _data[i]);
+ for(int i = 0; i < size(); i++)
ret[i] = (int) _data[i];
- }
return new IntegerArray(ret);
}
@Override
protected Array changeTypeLong() {
long[] ret = new long[size()];
- for(int i = 0; i < size(); i++) {
- if(_data[i] != (long) _data[i])
- throw new DMLRuntimeException("Unable to change
to Long from Double array because of value:" + _data[i]);
+ for(int i = 0; i < size(); i++)
ret[i] = (long) _data[i];
- }
return new LongArray(ret);
}
@Override
protected Array changeTypeHash64() {
long[] ret = new long[size()];
- for(int i = 0; i < size(); i++) {
- if(_data[i] != (long) _data[i])
- throw new DMLRuntimeException("Unable to change
to Long from Double array because of value:" + _data[i]);
+ for(int i = 0; i < size(); i++)
ret[i] = (long) _data[i];
- }
return new HashLongArray(ret);
}
(systemds) branch main updated (02d4b01f29 -> 4f52f55b89)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from 02d4b01f29 [SYSTEMDS-3656] Update Hadoop 3.3.6 add 4f52f55b89 [SYSTEMDS-3655] Update Spark Dependencies No new revisions were added by this update. Summary of changes: pom.xml| 18 +- src/assembly/bin.xml | 3 +-- src/main/java/org/apache/sysds/hops/UnaryOp.java | 2 +- .../sysds/runtime/compress/colgroup/APreAgg.java | 2 +- .../runtime/compress/colgroup/indexes/RangeIndex.java | 2 +- .../compress/colgroup/scheme/CompressionScheme.java| 2 +- .../runtime/compress/colgroup/scheme/SDCSchemeSC.java | 2 +- .../apache/sysds/runtime/compress/lib/CLALibMerge.java | 2 +- .../sysds/runtime/compress/utils/ACountHashMap.java| 2 +- .../apache/sysds/runtime/data/DenseBlockFP64DEDUP.java | 2 +- .../component/compress/CompressedLoggingTests.java | 10 ++ .../frame/compress/FrameCompressTestUtils.java | 2 +- 12 files changed, 25 insertions(+), 24 deletions(-)
(systemds) branch main updated: [SYSTEMDS-3656] Update Hadoop 3.3.6
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 02d4b01f29 [SYSTEMDS-3656] Update Hadoop 3.3.6 02d4b01f29 is described below commit 02d4b01f294083a16d9dc0b94dc4b82202b90adc Author: Badrul Chowdhury AuthorDate: Mon Jan 15 00:36:37 2024 +0100 [SYSTEMDS-3656] Update Hadoop 3.3.6 This commit update the used HADOOP version to the newest release. The update is as far as we tested backwards compatible with SYSTEMDS Closes #1961 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index b97cfb30ad..9808bfac51 100644 --- a/pom.xml +++ b/pom.xml @@ -39,7 +39,7 @@ - 3.3.4 + 3.3.6 4.8 3.20.3 3.3.1
(systemds) branch main updated: [MINOR] Fix readme federated tutorial command
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 5ad424a3d3 [MINOR] Fix readme federated tutorial command 5ad424a3d3 is described below commit 5ad424a3d36b698c3e22eb33782cdef714b140d6 Author: Sebastian Baunsgaard AuthorDate: Wed Jan 10 10:52:32 2024 +0100 [MINOR] Fix readme federated tutorial command --- scripts/tutorials/federated/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/tutorials/federated/README.md b/scripts/tutorials/federated/README.md index 7bd7e08f7d..5210a096ac 100644 --- a/scripts/tutorials/federated/README.md +++ b/scripts/tutorials/federated/README.md @@ -174,7 +174,7 @@ that port forward the list of ports from your local machine to the remote machin Note this only works if all the federated machines are remote machines, aka the address list contain no localhost. ```sh -portforward.sh +./portforward.sh ``` Note these process will just continue running in the background so have to be manually terminated.
(systemds) branch main updated: [SYSTEMDS-3663] Low overhead join indexes
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 3e6af1b814 [SYSTEMDS-3663] Low overhead join indexes
3e6af1b814 is described below
commit 3e6af1b814bf2c71e89d79a6ca4f88fb71608ebe
Author: Sebastian Baunsgaard
AuthorDate: Sun Jan 7 17:06:30 2024 +0100
[SYSTEMDS-3663] Low overhead join indexes
This commit adds a few more variations to indexes to allow
efficient combination and ordering of column indexes when co-coding.
This is critical in cases where thousands of columns are combined,
since the execution time suddenly is dominated not by combining columns
but the column indexes.
Closes #1979
---
.../compress/colgroup/indexes/AColIndex.java | 56 -
.../compress/colgroup/indexes/ColIndexFactory.java | 2 +
.../compress/colgroup/indexes/CombinedIndex.java | 246 +
.../compress/colgroup/indexes/IColIndex.java | 80 ++-
.../compress/colgroup/indexes/RangeIndex.java | 84 ---
.../compress/colgroup/indexes/TwoRangesIndex.java | 4 +-
6 files changed, 437 insertions(+), 35 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/AColIndex.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/AColIndex.java
index df4685a65d..81a5f5b480 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/AColIndex.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/AColIndex.java
@@ -21,6 +21,8 @@ package org.apache.sysds.runtime.compress.colgroup.indexes;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+import org.apache.sysds.runtime.data.SparseBlock;
+import org.apache.sysds.runtime.data.SparseBlockCSR;
public abstract class AColIndex implements IColIndex {
@@ -69,11 +71,55 @@ public abstract class AColIndex implements IColIndex {
@Override
public boolean containsAny(IColIndex idx) {
- final IIterate it = idx.iterator();
- while(it.hasNext())
- if(contains(it.next()))
- return true;
+ if(idx instanceof TwoRangesIndex){
+ TwoRangesIndex o = (TwoRangesIndex) idx;
+ return this.containsAny(o.idx1) ||
this.containsAny(o.idx2);
+ }
+ else if(idx instanceof CombinedIndex){
+ CombinedIndex ci = (CombinedIndex) idx;
+ return containsAny(ci.l) || containsAny(ci.r);
+ }
+ else{
+ final IIterate it = idx.iterator();
+ while(it.hasNext())
+ if(contains(it.next()))
+ return true;
+
+ return false;
+ }
+ }
- return false;
+ @Override
+ public void decompressToDenseFromSparse(SparseBlock sb, int vr, int
off, double[] c) {
+ if(sb instanceof SparseBlockCSR)
+ decompressToDenseFromSparseCSR((SparseBlockCSR)sb, vr,
off, c);
+ else
+ decompressToDenseFromSparseGeneric(sb, vr, off, c);
+ }
+
+ private void decompressToDenseFromSparseGeneric(SparseBlock sb, int vr,
int off, double[] c) {
+ if(sb.isEmpty(vr))
+ return;
+ final int apos = sb.pos(vr);
+ final int alen = sb.size(vr) + apos;
+ final int[] aix = sb.indexes(vr);
+ final double[] aval = sb.values(vr);
+ for(int j = apos; j < alen; j++)
+ c[off + get(aix[j])] += aval[j];
+ }
+
+ private void decompressToDenseFromSparseCSR(SparseBlockCSR sb, int vr,
int off, double[] c) {
+ final int apos = sb.pos(vr);
+ final int alen = sb.size(vr) + apos;
+ final int[] aix = sb.indexes(vr);
+ final double[] aval = sb.values(vr);
+ for(int j = apos; j < alen; j++)
+ c[off + get(aix[j])] += aval[j];
+ }
+
+ @Override
+ public void decompressVec(int nCol, double[] c, int off, double[]
values, int rowIdx) {
+ for(int j = 0; j < nCol; j++)
+ c[off + get(j)] += values[rowIdx + j];
}
}
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ColIndexFactory.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ColIndexFactory.java
index fd929b8a1a..c9a45e4aee 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/indexes/ColIndexFactory.java
+++
b/src/main/java/org/apache/sysds/runtime
(systemds) branch main updated: [MINOR] Fix incorrect merge of MatrixBlock
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new b0fc281cc6 [MINOR] Fix incorrect merge of MatrixBlock
b0fc281cc6 is described below
commit b0fc281cc616140d29c6e7406665b027dac0686e
Author: Sebastian Baunsgaard
AuthorDate: Sun Jan 7 19:59:58 2024 +0100
[MINOR] Fix incorrect merge of MatrixBlock
---
.../sysds/runtime/matrix/data/MatrixBlock.java | 21 -
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
index 085b6a5c52..6e3ad9f8b9 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
@@ -586,16 +586,19 @@ public class MatrixBlock extends MatrixValue implements
CacheBlock,
public final boolean isEmptyBlock() {
return isEmptyBlock(true);
}
-
- public boolean isEmptyBlock(boolean safe)
- {
- boolean ret = ( sparse && sparseBlock==null ) || ( !sparse &&
denseBlock==null );
- if( nonZeros==0 )
- {
- //prevent under-estimation
- if(safe)
+ /**
+* Get if this MatrixBlock is an empty block. The call can potentially
tricker a recomputation of non zeros if the
+* non-zero count is unknown.
+*
+* @param safe True if we want to ensure the count non zeros if the nnz
is unknown.
+* @return If the block is empty.
+*/
+ public boolean isEmptyBlock(boolean safe) {
+ boolean ret = (sparse && sparseBlock == null) || (!sparse &&
denseBlock == null);
+ if(nonZeros <= 0) { // estimate non zeros if unknown or 0.
+ if(safe) // only allow the recompute if safe flag is
false.
recomputeNonZeros();
- ret = (nonZeros==0);
+ ret = (nonZeros == 0);
}
return ret;
}
(systemds) 01/02: [MINOR] MatrixBlock improved generic Unary Agg
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit ccb589056c3b6fa29332d9aa0fe33747b0774250
Author: Sebastian Baunsgaard
AuthorDate: Sun Jan 7 19:30:21 2024 +0100
[MINOR] MatrixBlock improved generic Unary Agg
---
.../spark/AggregateUnarySPInstruction.java | 2 +-
.../sysds/runtime/matrix/data/CM_N_COVCell.java| 6 -
.../sysds/runtime/matrix/data/LibMatrixAgg.java| 60 +++
.../data/LibMatrixAggUnarySpecialization.java | 152
.../sysds/runtime/matrix/data/MatrixBlock.java | 191 ++---
.../sysds/runtime/matrix/data/MatrixCell.java | 39 ++---
.../sysds/runtime/matrix/data/MatrixValue.java | 6 +-
.../sysds/runtime/matrix/data/WeightedCell.java| 9 +-
8 files changed, 246 insertions(+), 219 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/instructions/spark/AggregateUnarySPInstruction.java
b/src/main/java/org/apache/sysds/runtime/instructions/spark/AggregateUnarySPInstruction.java
index 32b80a2360..ba7237ee35 100644
---
a/src/main/java/org/apache/sysds/runtime/instructions/spark/AggregateUnarySPInstruction.java
+++
b/src/main/java/org/apache/sysds/runtime/instructions/spark/AggregateUnarySPInstruction.java
@@ -279,7 +279,7 @@ public class AggregateUnarySPInstruction extends
UnarySPInstruction {
throws Exception
{
//unary aggregate operation (always keep the correction)
- return arg0._2.aggregateUnaryOperations(
+ return (MatrixBlock) arg0._2.aggregateUnaryOperations(
_op, new MatrixBlock(), _blen,
arg0._1());
}
}
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/CM_N_COVCell.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/CM_N_COVCell.java
index 8e58630abe..a367af4f7b 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/CM_N_COVCell.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/CM_N_COVCell.java
@@ -45,12 +45,6 @@ public class CM_N_COVCell extends MatrixValue
public String toString() {
return cm.toString();
}
-
- @Override
- public MatrixValue aggregateUnaryOperations(AggregateUnaryOperator op,
- MatrixValue result, int blen, MatrixIndexes indexesIn) {
- throw new DMLRuntimeException("operation not supported for
CM_N_COVCell");
- }
@Override
public MatrixValue binaryOperations(BinaryOperator op,
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
index 0891d7f1ae..5d5cbc14e8 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
@@ -61,6 +61,7 @@ import org.apache.sysds.runtime.functionobjects.ValueFunction;
import org.apache.sysds.runtime.instructions.InstructionUtils;
import org.apache.sysds.runtime.instructions.cp.CM_COV_Object;
import org.apache.sysds.runtime.instructions.cp.KahanObject;
+import org.apache.sysds.runtime.matrix.data.MatrixValue.CellIndex;
import org.apache.sysds.runtime.matrix.operators.AggregateOperator;
import org.apache.sysds.runtime.matrix.operators.AggregateTernaryOperator;
import org.apache.sysds.runtime.matrix.operators.AggregateUnaryOperator;
@@ -206,6 +207,24 @@ public class LibMatrixAgg {
}
+ public static MatrixBlock aggregateUnaryMatrix(AggregateUnaryOperator
op,MatrixBlock in, MatrixValue result,
+ int blen, MatrixIndexes indexesIn, boolean inCP){
+
+ MatrixBlock ret = LibMatrixAgg.prepareAggregateUnaryOutput(in,
op, result, blen);
+
+ if( LibMatrixAgg.isSupportedUnaryAggregateOperator(op) ) {
+ LibMatrixAgg.aggregateUnaryMatrix(in, ret, op,
op.getNumThreads());
+ LibMatrixAgg.recomputeIndexes(ret, op, blen, indexesIn);
+ }
+ else
+ LibMatrixAggUnarySpecialization.aggregateUnary(in, op,
ret, blen, indexesIn);
+
+ if(op.aggOp.existsCorrection() && inCP)
+ ret.dropLastRowsOrColumns(op.aggOp.correction);
+
+ return ret;
+ }
+
public static void aggregateUnaryMatrix(MatrixBlock in, MatrixBlock
out, AggregateUnaryOperator uaop) {
AggType aggtype = getAggType(uaop);
@@ -3672,6 +3691,47 @@ public class LibMatrixAgg {
}
+ public static MatrixBlock prepareAggregateUnaryOutput(MatrixBlock in,
AggregateUnaryOperator op, MatrixValue result, int blen){
+ CellIndex tempCellIndex = n
(systemds) 02/02: [MINOR] Change CLA to normal SUM
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit ab4ec284b9dbe320087c9108c041ebdeccc23282
Author: Sebastian Baunsgaard
AuthorDate: Sun Jan 7 19:31:20 2024 +0100
[MINOR] Change CLA to normal SUM
This commit change CLA to utilize the recently committed SUM operation
without KAHAN. This commit also modify the block size for the
parallelization to improve performance over a number of test files.
Closes #1977
---
.../sysds/runtime/compress/lib/CLALibCompAgg.java | 53 --
1 file changed, 29 insertions(+), 24 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibCompAgg.java
b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibCompAgg.java
index 95a460a2e0..999c95d54f 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibCompAgg.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibCompAgg.java
@@ -31,6 +31,7 @@ import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.sysds.api.DMLScript;
+import org.apache.sysds.common.Types.CorrectionLocationType;
import org.apache.sysds.runtime.DMLRuntimeException;
import org.apache.sysds.runtime.compress.CompressedMatrixBlock;
import org.apache.sysds.runtime.compress.CompressionSettings;
@@ -214,7 +215,7 @@ public final class CLALibCompAgg {
private static AggregateUnaryOperator
replaceKahnOperations(AggregateUnaryOperator op) {
if(op.aggOp.increOp.fn instanceof KahanPlus)
- return new AggregateUnaryOperator(new
AggregateOperator(0, Plus.getPlusFnObject()), op.indexFn,
+ return new AggregateUnaryOperator(new
AggregateOperator(0, Plus.getPlusFnObject(), CorrectionLocationType.NONE),
op.indexFn,
op.getNumThreads());
return op;
}
@@ -224,7 +225,7 @@ public final class CLALibCompAgg {
int k = op.getNumThreads();
// replace mean operation with plus.
AggregateUnaryOperator opm = (op.aggOp.increOp.fn instanceof
Mean) ? new AggregateUnaryOperator(
- new AggregateOperator(0, Plus.getPlusFnObject()),
op.indexFn) : op;
+ new AggregateOperator(0, Plus.getPlusFnObject(),
CorrectionLocationType.NONE), op.indexFn) : op;
if(isValidForParallelProcessing(m, op))
aggregateInParallel(m, o, opm, k);
@@ -415,7 +416,7 @@ public final class CLALibCompAgg {
final ArrayList tasks = new ArrayList<>();
final int nCol = m1.getNumColumns();
final int nRow = m1.getNumRows();
- final int blklen = Math.max(512, nRow / k);
+ final int blklen = Math.max(64, nRow / k);
final List groups = m1.getColGroups();
final boolean shouldFilter =
CLALibUtils.shouldPreFilter(groups);
if(shouldFilter) {
@@ -568,7 +569,7 @@ public final class CLALibCompAgg {
_op = op;
_rl = rl;
_ru = ru;
- _blklen = Math.max(65536 / ret.getNumColumns() /
filteredGroups.size(), 64);
+ _blklen = Math.max(16384 / nCol, 64);
_ret = ret;
_nCol = nCol;
}
@@ -581,7 +582,6 @@ public final class CLALibCompAgg {
private MatrixBlock decompressToTemp(MatrixBlock tmp, int rl,
int ru, AIterator[] its) {
Timing time = new Timing(true);
-
DenseBlock db = tmp.getDenseBlock();
for(int i = 0; i < _groups.size(); i++) {
AColGroup g = _groups.get(i);
@@ -619,12 +619,34 @@ public final class CLALibCompAgg {
for(int i = 0; i < _groups.size(); i++)
if(_groups.get(i) instanceof ASDCZero)
its[i] = ((ASDCZero)
_groups.get(i)).getIterator(_rl);
- if(_op.indexFn instanceof ReduceCol) {
+
+ if(_op.indexFn instanceof ReduceCol) { // row aggregates
+ reduceCol(tmp, its, isBinaryOp);
+ return null;
+ }
+ else if(_op.indexFn instanceof ReduceAll) {
+ decompressToTemp(tmp, _rl, _ru, its);
+ MatrixBlock outputBlock =
LibMatrixAgg.prepareAggregateUnaryOutput(tmp, _op, null, 1000);
+ LibMatrixAgg.aggregateUnaryMatrix(tmp,
outp
(systemds) branch main updated (b420bdf68c -> ab4ec284b9)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from b420bdf68c [MINOR] Matrix equals empty support new ccb589056c [MINOR] MatrixBlock improved generic Unary Agg new ab4ec284b9 [MINOR] Change CLA to normal SUM The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../sysds/runtime/compress/lib/CLALibCompAgg.java | 53 +++--- .../spark/AggregateUnarySPInstruction.java | 2 +- .../sysds/runtime/matrix/data/CM_N_COVCell.java| 6 - .../sysds/runtime/matrix/data/LibMatrixAgg.java| 60 +++ .../data/LibMatrixAggUnarySpecialization.java | 152 .../sysds/runtime/matrix/data/MatrixBlock.java | 191 ++--- .../sysds/runtime/matrix/data/MatrixCell.java | 39 ++--- .../sysds/runtime/matrix/data/MatrixValue.java | 6 +- .../sysds/runtime/matrix/data/WeightedCell.java| 9 +- 9 files changed, 275 insertions(+), 243 deletions(-) create mode 100644 src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAggUnarySpecialization.java
(systemds) branch main updated: [MINOR] Matrix equals empty support
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new b420bdf68c [MINOR] Matrix equals empty support
b420bdf68c is described below
commit b420bdf68caa5e7d109b4ac9901ac912fe9adade
Author: Sebastian Baunsgaard
AuthorDate: Sun Jan 7 16:38:25 2024 +0100
[MINOR] Matrix equals empty support
This commit makes minor improvements to the matrix equals operation.
If someone reads this is it possible to compare MatrixBlock via
a.equals(b) where a and b are MatrixBlocks internally.
The update fixes an edge case of empty MatrixBlock with unknown non zero
count.
Closes #1978
---
.../sysds/runtime/matrix/data/LibMatrixEquals.java | 41 --
.../sysds/runtime/matrix/data/MatrixBlock.java | 20 ++-
2 files changed, 27 insertions(+), 34 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixEquals.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixEquals.java
index 63536d4c04..39e5a43980 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixEquals.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixEquals.java
@@ -25,7 +25,7 @@ import org.apache.commons.logging.LogFactory;
/**
*
*
- * Equals library for MatrixBLocks:
+ * Equals library for MatrixBlocks:
*
*
*
@@ -39,6 +39,10 @@ import org.apache.commons.logging.LogFactory;
* Consistent
*
*
+ *
+ * The equals also is valid if the metadata of number of non zeros are unknown
in either input. An unknown number of non
+ * zero values is indicated by a negative nonzero count in the input matrices.
+ *
*/
public class LibMatrixEquals {
@@ -49,7 +53,7 @@ public class LibMatrixEquals {
private final MatrixBlock a;
/** second block */
private final MatrixBlock b;
- /** Epsilon */
+ /** Epsilon allowed between the blocks */
private final double eps;
/**
@@ -140,19 +144,20 @@ public class LibMatrixEquals {
* @return if the blocks are equivalent
*/
private boolean exec() {
+
if(isMetadataDifferent())
return false;
- Boolean empty = isEmpty();
- if(empty != null)
- return empty;
-
- if(a.denseBlock != null && b.denseBlock != null)
+ else if(a.isEmpty() && b.nonZeros != -1)
+ return b.isEmpty();
+ else if(b.isEmpty() && a.nonZeros != -1)
+ return false;
+ else if(a.denseBlock != null && b.denseBlock != null)
return a.denseBlock.equals(b.denseBlock, eps);
- if(a.sparseBlock != null && b.sparseBlock != null)
+ else if(a.sparseBlock != null && b.sparseBlock != null)
return a.sparseBlock.equals(b.sparseBlock, eps);
- if(a.sparseBlock != null && b.denseBlock != null &&
b.denseBlock.isContiguous())
+ else if(a.sparseBlock != null && b.denseBlock != null &&
b.denseBlock.isContiguous())
return a.sparseBlock.equals(b.denseBlock.values(0),
b.getNumColumns(), eps);
- if(b.sparseBlock != null && a.denseBlock != null &&
a.denseBlock.isContiguous())
+ else if(b.sparseBlock != null && a.denseBlock != null &&
a.denseBlock.isContiguous())
return b.sparseBlock.equals(a.denseBlock.values(0),
a.getNumColumns(), eps);
return genericEquals();
@@ -177,22 +182,6 @@ public class LibMatrixEquals {
return diff;
}
- /**
-* Empty metadata check. to verify if the content is empty and such.
-*
-* @return Boolean that is not null if something was found otherwise
null.
-*/
- private Boolean isEmpty() {
- final boolean emptyA = a.isEmpty();
- final boolean emptyB = b.isEmpty();
- // empty cases!
- if(emptyA != emptyB)
- return false;
- else if(emptyA)
- return true;
- return null;
- }
-
/**
* Generic implementation to cover all cases. But it is slow in most.
*
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
index 2995b15efb..276f1aacee 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/MatrixBlock.java
@@ -587,15 +587,19
(systemds) branch main updated: [MINOR] log4j prop Python systemds Context
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 3b18f06fc8 [MINOR] log4j prop Python systemds Context
3b18f06fc8 is described below
commit 3b18f06fc850f659929ad9a04dbc2e19933a4677
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 18:42:16 2024 +0100
[MINOR] log4j prop Python systemds Context
Default to LOG4JPROP environment variable for the log file settings
in python
Closes #1976
---
.../python/systemds/context/systemds_context.py | 21 -
1 file changed, 12 insertions(+), 9 deletions(-)
diff --git a/src/main/python/systemds/context/systemds_context.py
b/src/main/python/systemds/context/systemds_context.py
index 4cbc6a464d..5f34086807 100644
--- a/src/main/python/systemds/context/systemds_context.py
+++ b/src/main/python/systemds/context/systemds_context.py
@@ -183,16 +183,19 @@ class SystemDSContext(object):
command.append(classpath)
-files = glob(os.path.join(root, "conf", "log4j*.properties"))
-if len(files) > 1:
-self._log.warning(
-"Multiple logging files found selecting: " + files[0])
-if len(files) == 0:
-self._log.warning("No log4j file found at: "
- + os.path.join(root, "conf")
- + " therefore using default settings")
+if os.environ.get("LOG4JPROP") == None:
+files = glob(os.path.join(root, "conf", "log4j*.properties"))
+if len(files) > 1:
+self._log.warning(
+"Multiple logging files found selecting: " + files[0])
+if len(files) == 0:
+self._log.warning("No log4j file found at: "
+ + os.path.join(root, "conf")
+ + " therefore using default settings")
+else:
+command.append("-Dlog4j.configuration=file:" + files[0])
else:
-command.append("-Dlog4j.configuration=file:" + files[0])
+command.append("-Dlog4j.configuration=file:"
+os.environ.get("LOG4JPROP"))
command.append("org.apache.sysds.api.PythonDMLScript")
(systemds) branch main updated: [MINOR] Matrix Transpose optimizations
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 736060dcc6 [MINOR] Matrix Transpose optimizations
736060dcc6 is described below
commit 736060dcc64dafaae503e4c8ffaecfe4567b8b7b
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 17:32:41 2024 +0100
[MINOR] Matrix Transpose optimizations
Optimize direct access to underlying sparse block in transpose of
sparse blocks.
Closes #1974
---
.../sysds/runtime/matrix/data/LibMatrixReorg.java | 295 +++--
.../sysds/runtime/matrix/data/MatrixBlock.java | 9 +-
2 files changed, 225 insertions(+), 79 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixReorg.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixReorg.java
index ef28846084..7ad5fdc2bd 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixReorg.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixReorg.java
@@ -46,6 +46,7 @@ import org.apache.sysds.runtime.data.DenseBlockFactory;
import org.apache.sysds.runtime.data.SparseBlock;
import org.apache.sysds.runtime.data.SparseBlockCSR;
import org.apache.sysds.runtime.data.SparseBlockMCSR;
+import org.apache.sysds.runtime.data.SparseRow;
import org.apache.sysds.runtime.data.SparseRowVector;
import org.apache.sysds.runtime.functionobjects.DiagIndex;
import org.apache.sysds.runtime.functionobjects.RevIndex;
@@ -246,8 +247,8 @@ public class LibMatrixReorg {
allowCSR = allowCSR && (in.clen <= 4096 || out.nonZeros <
1000);
int[] cnt = null;
+ final ExecutorService pool = CommonThreadPool.get(k);
try {
- final ExecutorService pool = CommonThreadPool.get(k);
if(out.sparse && allowCSR) {
final int size = (int) out.nonZeros;
final Future f = countNNZColumns(in, k,
pool);
@@ -273,27 +274,42 @@ public class LibMatrixReorg {
// compute actual transpose and check for errors
ArrayList tasks = new ArrayList<>();
- boolean row = (in.sparse || in.rlen >= in.clen) &&
!out.sparse;
+ boolean allowReturnBlock = out.sparse && in.sparse &&
in.rlen >= in.clen && cnt == null;
+ boolean row = (in.sparse || in.rlen >= in.clen) &&
(!out.sparse || allowReturnBlock);
int len = row ? in.rlen : in.clen;
int blklen = (int) (Math.ceil((double) len / k));
blklen += (!out.sparse && (blklen % 8) != 0) ? 8 -
blklen % 8 : 0;
blklen = (in.sparse) ? Math.max(blklen, 32) : blklen;
+
for(int i = 0; i < k & i * blklen < len; i++)
- tasks.add(new TransposeTask(in, out, row, i *
blklen, Math.min((i + 1) * blklen, len), cnt));
- List> taskret = pool.invokeAll(tasks);
- pool.shutdown();
- for(Future task : taskret)
- task.get();
+ tasks.add(new TransposeTask(in, out, row, i *
blklen, Math.min((i + 1) * blklen, len), cnt, allowReturnBlock));
+ List blocks = allowReturnBlock ? new
ArrayList<>(): null;
+ // List> taskret =
pool.invokeAll(tasks);
+ for(Future task : pool.invokeAll(tasks)){
+ MatrixBlock m = task.get();
+ if(allowReturnBlock && m != null)
+ blocks.add(m);
+ }
+
+ if(allowReturnBlock)
+ combine(blocks, out, row, k);
}
catch(Exception ex) {
throw new DMLRuntimeException(ex);
}
+ finally{
+ pool.shutdown();
+ }
// System.out.println("r' k="+k+" ("+in.rlen+", "+in.clen+",
"+in.sparse+", "+out.sparse+") in "+time.stop()+" ms.");
return out;
}
+ private static void combine(List blocks, MatrixBlock out,
boolean row, int k){
+ MatrixBlock.append(blocks, out, row, k);
+ }
+
public static Future countNNZColumns(MatrixBlock in, int k,
ExecutorService pool)
throws InterruptedException, ExecutionEx
(systemds) branch main updated: [MINOR] Python set log4j
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new a83a17acbf [MINOR] Python set log4j
a83a17acbf is described below
commit a83a17acbfd44d7c5ba4de1a318e2a50f7d7628d
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 18:16:32 2024 +0100
[MINOR] Python set log4j
---
.github/workflows/python.yml | 2 ++
.../python/systemds/operator/algorithm/__init__.py | 6 +
.../systemds/operator/algorithm/builtin/auc.py | 2 +-
.../builtin/{auc.py => img_rotate_linearized.py} | 27 +++-
.../{auc.py => img_sample_pairing_linearized.py} | 25 ++-
.../builtin/{auc.py => img_shear_linearized.py}| 29 +-
6 files changed, 54 insertions(+), 37 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index 8b345cf79d..6e56dc812e 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -112,6 +112,7 @@ jobs:
export SYSTEMDS_ROOT=$(pwd)
export PATH=$SYSTEMDS_ROOT/bin:$PATH
export SYSDS_QUIET=1
+export LOG4JPROP=$SYSTEMDS_ROOT/src/test/resources/log4j.properties
cd src/main/python
unittest-parallel -t . -s tests
# python -m unittest discover -s tests -p 'test_*.py'
@@ -119,6 +120,7 @@ jobs:
- name: Run all python tests no environment
run: |
+export LOG4JPROP=$(pwd)/src/test/resources/log4j.properties
cd src/main/python
unittest-parallel -t . -s tests
# python -m unittest discover -s tests -p 'test_*.py'
diff --git a/src/main/python/systemds/operator/algorithm/__init__.py
b/src/main/python/systemds/operator/algorithm/__init__.py
index 52c470d201..690bfe07e8 100644
--- a/src/main/python/systemds/operator/algorithm/__init__.py
+++ b/src/main/python/systemds/operator/algorithm/__init__.py
@@ -90,8 +90,11 @@ from .builtin.img_mirror_linearized import
img_mirror_linearized
from .builtin.img_posterize import img_posterize
from .builtin.img_posterize_linearized import img_posterize_linearized
from .builtin.img_rotate import img_rotate
+from .builtin.img_rotate_linearized import img_rotate_linearized
from .builtin.img_sample_pairing import img_sample_pairing
+from .builtin.img_sample_pairing_linearized import
img_sample_pairing_linearized
from .builtin.img_shear import img_shear
+from .builtin.img_shear_linearized import img_shear_linearized
from .builtin.img_transform import img_transform
from .builtin.img_transform_linearized import img_transform_linearized
from .builtin.img_translate import img_translate
@@ -263,8 +266,11 @@ __all__ = ['WoE',
'img_posterize',
'img_posterize_linearized',
'img_rotate',
+ 'img_rotate_linearized',
'img_sample_pairing',
+ 'img_sample_pairing_linearized',
'img_shear',
+ 'img_shear_linearized',
'img_transform',
'img_transform_linearized',
'img_translate',
diff --git a/src/main/python/systemds/operator/algorithm/builtin/auc.py
b/src/main/python/systemds/operator/algorithm/builtin/auc.py
index 8df6835311..b5b3b67e7d 100644
--- a/src/main/python/systemds/operator/algorithm/builtin/auc.py
+++ b/src/main/python/systemds/operator/algorithm/builtin/auc.py
@@ -32,7 +32,7 @@ from systemds.utils.consts import VALID_INPUT_TYPES
def auc(Y: Matrix,
P: Matrix):
"""
- This builting function computes the area under the ROC curve (AUC)
+ This builtin function computes the area under the ROC curve (AUC)
for binary classifiers.
diff --git a/src/main/python/systemds/operator/algorithm/builtin/auc.py
b/src/main/python/systemds/operator/algorithm/builtin/img_rotate_linearized.py
similarity index 58%
copy from src/main/python/systemds/operator/algorithm/builtin/auc.py
copy to
src/main/python/systemds/operator/algorithm/builtin/img_rotate_linearized.py
index 8df6835311..f3698c93dd 100644
--- a/src/main/python/systemds/operator/algorithm/builtin/auc.py
+++
b/src/main/python/systemds/operator/algorithm/builtin/img_rotate_linearized.py
@@ -20,7 +20,7 @@
# -
# Autogenerated By : src/main/python/generator/generator.py
-# Autogenerated From : scripts/builtin/auc.dml
+# Autogenerated From : scripts/builtin/img_rotate_linearized.dml
from typing import Dict, Iterable
@@ -29,21 +29,24 @@ from systemds.script_building.dag import OutputType
from systemds.utils.consts import VALID_INPUT_TYPES
-def auc(Y: Matrix,
-P: Matrix):
+def img_rotate_linearized(img_in: Matrix,
+ radians: float,
+ fill_value: float,
+ s_cols: int,
+ s_rows: int):
"""
- This builting function com
(systemds) branch main updated: [MINOR] Lazy write buffer optimization
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 23bcd6d2b2 [MINOR] Lazy write buffer optimization
23bcd6d2b2 is described below
commit 23bcd6d2b2be2739bf9abb137d82917860d3fd6e
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 16:45:34 2024 +0100
[MINOR] Lazy write buffer optimization
This commit optimize the lazy write buffer to pass through byte
arrays if provided instead of lazily evaulating them.
If provided byte arrays are large enough this is faster than the
previous lazy evaluation. Especially because we previously copied
over the byte array allocating the elements twice,
This commit also fixes a bug where if you provide a byte array that
is larger than the buffer it does not crash.
Closes #1972
---
.../controlprogram/caching/LazyWriteBuffer.java| 117 +++--
1 file changed, 85 insertions(+), 32 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/LazyWriteBuffer.java
b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/LazyWriteBuffer.java
index 8c4bfc310f..73c86f9edc 100644
---
a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/LazyWriteBuffer.java
+++
b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/LazyWriteBuffer.java
@@ -23,12 +23,19 @@ import java.io.IOException;
import java.util.Map.Entry;
import java.util.concurrent.ExecutorService;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
import org.apache.sysds.api.DMLScript;
import org.apache.sysds.hops.OptimizerUtils;
+import org.apache.sysds.runtime.data.SparseBlock.Type;
+import org.apache.sysds.runtime.data.SparseBlockFactory;
+import org.apache.sysds.runtime.data.SparseBlockMCSR;
+import org.apache.sysds.runtime.matrix.data.MatrixBlock;
import org.apache.sysds.runtime.util.LocalFileUtils;
-public class LazyWriteBuffer
-{
+public class LazyWriteBuffer {
+ protected static final Log LOG =
LogFactory.getLog(LazyWriteBuffer.class.getName());
+
public enum RPolicy {
FIFO, //first-in, first-out eviction
LRU //least recently used eviction
@@ -52,38 +59,28 @@ public class LazyWriteBuffer
{
//obtain basic meta data of cache block
long lSize = getCacheBlockSize(cb);
+
+ if(lSize > _limit){ // if this block goes above limit
+ cb = compact(cb); // try to compact it
+ lSize = getCacheBlockSize(cb); // and update to new
size of block
+ if(lSize > _limit){// if we are still above limit
+ reAllocate(lSize); // try to compact all blocks
in memory.
+ }
+ }
+
boolean requiresWrite = (lSize > _limit//global buffer
limit
|| !ByteBuffer.isValidCapacity(lSize, cb)); //local
buffer limit
int numEvicted = 0;
- //handle caching/eviction if it fits in writebuffer
- if( !requiresWrite )
- {
+ //handle caching/eviction if it fits in the write buffer
+ if(!requiresWrite) {
//create byte buffer handle (no block allocation yet)
ByteBuffer bbuff = new ByteBuffer( lSize );
- //modify buffer pool
- synchronized( _mQueue )
- {
- //evict matrices to make room (by default FIFO)
- while( _size+lSize > _limit &&
!_mQueue.isEmpty() )
- {
- //remove first entry from eviction queue
- Entry entry =
_mQueue.removeFirst();
- String ftmp = entry.getKey();
- ByteBuffer tmp = entry.getValue();
-
- if( tmp != null ) {
- //wait for pending serialization
- tmp.checkSerialized();
-
- //evict matrix
- tmp.evictBuffer(ftmp);
- tmp.freeMemory();
- _size -= tmp.getSize();
- numEvicted++;
- }
- }
+
(systemds) branch main updated: [MINOR] Sparse Block pushdown operations
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 91291b6029 [MINOR] Sparse Block pushdown operations
91291b6029 is described below
commit 91291b6029d22e964825be6bea35f6c134e34877
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 16:42:01 2024 +0100
[MINOR] Sparse Block pushdown operations
A few optimization blocks for allocating and appending to sparse blocks.
This commit does not use them, but simply adds the primitives to verify
that it does not break anything else.
Closes #1973
---
.../org/apache/sysds/runtime/data/DenseBlock.java | 4 +++
.../org/apache/sysds/runtime/data/SparseBlock.java | 3 ---
.../apache/sysds/runtime/data/SparseBlockMCSR.java | 25 +++--
.../org/apache/sysds/runtime/data/SparseRow.java | 13 -
.../apache/sysds/runtime/data/SparseRowScalar.java | 31 +-
.../apache/sysds/runtime/data/SparseRowVector.java | 30 ++---
6 files changed, 70 insertions(+), 36 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
b/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
index 64e3789d4a..037231fa0e 100644
--- a/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/data/DenseBlock.java
@@ -734,6 +734,10 @@ public abstract class DenseBlock implements Serializable,
Block
return true;
}
+ public void fill(double value){
+ reset(_odims, value);
+ }
+
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
diff --git a/src/main/java/org/apache/sysds/runtime/data/SparseBlock.java
b/src/main/java/org/apache/sysds/runtime/data/SparseBlock.java
index bc6d4727d1..cd1bd751f3 100644
--- a/src/main/java/org/apache/sysds/runtime/data/SparseBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/data/SparseBlock.java
@@ -424,9 +424,6 @@ public abstract class SparseBlock implements Serializable,
Block
/**
* Get values of row r in the format of a sparse row.
*
-* NOTE: This method exists for incremental runtime integration and
might
-* be deleted in the future.
-*
* @param r row index starting at 0
* @return values of row r as a sparse row
*/
diff --git a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
index 08dbc8b0a4..c7e79b8dbc 100644
--- a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
+++ b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
@@ -271,7 +271,7 @@ public class SparseBlockMCSR extends SparseBlock
@Override
public long size(int rl, int ru) {
- int ret = 0;
+ long ret = 0;
for( int i=rl; i= 0 )
- throw new RuntimeException(
- "Invalid append to sparse row scalar.");
- index = col;
- value = v;
+ return this;
+ else if( index >= 0 ){ // if already set
+ SparseRowVector srv = new SparseRowVector();
+ srv.append(index, value);
+ srv.append(col, v);
+ return srv;
+ }
+ else{
+ index = col;
+ value = v;
+ return this;
+ }
+
}
@Override
@@ -116,6 +123,16 @@ public final class SparseRowScalar extends SparseRow{
return value;
}
+ @Override
+ public int searchIndexesFirstGTE(int col) {
+ return col <= index ? 0 : -1;
+ }
+
+ @Override
+ public int searchIndexesFirstGT(int col) {
+ return col < index ? 0 : -1;
+ }
+
@Override
public SparseRow copy(boolean deep){
return new SparseRowScalar(index, value);
diff --git a/src/main/java/org/apache/sysds/runtime/data/SparseRowVector.java
b/src/main/java/org/apache/sysds/runtime/data/SparseRowVector.java
index 3e433f15fc..50229e15df 100644
--- a/src/main/java/org/apache/sysds/runtime/data/SparseRowVector.java
+++ b/src/main/java/org/apache/sysds/runtime/data/SparseRowVector.java
@@ -54,6 +54,7 @@ public final class SparseRowVector extends SparseRow {
}
public SparseRowVector(int capacity) {
+ capacity = Math.max(initialCapacity, capacity);
estimatedNzs = capacity;
values = new double[capacity];
indexes = new int[capacity];
@@ -78
(systemds) branch main updated: [SYSTEMDS-3153] Missing value imputation using KNN
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new d443178a0f [SYSTEMDS-3153] Missing value imputation using KNN
d443178a0f is described below
commit d443178a0fd3d341189c8be96abe7bce42870dd2
Author: Christina Dionysio
AuthorDate: Fri Jan 5 17:14:09 2024 +0100
[SYSTEMDS-3153] Missing value imputation using KNN
This commit adds a perf test case for missing value imputation using
KNN. It is integrated into our perf suite.
Closes #1943
---
scripts/perftest/KnnMissingValueImputation.sh | 54 +++
scripts/perftest/runAll.sh| 1 +
scripts/perftest/scripts/ImputeByKNN.dml | 52 ++
3 files changed, 107 insertions(+)
diff --git a/scripts/perftest/KnnMissingValueImputation.sh
b/scripts/perftest/KnnMissingValueImputation.sh
new file mode 100755
index 00..aa7bf04be7
--- /dev/null
+++ b/scripts/perftest/KnnMissingValueImputation.sh
@@ -0,0 +1,54 @@
+#!/usr/bin/env bash
+#-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-
+
+CMD=$1
+MAXMEM=$2
+
+echo "KNN MISSING VALUE IMPUTATION" >>results/times.txt
+
+mkdir -p logs
+LogName='logs/KnnMissingValueImputation.log'
+rm -f $LogName # full log file
+rm -f $LogName.log # Reduced log file
+
+is=("1000 1 10 100 1000")
+
+for i in $is; do
+ for method in "dist" "dist_missing" "dist_sample"; do
+if [ $(((i*i*8)/10**6)) -gt $MAXMEM ] && [ $method == "dist" ]; then
+ continue;
+elif [ $(((i*9*i*8/100)/10**6)) -gt $MAXMEM ] && [ $method ==
"dist_missing" ]; then
+ continue;
+fi
+
+tstart=$(date +%s.%N)
+${CMD} -f ./scripts/ImputeByKNN.dml \
+--config conf/SystemDS-config.xml \
+--stats \
+--nvargs num_rows=$i method=$method max_mem=$MAXMEM \
+>>$LogName 2>&1
+ttrain=$(echo "$(date +%s.%N) - $tstart - .4" | bc)
+echo "KNN Missing Value Imputation $i rows, $method method:" $ttrain
>>results/times.txt
+ done
+done
+
+echo -e "\n\n" >>results/times.txt
\ No newline at end of file
diff --git a/scripts/perftest/runAll.sh b/scripts/perftest/runAll.sh
index 9b20606c1d..6d39043a74 100755
--- a/scripts/perftest/runAll.sh
+++ b/scripts/perftest/runAll.sh
@@ -126,6 +126,7 @@ echo -e "\n\n" >> results/times.txt
./runAllClustering.sh ${CMD} ${TEMPFOLDER} ${MAXMEM}
./runAllDimensionReduction.sh ${CMD} ${TEMPFOLDER} ${MAXMEM}
./runAllALS.sh ${CMD} ${TEMPFOLDER} ${MAXMEM}
+./KnnMissingValueImputation.sh ${CMD} ${MAXMEM}
### IO Benchmarks:
./runAllIO.sh ${CMD} ${TEMPFOLDER} ${MAXMEM}
diff --git a/scripts/perftest/scripts/ImputeByKNN.dml
b/scripts/perftest/scripts/ImputeByKNN.dml
new file mode 100755
index 00..0ec2ef6af8
--- /dev/null
+++ b/scripts/perftest/scripts/ImputeByKNN.dml
@@ -0,0 +1,52 @@
+#-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-
+
+max_mem = $max_m
(systemds) branch main updated: [MINOR] Vectorized string memory cost
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 7ecdb38a20 [MINOR] Vectorized string memory cost
7ecdb38a20 is described below
commit 7ecdb38a20d34ee095fcd7cfc07e4d754f82d18d
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 17:06:52 2024 +0100
[MINOR] Vectorized string memory cost
---
.../org/apache/sysds/utils/MemoryEstimates.java| 26 --
1 file changed, 24 insertions(+), 2 deletions(-)
diff --git a/src/main/java/org/apache/sysds/utils/MemoryEstimates.java
b/src/main/java/org/apache/sysds/utils/MemoryEstimates.java
index 43e9fddd25..5fd8b26701 100644
--- a/src/main/java/org/apache/sysds/utils/MemoryEstimates.java
+++ b/src/main/java/org/apache/sysds/utils/MemoryEstimates.java
@@ -189,12 +189,34 @@ public class MemoryEstimates {
* @return The array memory cost
*/
public static final double stringArrayCost(String[] strings) {
- long size = 0;
- for(int i = 0; i < strings.length; i++)
+ double size = 0;
+ int i = 0;
+ int by8 = strings.length - strings.length %8 ;
+ for(;i < by8; i+= 8)
+ size += stringArrayCostVec8(strings, i);
+ for(; i < strings.length; i++)
size += stringCost(strings[i]);
return size;
}
+ private static final double stringArrayCostVec8(String[] strings, int
r){
+ long size = 0;
+ size += stringCost(strings[r]);
+ size += stringCost(strings[r+1]);
+ size += stringCost(strings[r+2]);
+ size += stringCost(strings[r+3]);
+ size += stringCost(strings[r+4]);
+ size += stringCost(strings[r+5]);
+ size += stringCost(strings[r+6]);
+ size += stringCost(strings[r+7]);
+ return size;
+ }
+
+ public static final double stringArrayCost(int length, int
avgStringLength){
+ // if null 16 object + 8 array ref
+ return stringCost(avgStringLength) * length + 24.0d;
+ }
+
/**
* Get the worst case memory usage of a single string.
*
(systemds) branch main updated: [SYSTEMDS-3662] Parfor Merge Sparse
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new fdd60f6d10 [SYSTEMDS-3662] Parfor Merge Sparse
fdd60f6d10 is described below
commit fdd60f6d10acb72239fafc7507bd62b73941153f
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 13:06:58 2024 +0100
[SYSTEMDS-3662] Parfor Merge Sparse
This commit optimize the parfor merge.
In the case of Kmeans with 10 runs it optimize the merge phase from
19 to 1 sec because it exploits the sparsity of the merging blocks.
Closes #1971
---
.../controlprogram/parfor/ResultMergeMatrix.java | 192 +++--
1 file changed, 143 insertions(+), 49 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/ResultMergeMatrix.java
b/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/ResultMergeMatrix.java
index 6e0a3c4d0c..d90b9e177b 100644
---
a/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/ResultMergeMatrix.java
+++
b/src/main/java/org/apache/sysds/runtime/controlprogram/parfor/ResultMergeMatrix.java
@@ -22,37 +22,51 @@ package org.apache.sysds.runtime.controlprogram.parfor;
import java.util.List;
import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.compress.utils.Util;
import org.apache.sysds.runtime.controlprogram.caching.MatrixObject;
import org.apache.sysds.runtime.data.DenseBlock;
+import org.apache.sysds.runtime.data.SparseBlock;
import org.apache.sysds.runtime.matrix.data.MatrixBlock;
/**
+ *
* Due to independence of all iterations, any result has the following
properties:
- * (1) non local var, (2) matrix object, and (3) completely independent.
- * These properties allow us to realize result merging in parallel without any
synchronization.
+ *
*
+ *
+ * (1) non local var,
+ *
+ *
+ * (2) matrix object, and
+ *
+ *
+ * (3) completely independent.
+ *
+ *
+ *
+ * These properties allow us to realize result merging in parallel without any
synchronization.
+ *
*/
-public abstract class ResultMergeMatrix extends ResultMerge
-{
+public abstract class ResultMergeMatrix extends ResultMerge {
private static final long serialVersionUID = 5319002218804570071L;
-
+
public ResultMergeMatrix() {
super();
}
-
+
public ResultMergeMatrix(MatrixObject out, MatrixObject[] in, String
outputFilename, boolean accum) {
super(out, in, outputFilename, accum);
}
-
- protected void mergeWithoutComp( MatrixBlock out, MatrixBlock in,
boolean appendOnly ) {
+
+ protected void mergeWithoutComp(MatrixBlock out, MatrixBlock in,
boolean appendOnly) {
mergeWithoutComp(out, in, appendOnly, false);
}
-
- protected void mergeWithoutComp( MatrixBlock out, MatrixBlock in,
boolean appendOnly, boolean par ) {
- //pass through to matrix block operations
- if( _isAccum )
+
+ protected void mergeWithoutComp(MatrixBlock out, MatrixBlock in,
boolean appendOnly, boolean par) {
+ // pass through to matrix block operations
+ if(_isAccum)
out.binaryOperationsInPlace(PLUS, in);
- else{
+ else {
MatrixBlock out2 = out.merge(in, appendOnly, par);
if(out2 != out)
@@ -61,52 +75,132 @@ public abstract class ResultMergeMatrix extends
ResultMerge
}
/**
-* NOTE: append only not applicable for wiht compare because output
must be populated with
-* initial state of matrix - with append, this would result in
duplicates.
+* NOTE: append only not applicable for with compare because output
must be populated with initial state of matrix -
+* with append, this would result in duplicates.
*
-* @param out output matrix block
-* @param in input matrix block
-* @param compare ?
+* @param out output matrix block
+* @param in input matrix block
+* @param compare Comparison matrix of old values.
*/
- protected void mergeWithComp( MatrixBlock out, MatrixBlock in,
DenseBlock compare )
- {
- //Notes for result correctness:
- // * Always iterate over entire block in order to compare all
values
- // (using sparse iterator would miss values set to 0)
+ protected void mergeWithComp(MatrixBlock out, MatrixBlock in,
DenseBlock compare) {
+ // Notes for result correctness:
+ // * Always iterate over entire block in order to compare all
values
+ // (using sparse iterator would miss values set to 0)
// * Explicit NaN awareness because
(systemds) branch main updated: [SYSTEMDS-3592] Frame Compress Sample based
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new b3aac0d95b [SYSTEMDS-3592] Frame Compress Sample based
b3aac0d95b is described below
commit b3aac0d95b9e624c0122a69441f9d7c4e02d0296
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 12:48:09 2024 +0100
[SYSTEMDS-3592] Frame Compress Sample based
This commit change the frame compression to be sample based,
it also change the detect schema back to be sample based.
Closes #1970
---
.../runtime/frame/data/columns/ABooleanArray.java | 18 +++
.../sysds/runtime/frame/data/columns/Array.java| 124 ++--
.../runtime/frame/data/columns/ArrayFactory.java | 9 +-
.../runtime/frame/data/columns/BitSetArray.java| 4 +-
.../runtime/frame/data/columns/BooleanArray.java | 8 +-
.../runtime/frame/data/columns/CharArray.java | 6 +-
.../sysds/runtime/frame/data/columns/DDCArray.java | 165 -
.../runtime/frame/data/columns/DoubleArray.java| 10 +-
.../runtime/frame/data/columns/FloatArray.java | 21 ++-
.../runtime/frame/data/columns/HashLongArray.java | 53 ++-
.../runtime/frame/data/columns/IntegerArray.java | 8 +-
.../runtime/frame/data/columns/LongArray.java | 8 +-
.../runtime/frame/data/columns/OptionalArray.java | 95 +++-
.../runtime/frame/data/columns/RaggedArray.java| 4 +-
.../runtime/frame/data/columns/StringArray.java| 87 +--
.../data/compress/ArrayCompressionStatistics.java | 12 +-
.../data/compress/CompressedFrameBlockFactory.java | 28 ++--
.../frame/data/lib/FrameLibApplySchema.java| 14 +-
.../frame/data/lib/FrameLibDetectSchema.java | 25 +++-
.../sysds/runtime/frame/data/lib/FrameUtil.java| 4 +-
.../component/frame/FrameSerializationTest.java| 5 +
.../sysds/test/component/frame/FrameUtilTest.java | 92
.../component/frame/array/CustomArrayTests.java| 20 ++-
.../component/frame/array/FrameArrayTests.java | 7 +-
.../frame/compress/FrameCompressTest.java | 17 +++
.../frame/compress/FrameCompressTestUtils.java | 8 +-
26 files changed, 663 insertions(+), 189 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/columns/ABooleanArray.java
b/src/main/java/org/apache/sysds/runtime/frame/data/columns/ABooleanArray.java
index 206a0722d7..6d2f28d3dd 100644
---
a/src/main/java/org/apache/sysds/runtime/frame/data/columns/ABooleanArray.java
+++
b/src/main/java/org/apache/sysds/runtime/frame/data/columns/ABooleanArray.java
@@ -19,6 +19,9 @@
package org.apache.sysds.runtime.frame.data.columns;
+import java.util.HashMap;
+import java.util.Map;
+
public abstract class ABooleanArray extends Array {
public ABooleanArray(int size) {
@@ -43,4 +46,19 @@ public abstract class ABooleanArray extends Array {
public boolean possiblyContainsNaN(){
return false;
}
+
+ @Override
+ protected Map createRecodeMap() {
+ Map map = new HashMap<>();
+ long id = 1;
+ for(int i = 0; i < size() && id <= 2; i++) {
+ Boolean val = get(i);
+ if(val != null) {
+ Long v = map.putIfAbsent(val, id);
+ if(v == null)
+ id++;
+ }
+ }
+ return map;
+ }
}
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/columns/Array.java
b/src/main/java/org/apache/sysds/runtime/frame/data/columns/Array.java
index 11accc814b..d2021872ba 100644
--- a/src/main/java/org/apache/sysds/runtime/frame/data/columns/Array.java
+++ b/src/main/java/org/apache/sysds/runtime/frame/data/columns/Array.java
@@ -31,6 +31,8 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.io.Writable;
import org.apache.sysds.common.Types.ValueType;
import org.apache.sysds.runtime.DMLRuntimeException;
+import org.apache.sysds.runtime.compress.colgroup.mapping.AMapToData;
+import org.apache.sysds.runtime.compress.colgroup.mapping.MapToFactory;
import org.apache.sysds.runtime.compress.estim.sample.SampleEstimatorFactory;
import org.apache.sysds.runtime.frame.data.columns.ArrayFactory.FrameArrayType;
import org.apache.sysds.runtime.frame.data.compress.ArrayCompressionStatistics;
@@ -79,7 +81,8 @@ public abstract class Array implements Writable {
/**
* Get a recode map that maps each unique value in the array, to a long
ID. Null values are ignored, and not included
-* in the mapping. The resulting recode map in stored in a soft
reference to speed up repeated calls to the same column.
+* in the mapping. The resulting
(systemds) branch main updated: [MINOR] Split lineage and count distinct GitHub Actions
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new a9c29800e1 [MINOR] Split lineage and count distinct GitHub Actions a9c29800e1 is described below commit a9c29800e19d4a18d57113334c6fa7c30d9fc126 Author: Sebastian Baunsgaard AuthorDate: Fri Jan 5 13:12:59 2024 +0100 [MINOR] Split lineage and count distinct GitHub Actions --- .github/workflows/javaTests.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/javaTests.yml b/.github/workflows/javaTests.yml index 9768d2fb5a..ff151565e2 100644 --- a/.github/workflows/javaTests.yml +++ b/.github/workflows/javaTests.yml @@ -57,7 +57,8 @@ jobs: "**.component.p**.**,**.component.t**.**", "**.functions.a**.**,**.functions.binary.matrix.**,**.functions.binary.scalar.**,**.functions.binary.tensor.**", "**.functions.blocks.**,**.functions.data.rand.**,", - "**.functions.countDistinct.**,**.functions.countDistinctApprox.**,**.functions.data.misc.**,**.functions.lineage.**", + "**.functions.countDistinct.**,**.functions.countDistinctApprox.**", + "**.functions.data.misc.**,**.functions.lineage.**", "**.functions.compress.**,**.functions.data.tensor.**,**.functions.codegenalg.parttwo.**,**.functions.codegen.**,**.functions.caching.**", "**.functions.binary.matrix_full_cellwise.**,**.functions.binary.matrix_full_other.**", "**.functions.federated.algorithms.**,**.functions.federated.io.**,**.functions.federated.paramserv.**",
(systemds) branch main updated: [MINOR] LibMatrixAgg sum operator without KAHAN
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new aefed8f945 [MINOR] LibMatrixAgg sum operator without KAHAN
aefed8f945 is described below
commit aefed8f9456d4da52f67849f2056d3f614678ecd
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 13:01:45 2024 +0100
[MINOR] LibMatrixAgg sum operator without KAHAN
---
.../sysds/runtime/matrix/data/LibMatrixAgg.java| 205 +
1 file changed, 165 insertions(+), 40 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
index 70ee962162..0891d7f1ae 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixAgg.java
@@ -28,6 +28,7 @@ import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Future;
+import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.sysds.common.Types.CorrectionLocationType;
@@ -51,6 +52,7 @@ import org.apache.sysds.runtime.functionobjects.KahanPlus;
import org.apache.sysds.runtime.functionobjects.KahanPlusSq;
import org.apache.sysds.runtime.functionobjects.Mean;
import org.apache.sysds.runtime.functionobjects.Multiply;
+import org.apache.sysds.runtime.functionobjects.Plus;
import org.apache.sysds.runtime.functionobjects.ReduceAll;
import org.apache.sysds.runtime.functionobjects.ReduceCol;
import org.apache.sysds.runtime.functionobjects.ReduceDiag;
@@ -106,6 +108,8 @@ public class LibMatrixAgg {
private enum AggType {
KAHAN_SUM,
KAHAN_SUM_SQ,
+ SUM,
+ SUM_SQ,
CUM_KAHAN_SUM,
CUM_MIN,
CUM_MAX,
@@ -686,10 +690,12 @@ public class LibMatrixAgg {
return AggType.KAHAN_SUM_SQ;
}
+ final boolean rAll_rCol_rRow = ifn instanceof ReduceAll || ifn
instanceof ReduceCol || ifn instanceof ReduceRow;
+
//mean
if( vfn instanceof Mean
&& (op.aggOp.correction ==
CorrectionLocationType.LASTTWOCOLUMNS || op.aggOp.correction ==
CorrectionLocationType.LASTTWOROWS)
- && (ifn instanceof ReduceAll || ifn instanceof
ReduceCol || ifn instanceof ReduceRow) )
+ && rAll_rCol_rRow )
{
return AggType.MEAN;
}
@@ -699,22 +705,20 @@ public class LibMatrixAgg {
&& ((CM) vfn).getAggOpType() ==
AggregateOperationTypes.VARIANCE
&& (op.aggOp.correction ==
CorrectionLocationType.LASTFOURCOLUMNS ||
op.aggOp.correction ==
CorrectionLocationType.LASTFOURROWS)
- && (ifn instanceof ReduceAll || ifn instanceof
ReduceCol || ifn instanceof ReduceRow) )
+ && rAll_rCol_rRow )
{
return AggType.VAR;
}
//prod
- if( vfn instanceof Multiply
- && (ifn instanceof ReduceAll || ifn instanceof
ReduceCol || ifn instanceof ReduceRow))
- {
+ if(vfn instanceof Multiply && rAll_rCol_rRow)
return AggType.PROD;
- }
- //min / max
- if( vfn instanceof Builtin &&
- (ifn instanceof ReduceAll || ifn instanceof ReduceCol ||
ifn instanceof ReduceRow) )
- {
+ if(vfn instanceof Plus && rAll_rCol_rRow)
+ return AggType.SUM;
+
+ // min / max
+ if(vfn instanceof Builtin && rAll_rCol_rRow) {
BuiltinCode bfcode = ((Builtin)vfn).bFunc;
switch( bfcode ){
case MAX: return AggType.MAX;
@@ -1470,6 +1474,19 @@ public class LibMatrixAgg {
d_uakptrace(a, c, n, kbuff,
(KahanPlus)vFn, rl, ru);
break;
}
+ case SUM:{
+ if(a instanceof DenseBlockFP64DEDUP)
+ throw new NotImplementedException();
+ else if(ixFn instanceof ReduceAll) // SUM
+ d_uap(a, c, n, rl, ru);
+ else if(ixF
(systemds) branch main updated: [MINOR] add boolean flag for binary operators
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 0702c5518f [MINOR] add boolean flag for binary operators
0702c5518f is described below
commit 0702c5518f8dd410fb7c0d122b2d457cc5f6effe
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 12:55:42 2024 +0100
[MINOR] add boolean flag for binary operators
---
.../java/org/apache/sysds/runtime/functionobjects/And.java | 5 +
.../org/apache/sysds/runtime/functionobjects/Equals.java | 5 +
.../apache/sysds/runtime/functionobjects/GreaterThan.java | 5 +
.../sysds/runtime/functionobjects/GreaterThanEquals.java | 6 ++
.../org/apache/sysds/runtime/functionobjects/LessThan.java | 5 +
.../sysds/runtime/functionobjects/LessThanEquals.java | 5 +
.../java/org/apache/sysds/runtime/functionobjects/Not.java | 5 +
.../apache/sysds/runtime/functionobjects/NotEquals.java| 5 +
.../java/org/apache/sysds/runtime/functionobjects/Or.java | 5 +
.../sysds/runtime/functionobjects/ValueFunction.java | 14 +++---
.../java/org/apache/sysds/runtime/functionobjects/Xor.java | 5 +
11 files changed, 62 insertions(+), 3 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/functionobjects/And.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/And.java
index 5ae5017c2f..027e470bb7 100644
--- a/src/main/java/org/apache/sysds/runtime/functionobjects/And.java
+++ b/src/main/java/org/apache/sysds/runtime/functionobjects/And.java
@@ -44,4 +44,9 @@ public class And extends ValueFunction
public double execute(double in1, double in2) {
return ((in1 != 0) && (in2 != 0)) ? 1 : 0;
}
+
+ @Override
+ public boolean isBinary(){
+ return true;
+ }
}
diff --git a/src/main/java/org/apache/sysds/runtime/functionobjects/Equals.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/Equals.java
index 93160b2780..f8000b49ac 100644
--- a/src/main/java/org/apache/sysds/runtime/functionobjects/Equals.java
+++ b/src/main/java/org/apache/sysds/runtime/functionobjects/Equals.java
@@ -74,4 +74,9 @@ public class Equals extends ValueComparisonFunction
public boolean compare(String in1, String in2) {
return ( in1!=null && in1.equals(in2) );
}
+
+ @Override
+ public boolean isBinary(){
+ return true;
+ }
}
diff --git
a/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThan.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThan.java
index aa656ff12e..15ed75344e 100644
--- a/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThan.java
+++ b/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThan.java
@@ -74,4 +74,9 @@ public class GreaterThan extends ValueComparisonFunction
public boolean compare(String in1, String in2) {
return (in1!=null && in1.compareTo(in2)>0 );
}
+
+ @Override
+ public boolean isBinary(){
+ return true;
+ }
}
diff --git
a/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThanEquals.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThanEquals.java
index fb52d71592..907c32e387 100644
---
a/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThanEquals.java
+++
b/src/main/java/org/apache/sysds/runtime/functionobjects/GreaterThanEquals.java
@@ -74,4 +74,10 @@ public class GreaterThanEquals extends
ValueComparisonFunction
public boolean compare(String in1, String in2) {
return (in1!=null && in1.compareTo(in2)>=0 );
}
+
+ @Override
+ public boolean isBinary(){
+ return true;
+ }
+
}
diff --git
a/src/main/java/org/apache/sysds/runtime/functionobjects/LessThan.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/LessThan.java
index dc5cc4d277..108fd5b6de 100644
--- a/src/main/java/org/apache/sysds/runtime/functionobjects/LessThan.java
+++ b/src/main/java/org/apache/sysds/runtime/functionobjects/LessThan.java
@@ -73,4 +73,9 @@ public class LessThan extends ValueComparisonFunction
public boolean compare(String in1, String in2) {
return (in1!=null && in1.compareTo(in2)<0 );
}
+
+ @Override
+ public boolean isBinary(){
+ return true;
+ }
}
diff --git
a/src/main/java/org/apache/sysds/runtime/functionobjects/LessThanEquals.java
b/src/main/java/org/apache/sysds/runtime/functionobjects/LessThanEquals.java
index 54d46de687..e49e0c4beb 100644
--- a/src/main/java/org/apache/sysds/runtime/functionobjects/LessThanEquals.java
+++ b/src/main/java/org/apache/sysds/runtime/functionobjects/LessThanEquals.java
@@ -
(systemds) branch main updated: [MINOR] Fix Integer overflow in Metadata for rows and cols
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new efc843fab2 [MINOR] Fix Integer overflow in Metadata for rows and cols
efc843fab2 is described below
commit efc843fab24ea305c4274f8b71a95eb1e61c0db3
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 12:47:23 2024 +0100
[MINOR] Fix Integer overflow in Metadata for rows and cols
---
src/main/java/org/apache/sysds/runtime/meta/MetaDataAll.java | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/meta/MetaDataAll.java
b/src/main/java/org/apache/sysds/runtime/meta/MetaDataAll.java
index 43d8ac3840..60730ed960 100644
--- a/src/main/java/org/apache/sysds/runtime/meta/MetaDataAll.java
+++ b/src/main/java/org/apache/sysds/runtime/meta/MetaDataAll.java
@@ -165,8 +165,8 @@ public class MetaDataAll extends DataIdentifier {
private void parseMetaDataParam(Object key, Object val)
{
switch(key.toString()) {
- case DataExpression.READROWPARAM: _dim1 = (Integer)
val; break;
- case DataExpression.READCOLPARAM: _dim2 = (Integer)
val; break;
+ case DataExpression.READROWPARAM: _dim1 = val
instanceof Long ? (Long) val : (Integer) val; break;
+ case DataExpression.READCOLPARAM: _dim2 = val
instanceof Long ? (Long) val : (Integer) val; break;
case DataExpression.ROWBLOCKCOUNTPARAM:
setBlocksize((Integer) val); break;
case DataExpression.READNNZPARAM: setNnz(val instanceof
Long ? (Long) val : (Integer) val); break;
case DataExpression.FORMAT_TYPE:
setFormatTypeString((String) val); break;
@@ -238,6 +238,8 @@ public class MetaDataAll extends DataIdentifier {
}
public void setDelim(String delim) {
+ if(delim.length() == 0)
+ throw new RuntimeException("Invalid metadata delim,
cannot be empty string");
_delim = delim;
}
(systemds) branch main updated: [MINOR] Lop Properties toString for Debugging
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new b1fb351f59 [MINOR] Lop Properties toString for Debugging
b1fb351f59 is described below
commit b1fb351f59ad8c132efc431f0190681f4fb2cd7b
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 12:33:22 2024 +0100
[MINOR] Lop Properties toString for Debugging
---
src/main/java/org/apache/sysds/lops/Data.java | 10 --
src/main/java/org/apache/sysds/lops/Lop.java | 6 ++--
.../java/org/apache/sysds/lops/LopProperties.java | 37 +++---
src/main/java/org/apache/sysds/lops/Unary.java | 8 +++--
4 files changed, 33 insertions(+), 28 deletions(-)
diff --git a/src/main/java/org/apache/sysds/lops/Data.java
b/src/main/java/org/apache/sysds/lops/Data.java
index 93552852f2..a0546904c0 100644
--- a/src/main/java/org/apache/sysds/lops/Data.java
+++ b/src/main/java/org/apache/sysds/lops/Data.java
@@ -127,16 +127,6 @@ public class Data extends Lop
lps.setProperties ( inputs, ExecType.INVALID);
}
- /**
-* Data-Lop-specific method to set the execution type for persistent
write.
-* TODO: split lops into MR/CP lop.
-*
-* @param et execution type
-*/
- public void setExecType( ExecType et ) {
- lps.execType = et;
- }
-
/**
* method to get format type for input, output files.
* @return file format
diff --git a/src/main/java/org/apache/sysds/lops/Lop.java
b/src/main/java/org/apache/sysds/lops/Lop.java
index 5f32650e05..b7ae1ffe78 100644
--- a/src/main/java/org/apache/sysds/lops/Lop.java
+++ b/src/main/java/org/apache/sysds/lops/Lop.java
@@ -501,13 +501,13 @@ public abstract class Lop
/**
* Set the execution type of LOP.
+*
* @param newExecType new execution type
*/
- public void setExecType(ExecType newExecType){
- lps.setExecType(newExecType);
+ public void setExecType(ExecType newExecType) {
+ lps.setExecType(newExecType);
}
-
public boolean isExecSpark () {
return (lps.getExecType() == ExecType.SPARK);
}
diff --git a/src/main/java/org/apache/sysds/lops/LopProperties.java
b/src/main/java/org/apache/sysds/lops/LopProperties.java
index efa3cd2fe2..e2b55d160c 100644
--- a/src/main/java/org/apache/sysds/lops/LopProperties.java
+++ b/src/main/java/org/apache/sysds/lops/LopProperties.java
@@ -24,14 +24,9 @@ import java.util.ArrayList;
import org.apache.sysds.common.Types.ExecType;
import org.apache.sysds.runtime.controlprogram.parfor.util.IDSequence;
-public class LopProperties
-{
- // static variable to assign an unique ID to every lop that is created
- private static IDSequence UniqueLopID = null;
-
- static {
- UniqueLopID = new IDSequence();
- }
+public class LopProperties {
+ /** static variable to assign an unique ID to every lop that is created
*/
+ private static IDSequence UniqueLopID = new IDSequence();
/**
* Execution properties for each lop.
@@ -42,10 +37,13 @@ public class LopProperties
* isAligner = is this lop mainly used to reorder/sort/align the keys
*
*/
- long ID;
- int level;
- ExecType execType;
- boolean producesIntermediateOutput;
+ protected long ID;
+ /** The level in the dag. Specifying when this instruction can be
executed. */
+ protected int level;
+ /** The execution type of this lop node, CP, Spark, GPU, Federated,
etc*/
+ protected ExecType execType;
+ /** If this Lop produce some intermediate that have to be considered in
the memory estimations */
+ protected boolean producesIntermediateOutput;
public LopProperties() {
ID = UniqueLopID.getNextID();
@@ -99,4 +97,19 @@ public class LopProperties
execType = et;
setLevel(inputs);
}
+
+ @Override
+ public String toString(){
+ StringBuilder sb = new StringBuilder();
+ sb.append(this.getClass().getSimpleName());
+ sb.append(" ID: ");
+ sb.append(ID);
+ sb.append(" Level: ");
+ sb.append(level);
+ sb.append(" ExecType: ");
+ sb.append(execType);
+ sb.append(" Intermediate: ");
+ sb.append(producesIntermediateOutput);
+ return sb.toString();
+ }
}
diff --git a/src/main/java/org/apache/sysds/lops/Unary.java
b/src/main/java/org/apache/sysds/lops/Unary.java
index 5e83c1de4d..e7932695a8 100644
--- a/src/main/java/org/apache/sysds/lops/Unary.java
+++ b/src/ma
(systemds) branch main updated: [MINOR] Fix logging in spoof compiler
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 1519123fd5 [MINOR] Fix logging in spoof compiler
1519123fd5 is described below
commit 1519123fd5152a477ca28bc3d1061f4282068992
Author: Sebastian Baunsgaard
AuthorDate: Fri Jan 5 12:31:50 2024 +0100
[MINOR] Fix logging in spoof compiler
---
src/main/java/org/apache/sysds/hops/codegen/SpoofCompiler.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main/java/org/apache/sysds/hops/codegen/SpoofCompiler.java
b/src/main/java/org/apache/sysds/hops/codegen/SpoofCompiler.java
index 55d75b092a..aca07fb413 100644
--- a/src/main/java/org/apache/sysds/hops/codegen/SpoofCompiler.java
+++ b/src/main/java/org/apache/sysds/hops/codegen/SpoofCompiler.java
@@ -539,13 +539,13 @@ public class SpoofCompiler {
}
//explain debug output cplans or
generated source code
- if( LOG.isTraceEnabled() ||
DMLScript.EXPLAIN.isHopsType(recompile) ) {
+ if( LOG.isInfoEnabled() ||
DMLScript.EXPLAIN.isHopsType(recompile) ) {
LOG.info("Codegen EXPLAIN
(generated cplan for HopID: " + cplan.getKey() +
", line
"+tmp.getValue().getBeginLine() + ", hash="+tmp.getValue().hashCode()+"):");
LOG.info(tmp.getValue().getClassname()
+
Explain.explainCPlan(cplan.getValue().getValue()));
}
- if( LOG.isTraceEnabled() ||
DMLScript.EXPLAIN.isRuntimeType(recompile) ) {
+ if( LOG.isInfoEnabled() ||
DMLScript.EXPLAIN.isRuntimeType(recompile) ) {
LOG.info("JAVA Codegen EXPLAIN
(generated code for HopID: " + cplan.getKey() +
", line
"+tmp.getValue().getBeginLine() + ", hash="+tmp.getValue().hashCode()+"):");
LOG.info(CodegenUtils.printWithLineNumber(src));
(systemds) branch main updated: [MINOR] gitIgnore test files & refine javatest
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git The following commit(s) were added to refs/heads/main by this push: new 543303d843 [MINOR] gitIgnore test files & refine javatest 543303d843 is described below commit 543303d843075024b9b242941a671a5e074f654f Author: Sebastian Baunsgaard AuthorDate: Fri Jan 5 12:30:51 2024 +0100 [MINOR] gitIgnore test files & refine javatest --- .github/workflows/javaTests.yml | 2 +- .gitignore | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/javaTests.yml b/.github/workflows/javaTests.yml index 22cda7b67c..9768d2fb5a 100644 --- a/.github/workflows/javaTests.yml +++ b/.github/workflows/javaTests.yml @@ -55,7 +55,7 @@ jobs: "**.component.c**.**", "**.component.e**.**,**.component.f**.**,**.component.m**.**", "**.component.p**.**,**.component.t**.**", - "**.functions.a**.**,**.functions.binary.frame.**,**.functions.binary.matrix.**,**.functions.binary.scalar.**,**.functions.binary.tensor.**", + "**.functions.a**.**,**.functions.binary.matrix.**,**.functions.binary.scalar.**,**.functions.binary.tensor.**", "**.functions.blocks.**,**.functions.data.rand.**,", "**.functions.countDistinct.**,**.functions.countDistinctApprox.**,**.functions.data.misc.**,**.functions.lineage.**", "**.functions.compress.**,**.functions.data.tensor.**,**.functions.codegenalg.parttwo.**,**.functions.codegen.**,**.functions.caching.**", diff --git a/.gitignore b/.gitignore index 6695fcb64d..1a83a3a80e 100644 --- a/.gitignore +++ b/.gitignore @@ -78,8 +78,10 @@ docs/_site # Test Artifacts src/test/scripts/**/*.dmlt src/test/scripts/functions/mlcontextin/ +src/test/scripts/functions/frame/io/ src/test/java/org/apache/sysds/test/component/compress/io/files src/test/java/org/apache/sysds/test/component/compress/io/filesIOSpark/* +src/test/java/org/apache/sysds/test/component/compress/io/filesIOTest .factorypath # Excluded sources
(systemds) branch main updated: [SYSTEMDS-2985] Fix nested list cache management
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 61a385fc9d [SYSTEMDS-2985] Fix nested list cache management
61a385fc9d is described below
commit 61a385fc9d82f74642bc0fe2392b05cf556537ee
Author: MaximilianTUB
AuthorDate: Wed Dec 6 17:09:21 2023 +0100
[SYSTEMDS-2985] Fix nested list cache management
SystemDS was previously not supporting nested lists correctly
since the data of CacheableData objects within nested loops
were always deleted after a function call.
Normally, there are rmvar statements after function calls to
emove all variables used within the function. To protect
CacheableData objects (e.g. matrices) from having their data
removed by the rmvar statements we use a cleanup-enabled flag.
This flag was not correctly set for variables that were within
a nested list. These commits fix this problem by flagging all
elements, also within nested lists.
Automated tests have been added to test the changes.
Closes #1956
---
.../runtime/controlprogram/ParForProgramBlock.java | 7 +-
.../controlprogram/context/ExecutionContext.java | 69 --
.../instructions/cp/FunctionCallCPInstruction.java | 3 +-
.../sysds/runtime/instructions/cp/ListObject.java | 58 +++-
.../test/functions/caching/PinVariablesTest.java | 153 +
5 files changed, 242 insertions(+), 48 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/controlprogram/ParForProgramBlock.java
b/src/main/java/org/apache/sysds/runtime/controlprogram/ParForProgramBlock.java
index 790a92de58..06a548a753 100644
---
a/src/main/java/org/apache/sysds/runtime/controlprogram/ParForProgramBlock.java
+++
b/src/main/java/org/apache/sysds/runtime/controlprogram/ParForProgramBlock.java
@@ -31,6 +31,7 @@ import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
+import java.util.Queue;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -652,7 +653,7 @@ public class ParForProgramBlock extends ForProgramBlock {
//preserve shared input/result variables of cleanup
ArrayList varList = ec.getVarList();
- boolean[] varState = ec.pinVariables(varList);
+ Queue varState = ec.pinVariables(varList);
try
{
@@ -677,7 +678,7 @@ public class ParForProgramBlock extends ForProgramBlock {
catch(Exception ex) {
throw new DMLRuntimeException("PARFOR: Failed to
execute loop in parallel.",ex);
}
-
+
//reset state of shared input/result variables
ec.unpinVariables(varList, varState);
@@ -1198,7 +1199,7 @@ public class ParForProgramBlock extends ForProgramBlock {
}
}
- private void cleanupSharedVariables( ExecutionContext ec, boolean[]
varState ) {
+ private void cleanupSharedVariables( ExecutionContext ec,
Queue varState ) {
//TODO needs as precondition a systematic treatment of
persistent read information.
}
diff --git
a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java
b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java
index d98827a24e..0903b5abca 100644
---
a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java
+++
b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java
@@ -65,12 +65,14 @@ import org.apache.sysds.runtime.util.HDFSTool;
import org.apache.sysds.utils.Statistics;
import java.util.ArrayList;
+import java.util.LinkedList;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Future;
import java.util.stream.Collectors;
+import java.util.Queue;
public class ExecutionContext {
protected static final Log LOG =
LogFactory.getLog(ExecutionContext.class.getName());
@@ -753,45 +755,28 @@ public class ExecutionContext {
* @param varList variable list
* @return indicator vector of old cleanup state of matrix objects
*/
- public boolean[] pinVariables(List varList)
+ public Queue pinVariables(List varList)
{
- //analyze list variables
- int nlist = 0;
- int nlistItems = 0;
- for( int i=0; i )
- varsState[pos++] =
((CacheableData)dat).isCleanupEnabled();
- else if( dat instanceof ListObject )
- for( Data dat2 : ((List
(systemds) branch main updated: [MINOR] Uncompressed ColGroup Outer TSMM
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 246eea9784 [MINOR] Uncompressed ColGroup Outer TSMM
246eea9784 is described below
commit 246eea9784aa3b34c9eefdaee4666708b5a7db95
Author: Sebastian Baunsgaard
AuthorDate: Sat Dec 30 14:10:02 2023 +0100
[MINOR] Uncompressed ColGroup Outer TSMM
Add support for sparse outer TSMM for uncompressed column groups.
This was missing in 1c26e2d299ace9f0b3b4974c9d8bac665fd9692e
Closes #1968
---
.../compress/colgroup/ColGroupUncompressed.java| 35 +-
.../component/compress/colgroup/ColGroupTest.java | 21 -
2 files changed, 41 insertions(+), 15 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupUncompressed.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupUncompressed.java
index c4713d6e59..d5553deb41 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupUncompressed.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupUncompressed.java
@@ -532,14 +532,33 @@ public class ColGroupUncompressed extends AColGroup {
// tsmm but only upper triangle.
LibMatrixMult.matrixMultTransposeSelf(_data, tmp, true, false);
- // copy that upper triangle part to ret
- final int numColumns = ret.getNumColumns();
- final double[] result = ret.getDenseBlockValues();
- final double[] tmpV = tmp.getDenseBlockValues();
- for(int row = 0, offTmp = 0; row < tCol; row++, offTmp += tCol)
{
- final int offRet = _colIndexes.get(row) * numColumns;
- for(int col = row; col < tCol; col++)
- result[offRet + _colIndexes.get(col)] +=
tmpV[offTmp + col];
+ if(tmp.isInSparseFormat()){
+ final int numColumns = ret.getNumColumns();
+ final double[] result = ret.getDenseBlockValues();
+ final SparseBlock sb = tmp.getSparseBlock();
+ for(int row = 0; row < tCol; row++) {
+ final int offRet = _colIndexes.get(row) *
numColumns;
+ if(sb.isEmpty(row))
+ continue;
+ int apos = sb.pos(row);
+ int alen = sb.size(row) + apos;
+ int[] aix = sb.indexes(row);
+ double[] aval = sb.values(row);
+ for(int j = apos; j < alen; j++)
+ result[offRet +
_colIndexes.get(aix[j])] += aval[j];
+
+ }
+ }
+ else{
+ // copy that upper triangle part to ret
+ final int numColumns = ret.getNumColumns();
+ final double[] result = ret.getDenseBlockValues();
+ final double[] tmpV = tmp.getDenseBlockValues();
+ for(int row = 0, offTmp = 0; row < tCol; row++, offTmp
+= tCol) {
+ final int offRet = _colIndexes.get(row) *
numColumns;
+ for(int col = row; col < tCol; col++)
+ result[offRet + _colIndexes.get(col)]
+= tmpV[offTmp + col];
+ }
}
}
diff --git
a/src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupTest.java
b/src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupTest.java
index 14f4a56c18..54a543ad13 100644
---
a/src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupTest.java
+++
b/src/test/java/org/apache/sysds/test/component/compress/colgroup/ColGroupTest.java
@@ -1118,13 +1118,20 @@ public class ColGroupTest extends ColGroupBase {
@Test
public void tsmm() {
- final MatrixBlock bt = new MatrixBlock(maxCol, maxCol, false);
- final MatrixBlock ot = new MatrixBlock(maxCol, maxCol, false);
- ot.allocateDenseBlock();
- bt.allocateDenseBlock();
- base.tsmm(bt, nRow);
- other.tsmm(ot, nRow);
- compare(ot, bt);
+ try{
+
+ final MatrixBlock bt = new MatrixBlock(maxCol, maxCol,
false);
+ final MatrixBlock ot = new MatrixBlock(maxCol, maxCol,
false);
+ ot.allocateDenseBlock();
+ bt.allocateDenseBlock();
+ base.tsmm(bt, nRow);
+ o
(systemds) branch main updated: [SYSTEMDS-3545] Linearized Img Sample Shear & Rotate
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 3b48c4ae5b [SYSTEMDS-3545] Linearized Img Sample Shear & Rotate
3b48c4ae5b is described below
commit 3b48c4ae5b82ec8e44e1d9425a9a8662f755c2ba
Author: baristerzioglu
AuthorDate: Fri Sep 8 12:53:15 2023 +0200
[SYSTEMDS-3545] Linearized Img Sample Shear & Rotate
This commit merge the remaining linearized image operations
from #1914 #1913 and #1912
It contains the combination of image sample shear and rotate.
The commit is combined since the three PRs does not clearly
separate the changed files.
LDE Project SoSe 2023
Co-authored-by: baristerzioglu
Co-authored-by: slnkahveci <76944633+slnkahv...@users.noreply.github.com>
Closes #1914 #1913 #1912 #1965
---
scripts/builtin/img_rotate_linearized.dml | 62 ++
scripts/builtin/img_sample_pairing_linearized.dml | 48 +
scripts/builtin/img_shear_linearized.dml | 40
scripts/builtin/img_transform_linearized.dml | 3 -
.../java/org/apache/sysds/common/Builtins.java | 3 +
.../BuiltinImageSamplePairingLinearizedTest.java | 106 ++
.../pipelines/BuiltinImageRotateLinTest.java | 116 +++
.../pipelines/BuiltinImageShearLinTest.java| 122
.../pipelines/BuiltinImageTransformLinTest.java| 218 ++---
.../expected/ImageTransformLinRotated.csv | 1 +
.../expected/ImageTransformLinTransformed.csv | 1 -
.../functions/builtin/image_rotate_linearized.dml | 33
.../builtin/image_sample_pairing_linearized.dml| 37
.../functions/builtin/image_shear_linearized.dml | 34
.../functions/builtin/image_transform_linearized.R | 1 +
15 files changed, 711 insertions(+), 114 deletions(-)
diff --git a/scripts/builtin/img_rotate_linearized.dml
b/scripts/builtin/img_rotate_linearized.dml
new file mode 100644
index 00..f5ac43625d
--- /dev/null
+++ b/scripts/builtin/img_rotate_linearized.dml
@@ -0,0 +1,62 @@
+#-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+#-
+
+# The Linearized Image Rotate function rotates the linearized input images
counter-clockwise around the center.
+# Uses nearest neighbor sampling.
+#
+# INPUT:
+#
---
+# img_in Linearized input images as 2D matrix with top left corner at [1,
1]
+# radians The value by which to rotate in radian.
+# fill_value The background color revealed by the rotation
+#
---
+#
+# OUTPUT:
+#
-
+# img_out Output images in linearized form as 2D matrix with top left corner
at [1, 1]
+#
-
+
+m_img_rotate_linearized = function(Matrix[Double] img_in, Double radians,
Double fill_value, Integer s_cols, Integer s_rows) return (Matrix[Double]
img_out) {
+ # Translation matrix for moving the origin to the center of the image
+ t1 = matrix("1 0 0 0 1 0 0 0 1", rows=3, cols=3)
+ t1[1, 3] = -s_cols / 2
+ t1[2, 3] = -s_rows / 2
+
+ # Translation matrix for moving the origin back to the top left corner
+ t2 = matrix("1 0 0 0 1 0 0 0 1", rows=3, cols=3)
+ t2[1, 3] = s_cols / 2
+ t2[2, 3] = s_rows / 2
+
+ # The rotation matrix around the origin
+ rot = matrix("1 0 0 0 1 0 0 0 1", rows=3, cols=3)
+ c = cos(radians)
+ s = sin(radians)
+ rot[1, 1] = c
+ rot[1, 2] = s
+ rot[2, 1] = -s
+ rot[2, 2] = c
+
+ # Combined transformation matrix
+ m = t2 %*% rot %*% t1
+
+ # Transform image
+ img_out = img_transform_linearized(img_in, s_cols, s_rows,
as.scala
(systemds) branch main updated: [SYSTEMDS-3636] Improved ultra-sparse TSMM left w/ sparse output
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 1c26e2d299 [SYSTEMDS-3636] Improved ultra-sparse TSMM left w/ sparse
output
1c26e2d299 is described below
commit 1c26e2d299ace9f0b3b4974c9d8bac665fd9692e
Author: Christina Dionysio
AuthorDate: Thu Dec 7 09:29:06 2023 +0100
[SYSTEMDS-3636] Improved ultra-sparse TSMM left w/ sparse output
This patch provides the support for left transposed ultra-sparse tsmm.
Similar to the the implementation of the right transpose ultra-sparse tsmm,
binary search is used to populate the upper triangular part of a sparse
output matrix.
Operation:
t(X) %*% X
tests show an improvement of 17 to 30x, and support some new cases that
were not able to run before.
Closes #1955
---
.../sysds/runtime/matrix/data/LibMatrixMult.java | 117 +
.../FullMatrixMultiplicationTransposeSelfTest.java | 27 -
2 files changed, 94 insertions(+), 50 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
index 0f96a30dad..80d0230da9 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
@@ -442,9 +442,9 @@ public class LibMatrixMult
//Timing time = new Timing(true);
//pre-processing
- ret.sparse = isSparseOutputTSMM(m1, leftTranspose);
+ ret.sparse = isSparseOutputTSMM(m1);
ret.allocateBlock();
- MatrixBlock m1t = isSparseOutputTSMM(m1, leftTranspose, true) ?
+ MatrixBlock m1t = isSparseOutputTSMM(m1, true) ?
LibMatrixReorg.transpose(m1) : null;
//core tsmm operation
@@ -484,9 +484,9 @@ public class LibMatrixMult
//Timing time = new Timing(true);
//pre-processing (no need to check isThreadSafe)
- ret.sparse = isSparseOutputTSMM(m1, leftTranspose);
+ ret.sparse = isSparseOutputTSMM(m1);
ret.allocateBlock();
- MatrixBlock m1t = isSparseOutputTSMM(m1, leftTranspose, true) ?
+ MatrixBlock m1t = isSparseOutputTSMM(m1, true) ?
LibMatrixReorg.transpose(m1, k) : null;
//core multi-threaded matrix mult computation
@@ -2506,39 +2506,60 @@ public class LibMatrixMult
}
private static void matrixMultTransposeSelfUltraSparse( MatrixBlock m1,
MatrixBlock ret, boolean leftTranspose, int rl, int ru ) {
- if( leftTranspose )
- throw new DMLRuntimeException("Left tsmm with sparse
output not supported");
-
- // Operation X%*%t(X), sparse input and output
- SparseBlock a = m1.sparseBlock;
- SparseBlock c = ret.sparseBlock;
+SparseBlock a = m1.sparseBlock;
+SparseBlock c = ret.sparseBlock;
int m = m1.rlen;
-
- final int blocksize = 256;
- for(int bi=rl; bi=0) {
+ int len = apos + alen;
+ for(int i = rlix; i < len && aix[i] <
ru; i++) {
+ for (int k = a.posFIndexGTE(r,
aix[i]); k < len; k++) {
+ sr[aix[i]].add(c.pos(k)
+ aix[k], avals[i] * avals[k]);
+ }
+ }
+ }
+ }
+ }
+ else {
+ // Operation X%*%t(X), sparse input and output
+ final int blocksize = 256;
+ for(int bi=rl; bi 1) {
//X%*%t(X) SPARSE MATRIX
//directly via LibMatrixReorg in order to prevent
sparsity change
@@ -4489,16 +4516,16 @@ public class LibMatrixMult
return m2.clen < 4*1024 && sparseOut;
}
- public static boolean isSparseOutputTSMM(MatrixBlock m1, boolean
leftTranspose) {
- return isSparseOutputTSMM(m1, leftTranspose, false);
+ public static boolean isSparseOutputTSMM(MatrixBlock m1) {
+ return isSparseOutputTSMM(m1, false);
}
- public static boolean isSparseOutputTSMM(MatrixBlock m1, boolean
leftTranspose, boolean ultraSparse) {
+ public static boolean isSparseOutputTSMM(MatrixBlock m1, boolean
ultraSparse) {
double sp = m1.get
(systemds) branch main updated: [MINOR] Reduce Epochs PararmservTest
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new e6b54129f3 [MINOR] Reduce Epochs PararmservTest
e6b54129f3 is described below
commit e6b54129f35dac76d3cd69aa76e1664bdb927546
Author: Sebastian Baunsgaard
AuthorDate: Sat Dec 30 13:30:00 2023 +0100
[MINOR] Reduce Epochs PararmservTest
---
.../paramserv/ParamservLocalNNAveragingTest.java | 23 +++---
1 file changed, 12 insertions(+), 11 deletions(-)
diff --git
a/src/test/java/org/apache/sysds/test/functions/paramserv/ParamservLocalNNAveragingTest.java
b/src/test/java/org/apache/sysds/test/functions/paramserv/ParamservLocalNNAveragingTest.java
index b103adf7ef..ab1cf97ab9 100644
---
a/src/test/java/org/apache/sysds/test/functions/paramserv/ParamservLocalNNAveragingTest.java
+++
b/src/test/java/org/apache/sysds/test/functions/paramserv/ParamservLocalNNAveragingTest.java
@@ -39,55 +39,56 @@ public class ParamservLocalNNAveragingTest extends
AutomatedTestBase {
@Test
public void testParamservBSPBatchDisjointContiguous() {
- runDMLTest(10, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
+ runDMLTest(4, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
}
@Test
public void testParamservBSPEpoch() {
- runDMLTest(10, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.EPOCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
+ runDMLTest(4, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.EPOCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
}
@Test
public void testParamservBSPBatchDisjointRoundRobin() {
- runDMLTest(10, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_ROUND_ROBIN, true);
+ runDMLTest(4, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_ROUND_ROBIN, true);
}
@Test
public void testParamservBSPBatchDisjointRandom() {
- runDMLTest(10, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_RANDOM, true);
+ runDMLTest(4, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_RANDOM, true);
}
@Test
public void testParamservBSPBatchOverlapReshuffle() {
- runDMLTest(10, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.OVERLAP_RESHUFFLE, true);
+ runDMLTest(4, 2, Statement.PSUpdateType.BSP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.OVERLAP_RESHUFFLE, true);
}
@Test
public void testParamservSBPBatchDisjointContiguous() {
- runDMLTest(10, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
+ runDMLTest(4, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
}
@Test
public void testParamservSBPEpoch() {
- runDMLTest(10, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.EPOCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
+ runDMLTest(4, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.EPOCH, 32, Statement.PSScheme.DISJOINT_CONTIGUOUS, true);
}
@Test
public void testParamservSBPBatchDisjointRoundRobin() {
- runDMLTest(10, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_ROUND_ROBIN, true);
+ runDMLTest(4, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_ROUND_ROBIN, true);
}
@Test
public void testParamservSBPBatchDisjointRandom() {
- runDMLTest(10, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_RANDOM, true);
+ runDMLTest(4, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.DISJOINT_RANDOM, true);
}
@Test
public void testParamservSBPBatchOverlapReshuffle() {
- runDMLTest(10, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.OVERLAP_RESHUFFLE, true);
+ runDMLTest(4, 3, Statement.PSUpdateType.SBP,
Statement.PSFrequency.BATCH, 32, Statement.PSScheme.OVERLAP_RESHUFFLE, true);
}
- private void runDMLTest(int epochs, int workers, Statement.PSUpdateType
utype, Statement.PSFrequency freq, int batchsize, Statement.PSScheme scheme
(systemds) branch main updated: [MINOR] Write compressed test fix
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 96c620674f [MINOR] Write compressed test fix
96c620674f is described below
commit 96c620674f285a13d8f91f82750841b6fb15e74d
Author: Sebastian Baunsgaard
AuthorDate: Sat Dec 30 13:02:40 2023 +0100
[MINOR] Write compressed test fix
This commit solidify the already working compression test
to be more resilient in the GitHub Actions.
Closes #1966
---
.../sysds/test/component/compress/io/IOTest.java | 28 --
1 file changed, 5 insertions(+), 23 deletions(-)
diff --git
a/src/test/java/org/apache/sysds/test/component/compress/io/IOTest.java
b/src/test/java/org/apache/sysds/test/component/compress/io/IOTest.java
index 3708b52e7d..3c18cf049b 100644
--- a/src/test/java/org/apache/sysds/test/component/compress/io/IOTest.java
+++ b/src/test/java/org/apache/sysds/test/component/compress/io/IOTest.java
@@ -134,26 +134,7 @@ public class IOTest {
}
protected static void writeAndReadR(MatrixBlock mb, int rep) throws
Exception {
- try {
-
- String filename = getName();
- WriterCompressed.writeCompressedMatrixToHDFS(mb,
filename);
- File f = new File(filename);
- assertTrue(f.isFile() || f.isDirectory());
- MatrixBlock mbr = IOCompressionTestUtils.read(filename,
mb.getNumRows(), mb.getNumColumns(),
- OptimizerUtils.DEFAULT_BLOCKSIZE);
- IOCompressionTestUtils.verifyEquivalence(mb, mbr);
- }
- catch(Exception e) {
- if(rep < 3) {
- Thread.sleep(1000);
- writeAndReadR(mb, rep + 1);
- return;
- }
- e.printStackTrace();
- fail("Failed to write file");
- }
-
+ writeAndReadR(mb, OptimizerUtils.DEFAULT_BLOCKSIZE, rep);
}
protected static void write(MatrixBlock src, String path) throws
Exception {
@@ -177,11 +158,12 @@ public class IOTest {
protected static void writeAndReadR(MatrixBlock mb, int blen, int rep)
throws Exception {
try {
-
String filename = getName();
- WriterCompressed.writeCompressedMatrixToHDFS(mb,
filename, blen);
File f = new File(filename);
- assertTrue(f.isFile() || f.isDirectory());
+ f.delete();
+ WriterCompressed.writeCompressedMatrixToHDFS(mb,
filename, blen);
+ File f2 = new File(filename);
+ assertTrue(f2.isFile() || f2.isDirectory());
MatrixBlock mbr = IOCompressionTestUtils.read(filename,
mb.getNumRows(), mb.getNumColumns(), blen);
IOCompressionTestUtils.verifyEquivalence(mb, mbr);
}
(systemds) branch main updated: [MINOR] Performance improvement of dist
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new d01c61d7d5 [MINOR] Performance improvement of dist
d01c61d7d5 is described below
commit d01c61d7d56f250223f60fff6e773ba0870a7bee
Author: ramesesz
AuthorDate: Mon Dec 11 16:43:13 2023 +0100
[MINOR] Performance improvement of dist
This patch improves the builtin dist function
by removing the outer product operator. For 100
function calls on an arbitrary matrix with 4000
rows and 800 cols, the new dist function shortens
the runtime from 66.541s to 60.268s.
Closes #1959
---
scripts/builtin/dist.dml | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/scripts/builtin/dist.dml b/scripts/builtin/dist.dml
index 26ded9a197..f296fd717b 100644
--- a/scripts/builtin/dist.dml
+++ b/scripts/builtin/dist.dml
@@ -32,7 +32,8 @@
#
---
m_dist = function(Matrix[Double] X) return (Matrix[Double] Y) {
- G = X %*% t(X);
- Y = sqrt(-2 * G + outer(diag(G), t(diag(G)), "+"));
+ n = nrow(X)
+ s = rowSums(X^2)
+ Y = sqrt(-2 * X %*% t(X) + s + t(s))
Y = replace(target = Y, pattern=NaN, replacement = 0);
}
(systemds) branch main updated (c842072446 -> a2aea092a8)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
from c842072446 [MINOR] Bug fixes
add f1f37e724c [MINOR] Update syntax and deprecated in docker
add b41eccbdd1 [MINOR] C++ Build parallel
add 9781e1069b [MINOR] 100% test coverage of Dense-Sparse conversion of
Matrices
add 70fec49b27 [MINOR] LOG4j test ignore native support of HDFS
add 41db04537d [MINOR] Add Federated Timeouts
new a2aea092a8 [SYSTEMDS-3659] Federated GitHub Actions Fail
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/javaTests.yml| 12 +-
.gitignore | 1 +
docker/entrypoint.sh | 2 +-
docker/testsysds.Dockerfile| 3 +-
src/main/cpp/build.sh | 61 +++--
docker/entrypoint.sh => src/main/cpp/build_BLAS.sh | 31 ++-
docker/entrypoint.sh => src/main/cpp/build_HE.sh | 26 +--
docker/entrypoint.sh => src/main/cpp/build_mkl.sh | 32 +--
.../hops/fedplanner/PrivacyConstraintLoader.java | 14 +-
.../context/SparkExecutionContext.java | 7 +-
.../federated/FederatedStatistics.java | 8 +-
.../controlprogram/federated/FederationUtils.java | 12 +-
.../monitoring/services/StatisticsService.java | 14 +-
.../matrix/data/LibMatrixDenseToSparse.java| 20 +-
.../sysds/runtime/matrix/data/LibMatrixSketch.java | 62 +
.../matrix/data/LibMatrixSparseToDense.java| 10 +-
.../sysds/runtime/matrix/data/MatrixBlock.java | 21 +-
.../org/apache/sysds/test/AutomatedTestBase.java | 74 +-
src/test/java/org/apache/sysds/test/TestUtils.java | 6 +-
.../test/component/matrix/DenseAndSparseTest.java | 211 +
.../test/component/matrix/MatrixMultiplyTest.java | 122 --
.../primitives/FederatedCovarianceTest.java| 180 ---
.../primitives/FederatedQuantileTest.java | 215 -
.../primitives/FederatedQuantileWeightsTest.java | 203
.../{ => part1}/FederatedBinaryMatrixTest.java | 73 +++---
.../{ => part1}/FederatedBinaryVectorTest.java | 71 +++---
.../{ => part1}/FederatedBroadcastTest.java| 46 ++--
.../{ => part1}/FederatedCastToFrameTest.java | 59 +++--
.../{ => part1}/FederatedCastToMatrixTest.java | 81 +++
.../{ => part1}/FederatedCentralMomentTest.java| 109 -
.../{ => part1}/FederatedColAggregateTest.java | 149 ++--
.../{ => part1}/FederatedConstructionTest.java | 72 +++---
.../{ => part1}/FederatedLeftIndexTest.java| 130 ++-
.../{ => part1}/FederatedMisAlignedTest.java | 134 +--
.../{ => part2}/FederatedMultiplyTest.java | 72 +++---
.../{ => part2}/FederatedNegativeTest.java | 2 +-
.../primitives/{ => part2}/FederatedProdTest.java | 105 +
.../primitives/part2/FederatedQuantileTest.java| 249
.../part2/FederatedQuantileWeightsTest.java| 226 ++
.../{ => part2}/FederatedRCBindTest.java | 113 -
.../primitives/{ => part2}/FederatedRdiagTest.java | 117 +-
.../{ => part2}/FederatedRemoveEmptyTest.java | 87 +++
.../{ => part2}/FederatedReplaceTest.java | 101
.../{ => part2}/FederatedReshapeTest.java | 107 +
.../primitives/{ => part2}/FederatedRevTest.java | 105 -
.../{ => part2}/FederatedRightIndexTest.java | 103 +
.../{ => part2}/FederatedRowIndexTest.java | 101
.../primitives/{ => part3}/FederatedSplitTest.java | 77 ---
.../{ => part3}/FederatedStatisticsTest.java | 86 +++
.../primitives/{ => part3}/FederatedSumTest.java | 88 +++
.../{ => part3}/FederatedTokenizeTest.java | 101
.../FederatedTransferLocalDataTest.java| 76 +++---
.../primitives/{ => part3}/FederatedTriTest.java | 98
.../FederatedWeightedCrossEntropyTest.java | 104 +
.../FederatedWeightedDivMatrixMultTest.java| 97
.../{ => part3}/FederatedWeightedSigmoidTest.java | 84 +++
.../FederatedWeightedSquaredLossTest.java | 69 +++---
.../FederatedWeightedUnaryMatrixMultTest.java | 69 +++---
.../{ => part4}/FederatedLogicalTest.java | 254 ++---
.../{ => part4}/FederatedRowAggregateTest.java | 135 +--
.../primitives/part5/FederatedCovarianceTe
(systemds) branch main updated: [MINOR] Ignore flag on fail
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new be1fd75091 [MINOR] Ignore flag on fail
be1fd75091 is described below
commit be1fd750913acbaf506d500243aba2d0cace4651
Author: Sebastian Baunsgaard
AuthorDate: Sat Dec 2 19:49:01 2023 +0100
[MINOR] Ignore flag on fail
The federated central moment test fails with timeout online,
but it does work locally. I am unable to reproduce the bug online.
I have verified that the bug is not related to the threading
Therefore to move forward i added a jira task to fix it, and
ignored the test in main branch.
---
.../test/functions/federated/primitives/FederatedCentralMomentTest.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
b/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
index 03bd7b3014..c93de914b7 100644
---
a/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
+++
b/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
@@ -66,6 +66,7 @@ public class FederatedCentralMomentTest extends
AutomatedTestBase {
}
@Test
+ @Ignore // infinite runtime online but works locally.
public void federatedCentralMomentCP() {
federatedCentralMoment(Types.ExecMode.SINGLE_NODE); }
@Test
(systemds) branch main updated: [MINOR] Fix ultra sparse empty
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 7b53ca24b2 [MINOR] Fix ultra sparse empty
7b53ca24b2 is described below
commit 7b53ca24b2bbc07a4c7f134a5bd072d03fd1e4d5
Author: Sebastian Baunsgaard
AuthorDate: Thu Nov 30 22:15:00 2023 +0100
[MINOR] Fix ultra sparse empty
---
src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
index e956f61906..0f96a30dad 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
@@ -1903,8 +1903,10 @@ public class LibMatrixMult
private static void matrixMultUltraSparseSparseSparseLeftRowGeneric(int
i, int apos, int alen, int[] aixs,
double[] avals, SparseBlock b, SparseBlockMCSR c, int m, int n)
{
for(int k = apos; k < apos + alen; k++) {
- final double aval = avals[k];
final int aix = aixs[k];
+ if(b.isEmpty(aix))
+ continue;
+ final double aval = avals[k];
final int bpos = b.pos(aix);
final int blen = b.size(aix) + bpos;
final int[] bix = b.indexes(aix);
(systemds) branch main updated: [SYSTEMDS-3653] Ultra Sparse Right MM Optimization
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 88fe2b0eb4 [SYSTEMDS-3653] Ultra Sparse Right MM Optimization
88fe2b0eb4 is described below
commit 88fe2b0eb4eb1fd342f37c2741629056155c56a2
Author: Sebastian Baunsgaard
AuthorDate: Thu Nov 30 17:49:43 2023 +0100
[SYSTEMDS-3653] Ultra Sparse Right MM Optimization
Right side Ultra sparse optimizations goring from 8.525 to 4.575
on 100 repetitions of 100k by 1000 dense %*% 1000 by 1000 with 30 non zeros.
Closes #1952
---
.../sysds/runtime/matrix/data/LibMatrixMult.java | 47 +++---
1 file changed, 42 insertions(+), 5 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
index 41dc7f2264..e956f61906 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixMult.java
@@ -49,6 +49,7 @@ import org.apache.sysds.runtime.data.SparseBlock.Type;
import org.apache.sysds.runtime.data.SparseBlockCSR;
import org.apache.sysds.runtime.data.SparseBlockFactory;
import org.apache.sysds.runtime.data.SparseBlockMCSR;
+import org.apache.sysds.runtime.data.SparseRow;
import org.apache.sysds.runtime.data.SparseRowScalar;
import org.apache.sysds.runtime.data.SparseRowVector;
import org.apache.sysds.runtime.functionobjects.SwapIndex;
@@ -194,7 +195,7 @@ public class LibMatrixMult
(!fixedRet && isUltraSparseMatrixMult(m1, m2, m1Perm));
boolean sparse = !fixedRet && !ultraSparse && !m1Perm
&& isSparseOutputMatrixMult(m1, m2);
-
+
// allocate output
if(ret == null)
ret = new MatrixBlock(m1.rlen, m2.clen, ultraSparse |
sparse);
@@ -1718,7 +1719,6 @@ public class LibMatrixMult
matrixMultUltraSparseLeft(m1, m2, ret, rl, ru);
else
matrixMultUltraSparseRight(m1, m2, ret, rl, ru);
- //no need to recompute nonzeros because maintained internally
}
private static void matrixMultUltraSparseSelf(MatrixBlock m1,
MatrixBlock ret, int rl, int ru) {
@@ -1926,10 +1926,14 @@ public class LibMatrixMult
private static void matrixMultUltraSparseRight(MatrixBlock m1,
MatrixBlock m2, MatrixBlock ret, int rl, int ru) {
- if(!ret.isInSparseFormat() &&
ret.getDenseBlock().isContiguous())
+ if(ret.isInSparseFormat()){
+ if(m1.isInSparseFormat())
+
matrixMultUltraSparseRightSparseMCSRLeftSparseOut(m1, m2, ret, rl, ru);
+ else
+
matrixMultUltraSparseRightDenseLeftSparseOut(m1, m2, ret, rl, ru);
+ }
+ else if(ret.getDenseBlock().isContiguous())
matrixMultUltraSparseRightDenseOut(m1, m2, ret, rl, ru);
- else if(m1.isInSparseFormat() && ret.isInSparseFormat())
- matrixMultUltraSparseRightSparseMCSRLeftSparseOut(m1,
m2, ret, rl, ru);
else
matrixMultUltraSparseRightGeneric(m1, m2, ret, rl, ru);
}
@@ -1990,6 +1994,39 @@ public class LibMatrixMult
}
}
+ private static void
matrixMultUltraSparseRightDenseLeftSparseOut(MatrixBlock m1, MatrixBlock m2,
MatrixBlock ret, int rl, int ru) {
+ final int cd = m1.clen;
+ final DenseBlock a = m1.denseBlock;
+ final SparseBlock b = m2.sparseBlock;
+ final SparseBlockMCSR c = (SparseBlockMCSR) ret.sparseBlock;
+
+ for(int k = 0; k < cd; k++){
+ if(b.isEmpty(k))
+ continue; // skip emptry rows right side.
+ final int bpos = b.pos(k);
+ final int blen = b.size(k);
+ final int[] bixs = b.indexes(k);
+ final double[] bvals = b.values(k);
+ for(int i = rl; i < ru; i++)
+ mmDenseMatrixSparseRow(bpos, blen, bixs, bvals,
k, i, a, c);
+ }
+ }
+
+ private static void mmDenseMatrixSparseRow(int bpos, int blen, int[]
bixs, double[] bvals, int k, int i,
+ DenseBlock a, SparseBlockMCSR c) {
+ final double[] aval = a.values(i);
+ final int apos = a.pos(i);
+ if(!c.isAllocated(i))
+ c.allocate(i, Math.max(blen, 2));
+ final SparseRowVector srv = (SparseRowVect
(systemds) branch main updated: [MINOR] Increase central moment test startup time
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new ef5ba804c8 [MINOR] Increase central moment test startup time
ef5ba804c8 is described below
commit ef5ba804c8106da6b6423d8ec8c1b93acaca54bb
Author: Sebastian Baunsgaard
AuthorDate: Thu Nov 30 19:27:26 2023 +0100
[MINOR] Increase central moment test startup time
---
.../test/functions/federated/primitives/FederatedCentralMomentTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
b/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
index 98b72a9169..03bd7b3014 100644
---
a/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
+++
b/src/test/java/org/apache/sysds/test/functions/federated/primitives/FederatedCentralMomentTest.java
@@ -101,7 +101,7 @@ public class FederatedCentralMomentTest extends
AutomatedTestBase {
Thread t1 = startLocalFedWorkerThread(port1, FED_WORKER_WAIT_S);
Thread t2 = startLocalFedWorkerThread(port2, FED_WORKER_WAIT_S);
Thread t3 = startLocalFedWorkerThread(port3, FED_WORKER_WAIT_S);
- Thread t4 = startLocalFedWorkerThread(port4);
+ Thread t4 = startLocalFedWorkerThread(port4, FED_WORKER_WAIT +
1000);
// reference file should not be written to hdfs, so we set
platform here
rtplatform = execMode;
(systemds) branch main updated: [SYSTEMDS-3653] Ultra Sparse MM Optimization
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new fac955c757 [SYSTEMDS-3653] Ultra Sparse MM Optimization
fac955c757 is described below
commit fac955c757015fa6c2c187725f36afbaec6ee6f6
Author: Sebastian Baunsgaard
AuthorDate: Thu Nov 30 10:31:32 2023 +0100
[SYSTEMDS-3653] Ultra Sparse MM Optimization
This commit update the left side ultra sparse matrix
multiplication to remove indirections and optimize
JIT compilation. We see improvements of up to 9x in small examples.
Left side one non zero per row
100k by 1m %% 1m by 100 sp 0.1 -> Before: 6.5 After : 4.5 sec
Left side two non zero per row
200k by 1m %% 1m by 100 sp 0.1 -> Before 173.724 After : 19.5 sec
Left side one non zero per row
100k by 1m %% 1m by 100 sp 0.43 -> Before: 65.06 After : 29.039 sec
Closes #1951
---
.../runtime/compress/CompressedMatrixBlock.java| 9 +-
.../apache/sysds/runtime/data/SparseBlockMCSR.java | 2 +-
.../matrix/data/LibMatrixDenseToSparse.java| 160 +++--
.../sysds/runtime/matrix/data/LibMatrixMult.java | 198 +++--
.../matrix/data/LibMatrixSparseToDense.java| 184 +++
.../sysds/runtime/matrix/data/MatrixBlock.java | 93 +-
6 files changed, 369 insertions(+), 277 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/CompressedMatrixBlock.java
b/src/main/java/org/apache/sysds/runtime/compress/CompressedMatrixBlock.java
index 564037cb48..92200d4384 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/CompressedMatrixBlock.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/CompressedMatrixBlock.java
@@ -1152,12 +1152,17 @@ public class CompressedMatrixBlock extends MatrixBlock {
}
@Override
- public void examSparsity(boolean allowCSR) {
+ public void examSparsity(boolean allowCSR, int k) {
// do nothing
}
@Override
- public void sparseToDense() {
+ public void sparseToDense(int k) {
+ // do nothing
+ }
+
+ @Override
+ public void denseToSparse(boolean allowCSR, int k){
// do nothing
}
diff --git a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
index e889d58b68..08dbc8b0a4 100644
--- a/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
+++ b/src/main/java/org/apache/sysds/runtime/data/SparseBlockMCSR.java
@@ -291,7 +291,7 @@ public class SparseBlockMCSR extends SparseBlock
@Override
public final boolean isEmpty(int r) {
- return !isAllocated(r) || _rows[r].isEmpty();
+ return _rows[r] == null || _rows[r].isEmpty();
}
@Override
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixDenseToSparse.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixDenseToSparse.java
index 5280aa5f9b..7c687578d0 100644
---
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixDenseToSparse.java
+++
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixDenseToSparse.java
@@ -26,7 +26,6 @@ import java.util.concurrent.Future;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import
org.apache.sysds.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer;
import org.apache.sysds.runtime.data.DenseBlock;
import org.apache.sysds.runtime.data.SparseBlockCSR;
import org.apache.sysds.runtime.data.SparseBlockMCSR;
@@ -44,6 +43,10 @@ public interface LibMatrixDenseToSparse {
* @param allowCSR If CSR is allowed.
*/
public static void denseToSparse(MatrixBlock r, boolean allowCSR) {
+ denseToSparse(r, allowCSR, 1);
+ }
+
+ public static void denseToSparse(MatrixBlock r, boolean allowCSR, int
k) {
final DenseBlock a = r.getDenseBlock();
// set target representation, early abort on empty blocks
@@ -51,12 +54,10 @@ public interface LibMatrixDenseToSparse {
if(a == null)
return;
- final int k = InfrastructureAnalyzer.getLocalParallelism();
-
- if(k > 1 && r.getNumRows() > 1000)
+ if(k > 1 && r.getSparsity() > 0.01 && (r.rlen > 100 || ((long)
r.rlen * r.clen > 10)))
denseToSparseParallel(r, k, allowCSR);
else if(allowCSR && r.nonZeros <= Integer.MAX_VALUE)
- denseToSparseCSR(r);
+ denseToSparseCSRSafe(r);
else
denseToSparseMCSR(r);
(systemds) branch main updated: [MINOR] Forward pass for ResNet18 and 34
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 8dfe21167b [MINOR] Forward pass for ResNet18 and 34
8dfe21167b is described below
commit 8dfe21167b047998e1430626a25a5f398891c8f3
Author: MaximilianTUB
AuthorDate: Thu Nov 9 18:46:08 2023 +0100
[MINOR] Forward pass for ResNet18 and 34
This commit contains the building blocks
for the ResNet primitive of ResNet18 and ResNet34.
Closes #1944
---
scripts/nn/networks/resnet.dml | 64 +++
scripts/nn/networks/resnet18.dml | 94
scripts/nn/networks/resnet34.dml | 92 +++
3 files changed, 223 insertions(+), 27 deletions(-)
diff --git a/scripts/nn/networks/resnet.dml b/scripts/nn/networks/resnet.dml
index ae3c043b91..a7a62cb222 100644
--- a/scripts/nn/networks/resnet.dml
+++ b/scripts/nn/networks/resnet.dml
@@ -165,8 +165,8 @@ basic_block_forward = function(matrix[double] X,
list[unknown] weights,
ema_means_vars_upd = list(ema_mean_bn1_upd, ema_var_bn1_upd,
ema_mean_bn2_upd, ema_var_bn2_upd)
if (downsample) {
-ema_means_vars_upd = append(ema_means_vars, ema_mean_bn3_upd)
-ema_means_vars_upd = append(ema_means_vars, ema_var_bn3_upd)
+ema_means_vars_upd = append(ema_means_vars_upd, ema_mean_bn3_upd)
+ema_means_vars_upd = append(ema_means_vars_upd, ema_var_bn3_upd)
}
}
@@ -224,21 +224,25 @@ basic_reslayer_forward = function(matrix[double] X, int
Hin, int Win, int blocks
}
}
-resnet18_forward = function(matrix[double] X, int Hin, int Win,
-list[unknown] model, string mode,
-list[unknown] ema_means_vars)
+resnet_basic_forward = function(matrix[double] X, int Hin, int Win,
+list[unknown] layer_sizes,
+list[unknown] model, string mode,
+list[unknown] ema_means_vars)
return (matrix[double] out, list[unknown] ema_means_vars_upd) {
/*
- * Forward pass of the ResNet 18 model as introduced in
- * "Deep Residual Learning for Image Recognition" by
- * Kaiming He et. al. and inspired by the PyTorch
- * implementation.
+ * Forward pass of the ResNet 18 and 34 model as introduced
+ * in "Deep Residual Learning for Image Recognition" by
+ * Kaiming He et. al. and inspired by the PyTorch.
*
* Inputs:
* - X: Inputs, of shape (N, C_in*Hin*Win).
* C_in = 3 is expected.
* - Hin: Input height.
* - Win: Input width.
+ * - layer_sizes: List of the sizes of each of
+ * the 4 residual layers.
+ * For ResNet18: [2, 2, 2, 2]
+ * For ResNet34: [3, 4, 6, 3]
* - model: Weights and bias matrices of the model
* with the following order/content:
* -> 1: Weights of conv 1 7x7, of shape (64, 3*7*7)
@@ -254,10 +258,8 @@ resnet18_forward = function(matrix[double] X, int Hin, int
Win,
* with 512 base channels.
* List of residual layers 1, 2, 3 & 4 have
* the content/order:
- * -> 1: List of weights for first residual
- *block.
- * -> 2: List of weights for second residual
- *block.
+ * -> i: List of weights for residual block i.
+ *with i in {1, ..., layer_sizes[layer]}
* Each list of weights for a residual block
* must follow the same order as defined in
* the documentation of basic_block_forward().
@@ -276,8 +278,8 @@ resnet18_forward = function(matrix[double] X, int Hin, int
Win,
* -> 6: List of EMA means and vars for residual layer 4.
* Lists for EMAs of layer 1, 2, 3 & 4 must have the
* following order:
- * -> 1: List of EMA means and vars for residual block 1.
- * -> 2: List of EMA means and vars for residual block 2.
+ * -> i: List of EMA means and vars for residual block i.
+ *with i in {1, ..., layer_sizes[layer]}
* Each list of EMAs for a residual block
* must follow the same order as defined in
* the documentation of basic_block_forward().
@@ -330,28 +332,36 @@ resnet18_forward = function(matrix[double] X, int Hin,
int Win,
Wf=3, strideh=2, stridew=2, padh=1, padw=1)
# residual layer 1
+block_count = as.integer(as.scalar(layer_sizes[1]))
[out, Hout, Wout, emas1_upd] = basic_reslayer_forward(X=out, Hin=Hout,
- Win=Wout, blocks=2, strideh=1, stridew=1,
C_in=C,
- C_base=64, blocks_weights=weights_
(systemds) branch main updated: [MINOR] JIT optimize LibMatrixBinCell
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 2b8de1629b [MINOR] JIT optimize LibMatrixBinCell
2b8de1629b is described below
commit 2b8de1629b935d0b75caf38e4295c706980f0ce7
Author: Sebastian Baunsgaard
AuthorDate: Thu Oct 26 18:24:54 2023 +0200
[MINOR] JIT optimize LibMatrixBinCell
This commit move some of the code inside LibMatrixBincell around to
encourage jit compilation of some methods. In specific folloing methods
have been introduced.
- safeBinaryMvSparseRowVector
- fillZeroValuesEmpty
- fillZeroValuesDense
- fillZeroValuesSparse
- safeBinaryMMDenseDenseDensePM_Vec (Plus Multiply kernel vectorized)
- safeBinaryMMDenseDenseDensePM (Plus Multiply kernel small input)
- safeBinaryMMDenseDenseDenseContiguous (This one makes a big difference)
- safeBinaryMMDenseDenseDenseGeneric
In specific the safeBinaryMMDenseDenseDenseContiguous,
safeBinaryMMDenseDenseDensePMm and safeBinaryMMDenseDenseDensePM_Vec
improve the performance by much.
In LM_cg the performance:
Stats output:
+* 3.123 3000 (Before)
+* 1.991 3000 (After)
+ 1.125 2021 (Before)
+ 0.703 2015 (After)
This is training on Criteo 100k rows.
---
.../runtime/matrix/data/LibMatrixBincell.java | 430 +
.../sysds/runtime/matrix/data/LibMatrixMult.java | 2 +-
2 files changed, 269 insertions(+), 163 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixBincell.java
b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixBincell.java
index e53f09a7f4..e5ec7a0020 100644
--- a/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixBincell.java
+++ b/src/main/java/org/apache/sysds/runtime/matrix/data/LibMatrixBincell.java
@@ -851,85 +851,93 @@ public class LibMatrixBincell {
private static void safeBinaryMVSparse(MatrixBlock m1, MatrixBlock m2,
MatrixBlock ret, BinaryOperator op) {
boolean isMultiply = (op.fn instanceof Multiply);
boolean skipEmpty = (isMultiply || isSparseSafeDivide(op, m2));
-
- int rlen = m1.rlen;
- int clen = m1.clen;
- SparseBlock a = m1.sparseBlock;
BinaryAccessType atype = getBinaryAccessType(m1, m2);
-
- //early abort on skip and empty
- if( skipEmpty && (m1.isEmptyBlock(false) ||
m2.isEmptyBlock(false) ) )
+
+ // early abort on skip and empty
+ if(skipEmpty && (m1.isEmptyBlock(false) ||
m2.isEmptyBlock(false)))
return; // skip entire empty block
-
- //allocate once in order to prevent repeated reallocation
- if( ret.sparse )
+
+ // allocate once in order to prevent repeated reallocation
+ if(ret.sparse)
ret.allocateSparseRowsBlock();
-
- if( atype == BinaryAccessType.MATRIX_COL_VECTOR )
- {
- for( int i=0; i
aix[apos]){
- apos++;
- }
- // for each point in the sparse range
- for(; apos < alen && aix[apos] < len; apos++){
- if(!zeroIsZero){
- while(cpos < len && cpos <
aix[apos]){
- ret.appendValue(rpos,
cpos++, zero);
- }
- }
- cpos = aix[apos];
- final double v = op.fn.execute(0,
vals[apos]);
- ret.appendValue(rpos, aix[apos], v);
- // cpos++;
- }
- // process tail.
+ }
+ else {
+ // def
+ for(int k = cpos; k < len; k++) {
+ ret.appendValue(rpos, k, op.fn.execute(0,
vals[k]));
+ }
+ }
+ }
+
+ private static void fillZeroValuesSparse(BinaryOperator op, MatrixBlock
m2, MatrixBlock ret, boolean skipEmpty,
+ int rpos, int cpos, int len) {
+
+ final double zero = op.fn.execute(0.0, 0.0);
+ final boolean zeroIsZero = zero == 0.0;
+ final SparseBlock sb = m2.getSparseBlock();
+ if(sb.isEmpty(0)) {
+ if(!zeroIsZer
(systemds) branch main updated: [MINOR] Performance tests for compressed behavior
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 798a0df3fc [MINOR] Performance tests for compressed behavior
798a0df3fc is described below
commit 798a0df3fc179b3a4d7a903fd3755b23f52828c2
Author: Sebastian Baunsgaard
AuthorDate: Fri Oct 20 17:38:34 2023 +0200
[MINOR] Performance tests for compressed behavior
Closes #1928
---
.../java/org/apache/sysds/performance/Main.java| 8 ++-
.../org/apache/sysds/performance/PerfUtil.java | 12 ++--
.../org/apache/sysds/performance/TimingUtils.java | 2 +
.../sysds/performance/compression/Serialize.java | 79 ++
.../performance/compression/TransformPerf.java | 14 ++--
.../apache/sysds/performance/generators/Const.java | 2 +-
.../sysds/performance/generators/ConstFrame.java | 64 +-
.../sysds/performance/generators/FrameFile.java| 76 ++---
.../performance/generators/FrameTransformFile.java | 78 +
.../sysds/performance/generators/MatrixFile.java | 46 ++---
.../sysds/performance/simple/DetectTypeArray.java | 38 +--
.../org/apache/sysds/performance/simple/NNZ.java | 48 ++---
12 files changed, 253 insertions(+), 214 deletions(-)
diff --git a/src/test/java/org/apache/sysds/performance/Main.java
b/src/test/java/org/apache/sysds/performance/Main.java
index 4e8f566a30..185a43e2c3 100644
--- a/src/test/java/org/apache/sysds/performance/Main.java
+++ b/src/test/java/org/apache/sysds/performance/Main.java
@@ -132,8 +132,10 @@ public class Main {
double sparsity = Double.parseDouble(args[4]);
int k = Integer.parseInt(args[5]);
int n = Integer.parseInt(args[6]);
-
- Serialize s = new Serialize(n, new ConstMatrix(rows, cols,
unique, sparsity), k);
+ //args[7] is id
+ Serialize s = (args.length == 9) ? //
+ new Serialize(n, new ConstMatrix(rows, cols, unique,
sparsity), k) : //
+ new Serialize(n, new ConstMatrix(rows, cols, unique,
sparsity), k, args[7], args[8]);
if(id == -1)
s.run();
@@ -179,7 +181,7 @@ public class Main {
private static void run16(String[] args) {
int len = Integer.parseInt(args[1]);
- MatrixBlock mb =
TestUtils.ceil(TestUtils.generateTestMatrixBlock(len, len, 0, 100, 0.01, len
+1));
+ MatrixBlock mb =
TestUtils.ceil(TestUtils.generateTestMatrixBlock(len, len, 0, 100, 0.01, len +
1));
System.out.println(mb);
}
diff --git a/src/test/java/org/apache/sysds/performance/PerfUtil.java
b/src/test/java/org/apache/sysds/performance/PerfUtil.java
index f93b03bdb3..9115bf5878 100644
--- a/src/test/java/org/apache/sysds/performance/PerfUtil.java
+++ b/src/test/java/org/apache/sysds/performance/PerfUtil.java
@@ -25,10 +25,10 @@ import java.io.InputStream;
public interface PerfUtil {
-public static String readSpec(String path) throws IOException {
-InputStream in = new FileInputStream(path);
-String spec = new String(in.readAllBytes());
-in.close();
-return spec;
-}
+ public static String readSpec(String path) throws IOException {
+ InputStream in = new FileInputStream(path);
+ String spec = new String(in.readAllBytes());
+ in.close();
+ return spec;
+ }
}
diff --git a/src/test/java/org/apache/sysds/performance/TimingUtils.java
b/src/test/java/org/apache/sysds/performance/TimingUtils.java
index 11e2c1dca5..0faf01c9b0 100644
--- a/src/test/java/org/apache/sysds/performance/TimingUtils.java
+++ b/src/test/java/org/apache/sysds/performance/TimingUtils.java
@@ -21,6 +21,7 @@ package org.apache.sysds.performance;
import java.util.Arrays;
+import org.apache.sysds.api.DMLScript;
import org.apache.sysds.performance.generators.IGenerate;
import org.apache.sysds.runtime.controlprogram.parfor.stat.Timing;
@@ -93,6 +94,7 @@ public interface TimingUtils {
b.run();
while(bq.isEmpty())
Thread.sleep(bq.defaultWaitTime());
+ DMLScript.SEED = i + 1000;
time(f, times, i);
c.run();
}
diff --git
a/src/test/java/org/apache/sysds/performance/compression/Serialize.java
b/src/test/java/org/apache/sysds/performance/compression/Serialize.java
index 12316874c1..802e7f3a7b 100644
--- a/src/test/java/org/apache/sysds/performance/compression/Serialize.java
+++ b/src/test/java/org/apache/sysds/performance/compression/Serialize.java
@@ -38,9 +38,13 @@ import
org.apache.sysds.runtime.compress.CompressedMatrixBlock
(systemds) branch main updated: [MINOR] fix empty nnz Compressed LLM
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 9487373906 [MINOR] fix empty nnz Compressed LLM
9487373906 is described below
commit 948737390683c2a7b11e3f79d2a0303da4c77738
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 15:33:43 2023 +0100
[MINOR] fix empty nnz Compressed LLM
---
.../java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
index 30c1109d3a..d0983d4ae0 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
@@ -168,8 +168,8 @@ public final class CLALibLeftMultBy {
final List fLeft = CLALibUtils.filterGroups(leftCG,
cL);
// Force dense output
- ret.setNonZeros((long) ret.getNumRows() * ret.getNumColumns());
ret.allocateDenseBlock();
+ ret.setNonZeros((long) ret.getNumRows() * ret.getNumColumns());
final ExecutorService ex = CommonThreadPool.get(k);
final List> t = new ArrayList<>();
@@ -196,6 +196,7 @@ public final class CLALibLeftMultBy {
outerProduct(cL, CLALibUtils.getColSum(fRight,
cr, sd), retV);
if(containsRight)// if right -- multiply right with
left sum
outerProduct(CLALibUtils.getColSum(fLeft, rl,
sd), cR, retV);
+
for(Future f : t) {
MatrixBlock mb = f.get();
if(!mb.isEmpty()) {
(systemds) branch main updated: [MINOR] Workload Analyzer Warn on unknown
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 7136a6aa92 [MINOR] Workload Analyzer Warn on unknown
7136a6aa92 is described below
commit 7136a6aa922867aba3b047962e3931c820a66fac
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 15:07:12 2023 +0100
[MINOR] Workload Analyzer Warn on unknown
The AWARE workload analyzer previously errored out on operations that
are unknown, now instead we write a warning, and assume all unknown
operations are decompressing the output.
---
.../runtime/compress/workload/WorkloadAnalyzer.java| 18 +++---
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/workload/WorkloadAnalyzer.java
b/src/main/java/org/apache/sysds/runtime/compress/workload/WorkloadAnalyzer.java
index 68b60438fa..a4c15b2b53 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/workload/WorkloadAnalyzer.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/workload/WorkloadAnalyzer.java
@@ -60,7 +60,6 @@ import org.apache.sysds.parser.ParForStatementBlock;
import org.apache.sysds.parser.StatementBlock;
import org.apache.sysds.parser.WhileStatement;
import org.apache.sysds.parser.WhileStatementBlock;
-import org.apache.sysds.runtime.compress.DMLCompressionException;
import org.apache.sysds.runtime.compress.workload.AWTreeNode.WTNodeType;
import org.apache.sysds.utils.Explain;
@@ -68,7 +67,7 @@ public class WorkloadAnalyzer {
private static final Log LOG =
LogFactory.getLog(WorkloadAnalyzer.class.getName());
// indicator for more aggressive compression of intermediates
public static boolean ALLOW_INTERMEDIATE_CANDIDATES = false;
- // avoid wtree construction for assumptionly already compressed
intermediates
+ // avoid w-tree construction for already compressed intermediates
// (due to conditional control flow this might miss compression
opportunities)
public static boolean PRUNE_COMPRESSED_INTERMEDIATES = true;
@@ -96,6 +95,7 @@ public class WorkloadAnalyzer {
// construct workload tree for candidate
WorkloadAnalyzer wa = new WorkloadAnalyzer(prog);
WTreeRoot tree = wa.createWorkloadTree(cand);
+
map.put(cand.getHopID(), tree);
allWAs.add(wa);
}
@@ -337,6 +337,7 @@ public class WorkloadAnalyzer {
}
private void createOp(Hop hop, AWTreeNode parent) {
+
if(hop.getDataType().isMatrix()) {
Op o = null;
if(HopRewriteUtils.isData(hop, OpOpData.PERSISTENTREAD,
OpOpData.TRANSIENTREAD))
@@ -425,7 +426,11 @@ public class WorkloadAnalyzer {
o.setOverlapping();
}
else if(ol) {
-
treeLookup.get(in.get(0).getHopID()).setDecompressing();
+ if(in.get(0) != null) {
+ Op oo =
treeLookup.get(in.get(0).getHopID());
+ if(oo != null)
+
oo.setDecompressing();
+ }
return;
}
else {
@@ -500,16 +505,15 @@ public class WorkloadAnalyzer {
setDecompressionOnAllInputs(hop,
parent);
}
}
- else if(hop instanceof ParameterizedBuiltinOp) {
+ else if(hop instanceof ParameterizedBuiltinOp || hop
instanceof NaryOp) {
setDecompressionOnAllInputs(hop, parent);
return;
}
- else if(hop instanceof NaryOp){
+ else {
+ LOG.warn("Unknown Hop:" +
hop.getClass().getSimpleName() + "\n" + Explain.explain(hop));
setDecompressionOnAllInputs(hop, parent);
return;
}
- else
- throw new DMLCompressionException("Unknown
Hop:" +hop.getClass().getSimpleName() +"\n" + Explain.explain(hop));
(systemds) branch main updated: [MINOR] Parallel Compressed LMM
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new fb60577586 [MINOR] Parallel Compressed LMM
fb60577586 is described below
commit fb605775865d2ec0fbcc3aff81975576f8baa5e1
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 15:05:17 2023 +0100
[MINOR] Parallel Compressed LMM
---
.../runtime/compress/lib/CLALibLeftMultBy.java | 96 --
.../sysds/runtime/compress/lib/CLALibMMChain.java | 42 ++
.../runtime/compress/lib/CLALibRightMultBy.java| 4 +-
3 files changed, 133 insertions(+), 9 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
index 6029a87d46..30c1109d3a 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibLeftMultBy.java
@@ -32,11 +32,14 @@ import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.sysds.runtime.DMLRuntimeException;
import org.apache.sysds.runtime.compress.CompressedMatrixBlock;
+import org.apache.sysds.runtime.compress.DMLCompressionException;
import org.apache.sysds.runtime.compress.colgroup.AColGroup;
import org.apache.sysds.runtime.compress.colgroup.APreAgg;
import org.apache.sysds.runtime.data.DenseBlock;
import org.apache.sysds.runtime.data.SparseBlock;
import org.apache.sysds.runtime.functionobjects.Plus;
+import org.apache.sysds.runtime.matrix.data.LibMatrixBincell;
+import org.apache.sysds.runtime.matrix.data.LibMatrixMult;
import org.apache.sysds.runtime.matrix.data.LibMatrixReorg;
import org.apache.sysds.runtime.matrix.data.MatrixBlock;
import org.apache.sysds.runtime.matrix.operators.BinaryOperator;
@@ -45,7 +48,7 @@ import org.apache.sysds.runtime.util.CommonThreadPool;
public final class CLALibLeftMultBy {
private static final Log LOG =
LogFactory.getLog(CLALibLeftMultBy.class.getName());
- private CLALibLeftMultBy(){
+ private CLALibLeftMultBy() {
// private constructor
}
@@ -139,7 +142,15 @@ public final class CLALibLeftMultBy {
}
private static MatrixBlock
leftMultByCompressedTransposedMatrix(CompressedMatrixBlock right,
- CompressedMatrixBlock left, MatrixBlock ret, int k) {
+ CompressedMatrixBlock left, final MatrixBlock ret, int k) {
+ if(k > 1 && ret.getInMemorySize() < 100)
+ return
leftMultByCompressedTransposedMatrixParallel(right, left, ret, k);
+ else
+ return
leftMultByCompressedTransposedMatrixSingleThread(right, left, ret);
+ }
+
+ private static MatrixBlock
leftMultByCompressedTransposedMatrixParallel(CompressedMatrixBlock right,
+ CompressedMatrixBlock left, final MatrixBlock ret, int k) {
final int sd = right.getNumRows(); // shared dim
final int cr = right.getNumColumns();
@@ -149,18 +160,88 @@ public final class CLALibLeftMultBy {
final List leftCG = left.getColGroups();
final boolean containsRight =
CLALibUtils.shouldPreFilter(rightCG);
- double[] cR = containsRight ? new double[cr] : null;
+ final double[] cR = containsRight ? new double[cr] : null;
final List fRight =
CLALibUtils.filterGroups(rightCG, cR);
final boolean containsLeft =
CLALibUtils.shouldPreFilter(leftCG);
- double[] cL = containsLeft ? new double[rl] : null;
+ final double[] cL = containsLeft ? new double[rl] : null;
final List fLeft = CLALibUtils.filterGroups(leftCG,
cL);
+ // Force dense output
+ ret.setNonZeros((long) ret.getNumRows() * ret.getNumColumns());
+ ret.allocateDenseBlock();
+
+ final ExecutorService ex = CommonThreadPool.get(k);
+ final List> t = new ArrayList<>();
+
+ for(int j = 0; j < fLeft.size(); j++) {
+ final int jj = j;
+ t.add(ex.submit(() -> {
+ MatrixBlock retT = new
MatrixBlock(ret.getNumRows(), ret.getNumColumns(), false);
+ retT.allocateDenseBlock();
+ for(int i = 0; i < fRight.size(); i++) {
+
fRight.get(i).leftMultByAColGroup(fLeft.get(jj), retT, sd);
+ }
+ retT.examSparsity(true);
+ return retT;
+ }));
+ }
+
+ try {
+
(systemds) branch main updated: [MINOR] Fix Empty Binary CLA Empty
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 3126e5f794 [MINOR] Fix Empty Binary CLA Empty
3126e5f794 is described below
commit 3126e5f794ffc46ca66a61ebce28999fd952b09f
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 14:49:01 2023 +0100
[MINOR] Fix Empty Binary CLA Empty
This commit fixes binary Matrix Vector/Matrix CLA operations to support
empty sides in some edge case not supported yet, for instance <=.
---
.../runtime/compress/lib/CLALibBinaryCellOp.java | 30 +-
1 file changed, 24 insertions(+), 6 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibBinaryCellOp.java
b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibBinaryCellOp.java
index 13e5e3c938..ede9ca46aa 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibBinaryCellOp.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/lib/CLALibBinaryCellOp.java
@@ -74,9 +74,14 @@ public final class CLALibBinaryCellOp {
ScalarOperator sop = new RightScalarOperator(op.fn,
that.getValue(0, 0), op.getNumThreads());
return CLALibScalar.scalarOperations(sop, m1, result);
}
- if(that.isEmpty())
+ else if(that.isEmpty())
return binaryOperationsEmpty(op, m1, that, result);
+ else
+ return binaryOperationsRightFiltered(op, m1, that,
result);
+ }
+ private static MatrixBlock binaryOperationsRightFiltered(BinaryOperator
op, CompressedMatrixBlock m1,
+ MatrixBlock that, MatrixBlock result) {
LibMatrixBincell.isValidDimensionsBinaryExtended(m1, that);
BinaryAccessType atype =
LibMatrixBincell.getBinaryAccessTypeExtended(m1, that);
@@ -113,17 +118,16 @@ public final class CLALibBinaryCellOp {
final ValueFunction fn = op.fn;
if(fn instanceof Multiply)
- result =
CompressedMatrixBlockFactory.createConstant(m1Row, m1Col, 0);
+ return
CompressedMatrixBlockFactory.createConstant(m1Row, m1Col, 0);
else if(fn instanceof Minus1Multiply)
- result =
CompressedMatrixBlockFactory.createConstant(m1Row, m1Col, 1);
+ return
CompressedMatrixBlockFactory.createConstant(m1Row, m1Col, 1);
else if(fn instanceof Minus || fn instanceof Plus || fn
instanceof MinusMultiply || fn instanceof PlusMultiply) {
CompressedMatrixBlock ret = new CompressedMatrixBlock();
ret.copy(m1);
return ret;
}
else
- throw new NotImplementedException("Function Type: " +
fn);
- return result;
+ return binaryOperationsRightFiltered(op, m1, that,
result);
}
private static MatrixBlock
selectProcessingBasedOnAccessType(BinaryOperator op, CompressedMatrixBlock m1,
@@ -612,8 +616,11 @@ public final class CLALibBinaryCellOp {
}
private final void processRight(final int rl, final int ru) {
+
+ if(_m2.isEmpty())
+ processRightEmpty(rl, ru);
// all exec should have ret on left side
- if(_m2.isInSparseFormat())
+ else if(_m2.isInSparseFormat())
processRightSparse(rl, ru);
else
processRightDense(rl, ru);
@@ -662,6 +669,17 @@ public final class CLALibBinaryCellOp {
retV[c] = _op.fn.execute(retV[c],
m2V[c]);
}
}
+
+ private final void processRightEmpty(final int rl, final int
ru) {
+ final DenseBlock rv = _ret.getDenseBlock();
+ final int cols = _ret.getNumColumns();
+ for(int r = rl; r < ru; r++) {
+ final double[] retV = rv.values(r);
+ int off = rv.pos(r);
+ for(int c = off; c < cols + off; c++)
+ retV[c] = _op.fn.execute(retV[c], 0);
+ }
+ }
}
private static class BinaryMVColLeftTask implements Callable {
(systemds) branch main updated: [SYSTEMDS-3643] Fused Scaling Compressed Multiplication
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 0ba2aa994f [SYSTEMDS-3643] Fused Scaling Compressed Multiplication
0ba2aa994f is described below
commit 0ba2aa994f8f3006a2a660c8cad4fdd8e78ac94f
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 13:30:15 2023 +0100
[SYSTEMDS-3643] Fused Scaling Compressed Multiplication
This commit contains the code to fuse the scaling part into the
Matrix Multiplication kernels of CLA. This is used to not allocate
new Dictionaries, when the two column group sides have identical
index structures.
The change improve instructions such as MMChain and TSMM. The improvements
are biggest if there are few column groups.
Closes #1936
---
.../sysds/runtime/compress/colgroup/APreAgg.java | 5 +-
.../colgroup/dictionary/DictLibMatrixMult.java | 127 +--
.../compress/colgroup/dictionary/Dictionary.java | 48 -
.../compress/colgroup/dictionary/IDictionary.java | 94 ++---
.../colgroup/dictionary/IdentityDictionary.java| 168 +--
.../dictionary/IdentityDictionarySlice.java| 23 +-
.../colgroup/dictionary/MatrixBlockDictionary.java | 71 ++-
.../colgroup/dictionary/PlaceHolderDict.java | 18 ++
.../compress/colgroup/dictionary/QDictionary.java | 18 ++
.../sysds/runtime/data/SparseBlockFactory.java | 45 +++-
src/test/java/org/apache/sysds/test/TestUtils.java | 11 +
.../compress/dictionary/DictionaryTests.java | 232 -
.../sysds/test/component/matrix/SparseFactory.java | 42
13 files changed, 821 insertions(+), 81 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
index 8b8a7b7df0..7f585f2d7a 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
@@ -85,9 +85,12 @@ public abstract class APreAgg extends AColGroupValue {
* @return A aggregate dictionary
*/
public final IDictionary preAggregateThatIndexStructure(APreAgg that) {
- long outputLength = (long)that._colIndexes.size() *
this.getNumValues();
+ final long outputLength = (long)that._colIndexes.size() *
this.getNumValues();
if(outputLength > Integer.MAX_VALUE)
throw new NotImplementedException("Not supported pre
aggregate of above integer length");
+ if(outputLength <= 0) // if the pre aggregate output is empty
or nothing, return null
+ return null;
+
// create empty Dictionary that we slowly fill, hence the
dictionary is empty and no check
final Dictionary ret = Dictionary.createNoCheck(new
double[(int)outputLength]);
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/dictionary/DictLibMatrixMult.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/dictionary/DictLibMatrixMult.java
index 240e57cc12..9aba711a30 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/dictionary/DictLibMatrixMult.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/dictionary/DictLibMatrixMult.java
@@ -65,11 +65,7 @@ public class DictLibMatrixMult {
*/
public static void MMDictsWithScaling(IDictionary left, IDictionary
right, IColIndex leftRows,
IColIndex rightColumns, MatrixBlock result, int[] counts) {
- LOG.warn("Inefficient double allocation of dictionary");
- final boolean modifyRight = right.getInMemorySize() >
left.getInMemorySize();
- final IDictionary rightM = modifyRight ?
right.scaleTuples(counts, rightColumns.size()) : right;
- final IDictionary leftM = modifyRight ? left :
left.scaleTuples(counts, leftRows.size());
- MMDicts(leftM, rightM, leftRows, rightColumns, result);
+ left.MMDictScaling(right, leftRows, rightColumns, result,
counts);
}
/**
@@ -198,17 +194,43 @@ public class DictLibMatrixMult {
protected static void MMDictsDenseDense(double[] left, double[] right,
IColIndex rowsLeft, IColIndex colsRight,
MatrixBlock result) {
- final int commonDim = Math.min(left.length / rowsLeft.size(),
right.length / colsRight.size());
+ final int leftSide = rowsLeft.size();
+ final int rightSide = colsRight.size();
+ final int commonDim = Math.min(left.length / leftSide,
right.length / rightSide);
final int resCols = result.getNumColumns();
(systemds) branch main updated: [SYSTEMDS-3644] Compressed-Compressed Transform Encode (PassThrough)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new c398e8ec5e [SYSTEMDS-3644] Compressed-Compressed Transform Encode
(PassThrough)
c398e8ec5e is described below
commit c398e8ec5e163647706ac309b8c854a62b594c97
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 13:55:26 2023 +0100
[SYSTEMDS-3644] Compressed-Compressed Transform Encode (PassThrough)
Initial instance of direct compressed frame to compressed matrix
transform encode, to start with in the case of PassThrough.
---
.../sysds/runtime/frame/data/columns/DDCArray.java| 6 +-
.../runtime/transform/encode/CompressedEncode.java| 19 +++
.../runtime/transform/encode/MultiColumnEncoder.java | 5 +++--
3 files changed, 27 insertions(+), 3 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/frame/data/columns/DDCArray.java
b/src/main/java/org/apache/sysds/runtime/frame/data/columns/DDCArray.java
index b634cfe6ff..8f3dcd9dcb 100644
--- a/src/main/java/org/apache/sysds/runtime/frame/data/columns/DDCArray.java
+++ b/src/main/java/org/apache/sysds/runtime/frame/data/columns/DDCArray.java
@@ -55,10 +55,14 @@ public class DDCArray extends ACompressedArray {
}
}
- protected Array getDict(){
+ public Array getDict(){
return dict;
}
+ public AMapToData getMap(){
+ return map;
+ }
+
/**
* Try to compress array into DDC format.
*
diff --git
a/src/main/java/org/apache/sysds/runtime/transform/encode/CompressedEncode.java
b/src/main/java/org/apache/sysds/runtime/transform/encode/CompressedEncode.java
index 8ca8b6d9fc..7fbdb1ea3c 100644
---
a/src/main/java/org/apache/sysds/runtime/transform/encode/CompressedEncode.java
+++
b/src/main/java/org/apache/sysds/runtime/transform/encode/CompressedEncode.java
@@ -49,7 +49,9 @@ import
org.apache.sysds.runtime.compress.colgroup.indexes.IColIndex;
import org.apache.sysds.runtime.compress.colgroup.mapping.AMapToData;
import org.apache.sysds.runtime.compress.colgroup.mapping.MapToFactory;
import org.apache.sysds.runtime.frame.data.FrameBlock;
+import org.apache.sysds.runtime.frame.data.columns.ACompressedArray;
import org.apache.sysds.runtime.frame.data.columns.Array;
+import org.apache.sysds.runtime.frame.data.columns.DDCArray;
import org.apache.sysds.runtime.matrix.data.MatrixBlock;
import org.apache.sysds.runtime.util.CommonThreadPool;
import org.apache.sysds.runtime.util.UtilFunctions;
@@ -164,6 +166,7 @@ public class CompressedEncode {
IColIndex colIndexes = ColIndexFactory.create(0, domain);
if(domain == 1 && !containsNull)
return ColGroupConst.create(colIndexes, new double[]
{1});
+
ADictionary d = new IdentityDictionary(colIndexes.size(),
containsNull);
AMapToData m = createMappingAMapToData(a, map, containsNull);
return ColGroupDDC.create(colIndexes, d, m, null);
@@ -288,6 +291,22 @@ public class CompressedEncode {
IColIndex colIndexes = ColIndexFactory.create(1);
int colId = c._colID;
Array a = in.getColumn(colId - 1);
+ if(a instanceof ACompressedArray){
+ switch(a.getFrameArrayType()) {
+ case DDC:
+ DDCArray aDDC = (DDCArray) a;
+ Array dict = aDDC.getDict();
+ double[] vals = new double[dict.size()];
+ for(int i = 0; i < dict.size(); i++) {
+ vals[i] = dict.getAsDouble(i);
+ }
+ ADictionary d = Dictionary.create(vals);
+
+ return ColGroupDDC.create(colIndexes,
d, aDDC.getMap(), null);
+ default:
+ throw new NotImplementedException();
+ }
+ }
boolean containsNull = a.containsNull();
HashMap map = (HashMap)
a.getRecodeMap();
final int blockSz =
ConfigurationManager.getDMLConfig().getIntValue(DMLConfig.DEFAULT_BLOCK_SIZE);
diff --git
a/src/main/java/org/apache/sysds/runtime/transform/encode/MultiColumnEncoder.java
b/src/main/java/org/apache/sysds/runtime/transform/encode/MultiColumnEncoder.java
index f1813e29a7..bd9e2ba79f 100644
---
a/src/main/java/org/apache/sysds/runtime/transform/encode/MultiColumnEncoder.java
+++
b/src/main/java/org/apache/sysds/runtime/transform/encode/MultiColumnEncoder.java
@@ -102,11 +102,12 @@ public class MultiCo
(systemds) branch main updated: [MINOR] Refine Error on Scalar compression
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 8ca0bf1eb4 [MINOR] Refine Error on Scalar compression
8ca0bf1eb4 is described below
commit 8ca0bf1eb4e4e5c55f4aa610d2cc54ce9705b77b
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 13:51:01 2023 +0100
[MINOR] Refine Error on Scalar compression
---
.../runtime/instructions/cp/CompressionCPInstruction.java | 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/instructions/cp/CompressionCPInstruction.java
b/src/main/java/org/apache/sysds/runtime/instructions/cp/CompressionCPInstruction.java
index b59e4d9db8..c9dd5c8961 100644
---
a/src/main/java/org/apache/sysds/runtime/instructions/cp/CompressionCPInstruction.java
+++
b/src/main/java/org/apache/sysds/runtime/instructions/cp/CompressionCPInstruction.java
@@ -22,6 +22,7 @@ package org.apache.sysds.runtime.instructions.cp;
import java.util.ArrayList;
import java.util.List;
+import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -122,10 +123,13 @@ public class CompressionCPInstruction extends
ComputationCPInstruction {
final int k = OptimizerUtils.getConstrainedNumThreads(-1);
- if(ec.isMatrixObject(input1.getName()))
- processMatrixBlockCompression(ec,
ec.getMatrixInput(input1.getName()), k, root);
- else
+ if(ec.isFrameObject(input1.getName()))
processFrameBlockCompression(ec,
ec.getFrameInput(input1.getName()), k, root);
+ else if(ec.isMatrixObject(input1.getName()))
+ processMatrixBlockCompression(ec,
ec.getMatrixInput(input1.getName()), k, root);
+ else{
+ throw new NotImplementedException("Not supported other
types of input for compression than frame and matrix");
+ }
}
private void processMatrixBlockCompression(ExecutionContext ec,
MatrixBlock in, int k, WTreeRoot root) {
(systemds) branch main updated: [MINOR] JIT optimize LMM Pre-aggregate
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new a826c10a51 [MINOR] JIT optimize LMM Pre-aggregate
a826c10a51 is described below
commit a826c10a5149f139918395151ce6d573a97dd663
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 13:24:32 2023 +0100
[MINOR] JIT optimize LMM Pre-aggregate
Because of abstract classes the efficiency of the JIT compiler
is subpar in the AMapToData instance. To improve this i have added
individual overwritten instructions in some of the Map types.
This duplicate code, but improve performance by 30-50% according to the
profiler.
---
.../compress/colgroup/mapping/AMapToData.java | 85 ++
.../compress/colgroup/mapping/MapToByte.java | 27 ---
.../compress/colgroup/mapping/MapToChar.java | 52 +
.../compress/colgroup/mapping/MapToCharPByte.java | 23 ++
.../compress/colgroup/mapping/MapToInt.java| 28 ---
.../compress/colgroup/mapping/MapToUByte.java | 28 ---
6 files changed, 167 insertions(+), 76 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/mapping/AMapToData.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/mapping/AMapToData.java
index b12461bf7c..b66c7ddb87 100644
---
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/mapping/AMapToData.java
+++
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/mapping/AMapToData.java
@@ -129,8 +129,8 @@ public abstract class AMapToData implements Serializable {
*
* @param n index to set.
* @param v the value to set it to.
-* @return v as encoded, note this value can be different that the one
put in if the map is not able to represent
-* the value
+* @return v as encoded, note this value can be different that the one
put in if the map is not able to represent the
+* value
*/
public abstract int setAndGet(int n, int v);
@@ -235,16 +235,19 @@ public abstract class AMapToData implements Serializable {
off += cl;
for(int rc = cl; rc < cl + h; rc++, off++)
preAV[getIndex(rc)] += mV[off];
- for(int rc = cl + h; rc < cu; rc += 8, off += 8) {
- preAV[getIndex(rc)] += mV[off];
- preAV[getIndex(rc + 1)] += mV[off + 1];
- preAV[getIndex(rc + 2)] += mV[off + 2];
- preAV[getIndex(rc + 3)] += mV[off + 3];
- preAV[getIndex(rc + 4)] += mV[off + 4];
- preAV[getIndex(rc + 5)] += mV[off + 5];
- preAV[getIndex(rc + 6)] += mV[off + 6];
- preAV[getIndex(rc + 7)] += mV[off + 7];
- }
+ for(int rc = cl + h; rc < cu; rc += 8, off += 8)
+ preAggregateDenseToRowVec8(mV, preAV, rc, off);
+ }
+
+ protected void preAggregateDenseToRowVec8(double[] mV, double[] preAV,
int rc, int off){
+ preAV[getIndex(rc)] += mV[off];
+ preAV[getIndex(rc + 1)] += mV[off + 1];
+ preAV[getIndex(rc + 2)] += mV[off + 2];
+ preAV[getIndex(rc + 3)] += mV[off + 3];
+ preAV[getIndex(rc + 4)] += mV[off + 4];
+ preAV[getIndex(rc + 5)] += mV[off + 5];
+ preAV[getIndex(rc + 6)] += mV[off + 6];
+ preAV[getIndex(rc + 7)] += mV[off + 7];
}
/**
@@ -329,8 +332,7 @@ public abstract class AMapToData implements Serializable {
* @param cu The column in m to end at (not inclusive)
* @param indexes The Offset Indexes to iterate through
*/
- public final void preAggregateDense(MatrixBlock m, double[] preAV, int
rl, int ru, int cl, int cu,
- AOffset indexes) {
+ public final void preAggregateDense(MatrixBlock m, double[] preAV, int
rl, int ru, int cl, int cu, AOffset indexes) {
indexes.preAggregateDenseMap(m, preAV, rl, ru, cl, cu,
getUnique(), this);
}
@@ -417,6 +419,8 @@ public abstract class AMapToData implements Serializable {
* @param nCol The number of columns
*/
public final void preAggregateDDC_DDC(AMapToData tm, IDictionary td,
Dictionary ret, int nCol) {
+ if(td.getNumberOfValues(nCol) != tm.nUnique)
+ throw new DMLCompressionException("Invalid map and dict
combination");
if(nCol == 1)
preAggregateDDC_DDCSingleCol(tm, td.getValues(),
ret.getValues());
else
@@ -431,31 +435,55 @@ public abstract class AMapToData implements Serializable {
* @param ret The output dict
(systemds) branch main updated: [SYSTEMDS-3642] CLA NaN in Dictionaries replace
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new c21fa9997d [SYSTEMDS-3642] CLA NaN in Dictionaries replace
c21fa9997d is described below
commit c21fa9997deadc7534b40b3b303a445b3c68c630
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 12:34:43 2023 +0100
[SYSTEMDS-3642] CLA NaN in Dictionaries replace
This commit fixes a bug of replace in ColumnGroups that did not
correctly replace NaN values with replacement values.
Example:
X_test = replace(target=X_test, pattern=NaN, replacement=0);
---
.../runtime/compress/colgroup/ColGroupDDC.java | 40 +++---
.../runtime/compress/colgroup/ColGroupDDCFOR.java | 5 +--
.../runtime/compress/colgroup/ColGroupSDC.java | 3 +-
.../runtime/compress/colgroup/ColGroupSDCFOR.java | 3 +-
.../compress/colgroup/ColGroupSDCSingle.java | 3 +-
.../compress/colgroup/ColGroupUncompressed.java| 14 +---
.../compress/colgroup/mapping/AMapToData.java | 13 +++
7 files changed, 66 insertions(+), 15 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupDDC.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupDDC.java
index 8f5fccaf7d..6340affede 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupDDC.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/colgroup/ColGroupDDC.java
@@ -35,6 +35,8 @@ import
org.apache.sysds.runtime.compress.colgroup.dictionary.MatrixBlockDictiona
import org.apache.sysds.runtime.compress.colgroup.indexes.ColIndexFactory;
import org.apache.sysds.runtime.compress.colgroup.indexes.IColIndex;
import org.apache.sysds.runtime.compress.colgroup.mapping.AMapToData;
+import org.apache.sysds.runtime.compress.colgroup.mapping.MapToByte;
+import org.apache.sysds.runtime.compress.colgroup.mapping.MapToChar;
import org.apache.sysds.runtime.compress.colgroup.mapping.MapToFactory;
import org.apache.sysds.runtime.compress.colgroup.offset.AOffsetIterator;
import org.apache.sysds.runtime.compress.colgroup.scheme.DDCScheme;
@@ -78,8 +80,8 @@ public class ColGroupDDC extends APreAgg implements
IMapToDataGroup {
int[] c = getCounts();
if(c.length !=
dict.getNumberOfValues(colIndexes.size()))
throw new DMLCompressionException("Invalid DDC
Construction");
+ data.verify();
}
-
}
public static AColGroup create(IColIndex colIndexes, IDictionary dict,
AMapToData data, int[] cachedCounts) {
@@ -157,8 +159,37 @@ public class ColGroupDDC extends APreAgg implements
IMapToDataGroup {
private final void
decompressToDenseBlockDenseDictSingleColOutContiguous(DenseBlock db, int rl,
int ru, int offR,
int offC, double[] values) {
final double[] c = db.values(0);
- for(int i = rl, offT = rl + offR + _colIndexes.get(0) + offC; i
< ru; i++, offT++)
- c[offT] += values[_data.getIndex(i)];
+ decompressToDenseBlockDenseDictSingleColOutContiguous(c, rl,
ru, offR + _colIndexes.get(0), values, _data);
+ }
+
+ private final static void
decompressToDenseBlockDenseDictSingleColOutContiguous(double[] c, int rl, int
ru, int offR,
+ double[] values, AMapToData data) {
+
+ if(data instanceof MapToByte)
+
decompressToDenseBlockDenseDictSingleColOutContiguousByteM(c, rl, ru, offR,
values, (MapToByte) data);
+ else if(data instanceof MapToChar)
+
decompressToDenseBlockDenseDictSingleColOutContiguousCharM(c, rl, ru, offR,
values, (MapToChar) data);
+ else
+
decompressToDenseBlockDenseDictSingleColOutContiguousGenM(c, rl, ru, offR,
values, data);
+
+ }
+
+ private final static void
decompressToDenseBlockDenseDictSingleColOutContiguousByteM(double[] c, int rl,
int ru,
+ int offR, double[] values, MapToByte data) {
+ for(int i = rl, offT = rl + offR; i < ru; i++, offT++)
+ c[offT] += values[data.getIndex(i)];
+ }
+
+ private final static void
decompressToDenseBlockDenseDictSingleColOutContiguousCharM(double[] c, int rl,
int ru,
+ int offR, double[] values, MapToChar data) {
+ for(int i = rl, offT = rl + offR; i < ru; i++, offT++)
+ c[offT] += values[data.getIndex(i)];
+ }
+
+ private final static void
decompressToDenseBlockDenseDictSingleColOutContiguousGenM(double[] c, int rl,
int ru,
+ int offR, double[] values, AMapToData data) {
+ for(int i = rl, offT = rl + offR
(systemds) branch main updated: [MINOR] Filter pre-aggregate warning
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 7f04d1642c [MINOR] Filter pre-aggregate warning
7f04d1642c is described below
commit 7f04d1642c1a679457c8dc4d6f9003e5e2fc4bf3
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 12:24:06 2023 +0100
[MINOR] Filter pre-aggregate warning
In compressed linear algebra, we print a warning in case of
uncompressed matrix multiplication.
This commit filters that error out if the input is one column.
The one-column case is special since transposition is a no-op that
only touches metadata. Therefore, we filter this error.
Also introduced in this commit is an error where we try to allocate a
pre-aggregate output larger than Integer.MAX_VALUE. This happens
in cases where the number of columns in a single-column group is
large, such as in a recode-bin encoding scenario of transform encoding.
---
.../org/apache/sysds/runtime/compress/colgroup/APreAgg.java| 10 +++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
b/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
index 17f210865b..8b8a7b7df0 100644
--- a/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
+++ b/src/main/java/org/apache/sysds/runtime/compress/colgroup/APreAgg.java
@@ -19,6 +19,7 @@
package org.apache.sysds.runtime.compress.colgroup;
+import org.apache.commons.lang.NotImplementedException;
import org.apache.sysds.runtime.DMLRuntimeException;
import org.apache.sysds.runtime.compress.DMLCompressionException;
import org.apache.sysds.runtime.compress.colgroup.dictionary.IDictionary;
@@ -84,9 +85,11 @@ public abstract class APreAgg extends AColGroupValue {
* @return A aggregate dictionary
*/
public final IDictionary preAggregateThatIndexStructure(APreAgg that) {
- int outputLength = that._colIndexes.size() *
this.getNumValues();
+ long outputLength = (long)that._colIndexes.size() *
this.getNumValues();
+ if(outputLength > Integer.MAX_VALUE)
+ throw new NotImplementedException("Not supported pre
aggregate of above integer length");
// create empty Dictionary that we slowly fill, hence the
dictionary is empty and no check
- final Dictionary ret = Dictionary.createNoCheck(new
double[outputLength]);
+ final Dictionary ret = Dictionary.createNoCheck(new
double[(int)outputLength]);
if(that instanceof ColGroupDDC)
preAggregateThatDDCStructure((ColGroupDDC) that, ret);
@@ -224,7 +227,8 @@ public abstract class APreAgg extends AColGroupValue {
}
private void leftMultByUncompressedColGroup(ColGroupUncompressed lhs,
MatrixBlock result) {
- LOG.warn("Transpose of uncompressed to fit to template need
t(a) %*% b");
+ if(lhs.getNumCols() != 1)
+ LOG.warn("Transpose of uncompressed to fit to template
need t(a) %*% b");
final MatrixBlock tmp = LibMatrixReorg.transpose(lhs.getData(),
InfrastructureAnalyzer.getLocalParallelism());
final int numVals = getNumValues();
final MatrixBlock preAgg = new MatrixBlock(tmp.getNumRows(),
numVals, false);
(systemds) branch main updated: [MINOR] Remove potential for compression Scalars
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 9426792b00 [MINOR] Remove potential for compression Scalars
9426792b00 is described below
commit 9426792b009b638667a8415c58552945e1be3d1b
Author: Sebastian Baunsgaard
AuthorDate: Mon Oct 30 12:22:00 2023 +0100
[MINOR] Remove potential for compression Scalars
---
.../java/org/apache/sysds/hops/rewrite/RewriteCompressedReblock.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/src/main/java/org/apache/sysds/hops/rewrite/RewriteCompressedReblock.java
b/src/main/java/org/apache/sysds/hops/rewrite/RewriteCompressedReblock.java
index 8dd323dd44..ec917b0145 100644
--- a/src/main/java/org/apache/sysds/hops/rewrite/RewriteCompressedReblock.java
+++ b/src/main/java/org/apache/sysds/hops/rewrite/RewriteCompressedReblock.java
@@ -156,7 +156,7 @@ public class RewriteCompressedReblock extends
StatementBlockRewriteRule {
public static boolean satisfiesCompressionCondition(Hop hop) {
boolean satisfies = false;
if(satisfiesSizeConstraintsForCompression(hop)){
- satisfies |= HopRewriteUtils.isData(hop,
OpOpData.PERSISTENTREAD);
+ satisfies |= HopRewriteUtils.isData(hop,
OpOpData.PERSISTENTREAD) && !hop.isScalar();
satisfies |= HopRewriteUtils.isTransformEncode(hop);
}
return satisfies;
@@ -171,7 +171,7 @@ public class RewriteCompressedReblock extends
StatementBlockRewriteRule {
satisfies |= HopRewriteUtils.isTernary(hop,
OpOp3.CTABLE)
&& hop.getInput(0).getDataType().isMatrix()
&& hop.getInput(1).getDataType().isMatrix();
- satisfies |= HopRewriteUtils.isData(hop,
OpOpData.PERSISTENTREAD);
+ satisfies |= HopRewriteUtils.isData(hop,
OpOpData.PERSISTENTREAD) && !hop.isScalar();
satisfies |= HopRewriteUtils.isUnary(hop, OpOp1.ROUND,
OpOp1.FLOOR, OpOp1.NOT, OpOp1.CEIL);
satisfies |= HopRewriteUtils.isBinary(hop, OpOp2.EQUAL,
OpOp2.NOTEQUAL, OpOp2.LESS,
OpOp2.LESSEQUAL, OpOp2.GREATER,
OpOp2.GREATEREQUAL, OpOp2.AND, OpOp2.OR, OpOp2.MODULUS);
[systemds] branch main updated (7561f61a14 -> bc277e546d)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git from 7561f61a14 [SYSTEMDS-3640] Hash Column add bc277e546d [SYSTEMDS-3637] Manifest jar with ClassPath No new revisions were added by this update. Summary of changes: pom.xml | 18 +++--- 1 file changed, 15 insertions(+), 3 deletions(-)
[systemds] branch main updated: [SYSTEMDS-3640] Hash Column
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 7561f61a14 [SYSTEMDS-3640] Hash Column
7561f61a14 is described below
commit 7561f61a14dc1097e3bfcfee497a90451b4564f1
Author: Sebastian Baunsgaard
AuthorDate: Wed Oct 25 10:38:02 2023 +0200
[SYSTEMDS-3640] Hash Column
This commit adds a new value type HASH64 for that can contain hashes
of 16 hex encoded characters. It behaves internally as if it is a string
column, but allocate a single long value per cell.
This reduce the allocation of columns with hash values from 40+ byte per
value to 8 byte.
Closes #1933
---
src/main/java/org/apache/sysds/common/Types.java | 17 +-
.../sysds/runtime/compress/colgroup/APreAgg.java | 2 +-
.../sysds/runtime/compress/lib/CLALibScalar.java | 2 +-
.../sysds/runtime/frame/data/columns/Array.java| 11 ++
.../runtime/frame/data/columns/ArrayFactory.java | 33 +++-
.../runtime/frame/data/columns/BitSetArray.java| 8 +
.../runtime/frame/data/columns/BooleanArray.java | 8 +
.../runtime/frame/data/columns/CharArray.java | 8 +
.../sysds/runtime/frame/data/columns/DDCArray.java | 5 +
.../runtime/frame/data/columns/DoubleArray.java| 11 ++
.../runtime/frame/data/columns/FloatArray.java | 8 +
.../columns/{LongArray.java => HashLongArray.java} | 213 +
.../runtime/frame/data/columns/IntegerArray.java | 8 +
.../runtime/frame/data/columns/LongArray.java | 5 +
.../runtime/frame/data/columns/OptionalArray.java | 17 ++
.../runtime/frame/data/columns/RaggedArray.java| 5 +
.../runtime/frame/data/columns/StringArray.java| 31 ++-
.../frame/data/lib/FrameLibApplySchema.java| 1 +
.../sysds/runtime/frame/data/lib/FrameUtil.java| 20 +-
.../apache/sysds/runtime/util/UtilFunctions.java | 16 +-
src/test/java/org/apache/sysds/test/TestUtils.java | 1 +
.../component/frame/array/CustomArrayTests.java| 55 +-
.../frame/array/FrameArrayConstantTests.java | 2 +
.../component/frame/array/FrameArrayTests.java | 159 +--
.../component/frame/iterators/IteratorTest.java| 37 ++--
25 files changed, 549 insertions(+), 134 deletions(-)
diff --git a/src/main/java/org/apache/sysds/common/Types.java
b/src/main/java/org/apache/sysds/common/Types.java
index 4b8f1c3a00..84019e8078 100644
--- a/src/main/java/org/apache/sysds/common/Types.java
+++ b/src/main/java/org/apache/sysds/common/Types.java
@@ -77,17 +77,21 @@ public class Types
public enum ValueType {
UINT4, UINT8, // Used for parsing in UINT values from numpy.
FP32, FP64, INT32, INT64, BOOLEAN, STRING, UNKNOWN,
+ HASH64, // Indicate that the value is a hash of 64 bit.
CHARACTER;
public boolean isNumeric() {
return this == UINT8 || this == INT32 || this == INT64
|| this == FP32 || this == FP64 || this== UINT4;
}
+
public boolean isUnknown() {
return this == UNKNOWN;
}
+
public boolean isPseudoNumeric() {
return isNumeric() || this == BOOLEAN || this ==
CHARACTER;
}
+
public String toExternalString() {
switch(this) {
case FP32:
@@ -100,10 +104,13 @@ public class Types
default: return toString();
}
}
+
public static ValueType fromExternalString(String value) {
//for now we support both internal and external strings
//until we have completely changed the external types
- String lValue = (value != null) ? value.toUpperCase() :
null;
+ if(value == null)
+ throw new DMLRuntimeException("Unknown null
value type");
+ final String lValue = value.toUpperCase();
switch(lValue) {
case "FP32": return FP32;
case "FP64":
@@ -117,6 +124,7 @@ public class Types
case "STRING": return STRING;
case "CHARACTER": return CHARACTER;
case "UNKNOWN": return UNKNOWN;
+ case "HASH64": return HASH64;
default:
throw new DMLRuntimeException("Unknown
value type: "+value);
[systemds] branch main updated: [MINOR] DML Startup
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 351828d618 [MINOR] DML Startup
351828d618 is described below
commit 351828d6184c234e5ffa10279ad7c370834b59e5
Author: Sebastian Baunsgaard
AuthorDate: Thu Oct 19 13:37:26 2023 +0200
[MINOR] DML Startup
At startup the first thing we do is to call Hadoop to parse the Hadoop
specific arguments. This takes ~ 200 ms at startup before we start our
timing of SystemDS.
The script: 'print("Hello, World!")'
Before the change it ran 1,6187 sec on my laptop and 1.6764 on a scale
out cluster node. With this commit change, it speeds up to: 1,4366 on
the laptop and 1.519 on a scale out cluster node.
Closes #1926
---
src/main/java/org/apache/sysds/api/DMLScript.java| 12
.../java/org/apache/sysds/test/AutomatedTestBase.java| 16 ++--
2 files changed, 10 insertions(+), 18 deletions(-)
diff --git a/src/main/java/org/apache/sysds/api/DMLScript.java
b/src/main/java/org/apache/sysds/api/DMLScript.java
index bf638dfcf7..aa680a97f3 100644
--- a/src/main/java/org/apache/sysds/api/DMLScript.java
+++ b/src/main/java/org/apache/sysds/api/DMLScript.java
@@ -41,10 +41,8 @@ import org.apache.commons.cli.HelpFormatter;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.sysds.common.Types.ExecMode;
import org.apache.sysds.conf.CompilerConfig;
import org.apache.sysds.conf.ConfigurationManager;
@@ -204,16 +202,15 @@ public class DMLScript
public static void main(String[] args)
{
try{
- Configuration conf = new
Configuration(ConfigurationManager.getCachedJobConf());
- String[] otherArgs = new GenericOptionsParser(conf,
args).getRemainingArgs();
- DMLScript.executeScript(conf, otherArgs);
+ DMLScript.executeScript(args);
} catch(Exception e){
- errorPrint(e);
for(String s: args){
if(s.trim().contains("-debug")){
e.printStackTrace();
+ return;
}
}
+ errorPrint(e);
}
}
@@ -221,12 +218,11 @@ public class DMLScript
* Single entry point for all public invocation alternatives (e.g.,
* main, executeScript, JaqlUdf etc)
*
-* @param conf Hadoop configuration
* @param args arguments
* @return true if success, false otherwise
* @throws IOException If an internal IOException happens.
*/
- public static boolean executeScript( Configuration conf, String[] args )
+ public static boolean executeScript( String[] args )
throws IOException, ParseException, DMLScriptException
{
//parse arguments and set execution properties
diff --git a/src/test/java/org/apache/sysds/test/AutomatedTestBase.java
b/src/test/java/org/apache/sysds/test/AutomatedTestBase.java
index 354fa12feb..f63fbb987a 100644
--- a/src/test/java/org/apache/sysds/test/AutomatedTestBase.java
+++ b/src/test/java/org/apache/sysds/test/AutomatedTestBase.java
@@ -19,6 +19,11 @@
package org.apache.sysds.test;
+import static java.lang.Math.ceil;
+import static java.lang.Thread.sleep;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
@@ -38,18 +43,12 @@ import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
-import static java.lang.Math.ceil;
-import static java.lang.Thread.sleep;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.SparkSession.Builder;
import org.apache.sysds.api.DMLScript;
@@ -59,7 +58,6 @@ import org.apache.sysds.common.Types.ExecMode;
import org.apache.sysds.common.Typ
[systemds] branch main updated: [MINOR] CSV frame reader refine csv parsing
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
The following commit(s) were added to refs/heads/main by this push:
new 1697c7ba16 [MINOR] CSV frame reader refine csv parsing
1697c7ba16 is described below
commit 1697c7ba16792831de05c2d6ae08e3aeb2f38ff3
Author: Sebastian Baunsgaard
AuthorDate: Tue Oct 24 17:53:23 2023 +0200
[MINOR] CSV frame reader refine csv parsing
This commit adds a few shortcuts in the CSV parsing to:
1. reduce call time of trim by filtering strings not containing whitespace
this is a trade off, that makes it slower for strings with whitespace, and
faster for the common case of no white spaces.
2. Specialize the split CSV to a case with a single char delimiter,
this simplify the splitting logic. But only implemented for the case of
no quotation marks in the line input, since quotations make the rules
change for csv parsing.
Closes 1932
---
.../sysds/runtime/io/FrameReaderTextCSV.java | 59 +++---
.../apache/sysds/runtime/io/IOUtilFunctions.java | 121 -
.../runtime/util/FastBufferedDataOutputStream.java | 2 +-
3 files changed, 140 insertions(+), 42 deletions(-)
diff --git a/src/main/java/org/apache/sysds/runtime/io/FrameReaderTextCSV.java
b/src/main/java/org/apache/sysds/runtime/io/FrameReaderTextCSV.java
index d8de58f058..cfe4a5e45b 100644
--- a/src/main/java/org/apache/sysds/runtime/io/FrameReaderTextCSV.java
+++ b/src/main/java/org/apache/sysds/runtime/io/FrameReaderTextCSV.java
@@ -144,9 +144,8 @@ public class FrameReaderTextCSV extends FrameReader {
String[] parts = null; // cache array for line reading.
while(reader.next(key, value)) // foreach line
{
- String cellStr = value.toString();
boolean emptyValuesFound = false;
- cellStr = IOUtilFunctions.trim(cellStr);
+ String cellStr =
IOUtilFunctions.trim(value.toString());
parts = IOUtilFunctions.splitCSV(cellStr,
delim, parts);
// sanity checks for empty values and number of
columns
@@ -154,13 +153,12 @@ public class FrameReaderTextCSV extends FrameReader {
final boolean mtdx =
parts[0].equals(TfUtils.TXMTD_NDPREFIX);
// parse frame meta data (missing values / num
distinct)
if(mtdP || mtdx) {
- parts =
IOUtilFunctions.splitCSV(cellStr, delim);
if(parts.length != dest.getNumColumns()
+ 1){
LOG.warn("Invalid metadata ");
parts = null;
continue;
}
- if(mtdP)
+ else if(mtdP)
for(int j = 0; j <
dest.getNumColumns(); j++)
dest.getColumnMetadata(j).setMvValue(parts[j + 1]);
else if(mtdx)
@@ -169,17 +167,8 @@ public class FrameReaderTextCSV extends FrameReader {
parts = null;
continue;
}
-
- for(int col = 0; col < nCol; col++) {
- String part =
IOUtilFunctions.trim(parts[col]);
- if(part.isEmpty() || (naValues != null
&& naValues.contains(part))) {
- if(isFill && dfillValue != 0)
- dest.set(row, col,
sfillValue);
- emptyValuesFound = true;
- }
- else
- dest.set(row, col, part);
- }
+ assignColumns(row, nCol, dest, parts, naValues,
isFill, dfillValue, sfillValue);
+
IOUtilFunctions.checkAndRaiseErrorCSVEmptyField(cellStr, isFill,
emptyValuesFound);
IOUtilFunctions.checkAndRaiseErrorCSVNumColumns("", cellStr, parts, clen);
row++;
@@ -195,6 +184,46 @@ public class FrameReaderTextCSV extends FrameReader {
return row;
}
+ private boolean assign
[systemds-website] branch main updated (78406f94 -> 9bb1832b)
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/systemds-website.git from 78406f94 Bump socket.io-parser from 4.2.1 to 4.2.3 (#128) add 9bb1832b [MINOR] Add Hadoop native resource No new revisions were added by this update. Summary of changes: _src/assets/datasets/hadoop/native-3.3.4.zip | Bin 0 -> 52742946 bytes 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 _src/assets/datasets/hadoop/native-3.3.4.zip
[systemds] 02/02: [MINOR] Python generate API
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
commit 113aecc7a6bb8b59e5fffa296bf60dae686a2e43
Author: Sebastian Baunsgaard
AuthorDate: Thu Oct 19 12:06:25 2023 +0200
[MINOR] Python generate API
This commit generates the Python API and fixes an edge case
where there are no returns in the method, such as differenceStatistics.
This method now returns an operation node that can be used just like
a print statements operation node.
---
src/main/python/create_python_dist.py | 6 +--
.../source/code/guide/algorithms/FullScript.py | 2 +-
.../docs/source/code/guide/end_to_end/part1.py | 2 +-
.../python/docs/source/guide/algorithms_basics.rst | 4 +-
src/main/python/generator/dml_parser.py| 28 +-
src/main/python/generator/generator.py | 14 -
.../python/systemds/operator/algorithm/__init__.py | 20 ++-
.../operator/algorithm/builtin/csplineCG.py| 2 +-
.../systemds/operator/algorithm/builtin/dbscan.py | 16 +++---
.../{scaleMinMax.py => differenceStatistics.py}| 22
...malizeApply.py => img_brightness_linearized.py} | 27 +-
.../{scaleMinMax.py => img_crop_linearized.py} | 28 +++---
...{lmPredictStats.py => img_cutout_linearized.py} | 35 +++-
.../{scaleMinMax.py => img_invert_linearized.py} | 18 ---
...{lmPredictStats.py => img_mirror_linearized.py} | 30 ++-
...{scaleMinMax.py => img_posterize_linearized.py} | 20 ---
.../algorithm/builtin/img_transform_linearized.py | 62 ++
.../algorithm/builtin/img_translate_linearized.py | 60 +
.../systemds/operator/algorithm/builtin/lm.py | 7 +--
.../systemds/operator/algorithm/builtin/lmCG.py| 4 +-
.../systemds/operator/algorithm/builtin/lmDS.py| 4 +-
.../operator/algorithm/builtin/lmPredictStats.py | 8 +--
.../algorithm/builtin/multiLogRegPredict.py| 3 +-
.../operator/algorithm/builtin/normalizeApply.py | 4 +-
.../operator/algorithm/builtin/scaleMinMax.py | 2 +
.../python/tests/algorithms/test_multiLogReg.py| 2 +-
.../python/tests/examples/tutorials/test_adult.py | 2 +-
.../python/tests/examples/tutorials/test_mnist.py | 4 +-
.../python/tests/federated/test_federated_mnist.py | 2 +-
.../tests/manual_tests/multi_log_reg_mnist.py | 2 +-
30 files changed, 310 insertions(+), 130 deletions(-)
diff --git a/src/main/python/create_python_dist.py
b/src/main/python/create_python_dist.py
index 4718881a36..f02578fa3a 100755
--- a/src/main/python/create_python_dist.py
+++ b/src/main/python/create_python_dist.py
@@ -23,6 +23,6 @@
import subprocess
f = open("generator.log","w")
-subprocess.run("python generator/generator.py",shell=True, check=True, stdout
=f, stderr=f)
-subprocess.run("python pre_setup.py",shell=True, check=True)
-subprocess.run("python setup.py sdist bdist_wheel",shell=True, check=True)
+subprocess.run("python3 generator/generator.py",shell=True, check=True, stdout
=f, stderr=f)
+subprocess.run("python3 pre_setup.py",shell=True, check=True)
+subprocess.run("python3 setup.py sdist bdist_wheel",shell=True, check=True)
diff --git a/src/main/python/docs/source/code/guide/algorithms/FullScript.py
b/src/main/python/docs/source/code/guide/algorithms/FullScript.py
index 0340886175..e8cd82cc1f 100644
--- a/src/main/python/docs/source/code/guide/algorithms/FullScript.py
+++ b/src/main/python/docs/source/code/guide/algorithms/FullScript.py
@@ -39,6 +39,6 @@ with SystemDSContext() as sds:
# Test data
Xt_ds = sds.from_numpy(Xt)
Yt_ds = sds.from_numpy(Yt) + 1.0
-[m, y_pred, acc] = multiLogRegPredict(Xt_ds, bias, Yt_ds,
verbose=False).compute()
+[m, y_pred, acc] = multiLogRegPredict(Xt_ds, bias, Y=Yt_ds,
verbose=False).compute()
logging.info(acc)
diff --git a/src/main/python/docs/source/code/guide/end_to_end/part1.py
b/src/main/python/docs/source/code/guide/end_to_end/part1.py
index 4b45679049..55ce7eca13 100644
--- a/src/main/python/docs/source/code/guide/end_to_end/part1.py
+++ b/src/main/python/docs/source/code/guide/end_to_end/part1.py
@@ -54,7 +54,7 @@ with SystemDSContext() as sds:
betas = multiLogReg(X, Y, verbose=False)
# Apply model
-[_, y_pred, acc] = multiLogRegPredict(Xt, betas, Yt)
+[_, y_pred, acc] = multiLogRegPredict(Xt, betas, Y=Yt)
# Confusion Matrix
confusion_matrix_abs, _ = confusionMatrix(y_pred, Yt).compute()
diff --git a/src/main/python/docs/source/guide/algorithms_basics.rst
b/src/main/python/docs/source/guide/algorithms_basics.rst
index 6c25b8b39d..7206605222 100644
--- a/src/main/python/docs/source/guide/algorithms_basics.rst
+++ b/src/main/python/docs/source/guide/algorithms_basics.
[systemds] branch main updated (4fa8b122ed -> 113aecc7a6)
This is an automated email from the ASF dual-hosted git repository.
baunsgaard pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/systemds.git
from 4fa8b122ed [SYSTEMDS-3153] Fix KNN
new 23177b7779 [MINOR] Various Builtin Algorithm Cleanups
new 113aecc7a6 [MINOR] Python generate API
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/site/builtins-reference.md| 34
scripts/builtin/confusionMatrix.dml| 2 +-
scripts/builtin/csplineCG.dml | 4 +-
scripts/builtin/dbscan.dml | 77 +++
scripts/builtin/decisionTree.dml | 2 +-
scripts/builtin/dist.dml | 2 +-
scripts/builtin/l2svmPredict.dml | 21 +-
scripts/builtin/lm.dml | 18 +-
scripts/builtin/lmCG.dml | 225 +
scripts/builtin/lmDS.dml | 156 +-
scripts/builtin/lmPredictStats.dml | 59 --
scripts/builtin/multiLogRegPredict.dml | 17 +-
scripts/builtin/normalizeApply.dml | 10 +-
scripts/builtin/scaleApply.dml | 8 +-
scripts/builtin/scaleMinMax.dml| 14 +-
.../java/org/apache/sysds/parser/DMLProgram.java | 1 +
.../sysds/parser/FunctionStatementBlock.java | 14 +-
src/main/python/create_python_dist.py | 6 +-
.../source/code/guide/algorithms/FullScript.py | 2 +-
.../docs/source/code/guide/end_to_end/part1.py | 2 +-
.../python/docs/source/guide/algorithms_basics.rst | 4 +-
src/main/python/generator/dml_parser.py| 28 +--
src/main/python/generator/generator.py | 14 +-
.../python/systemds/operator/algorithm/__init__.py | 20 +-
.../operator/algorithm/builtin/csplineCG.py| 2 +-
.../systemds/operator/algorithm/builtin/dbscan.py | 16 +-
.../{intersect.py => differenceStatistics.py} | 22 +-
..._brightness.py => img_brightness_linearized.py} | 16 +-
.../{img_crop.py => img_crop_linearized.py}| 30 +--
.../{img_cutout.py => img_cutout_linearized.py}| 26 ++-
.../{img_invert.py => img_invert_linearized.py}| 14 +-
.../{img_mirror.py => img_mirror_linearized.py}| 26 ++-
...mg_posterize.py => img_posterize_linearized.py} | 14 +-
...mg_transform.py => img_transform_linearized.py} | 40 ++--
...mg_translate.py => img_translate_linearized.py} | 33 +--
.../systemds/operator/algorithm/builtin/lm.py | 7 +-
.../systemds/operator/algorithm/builtin/lmCG.py| 4 +-
.../systemds/operator/algorithm/builtin/lmDS.py| 4 +-
.../operator/algorithm/builtin/lmPredictStats.py | 8 +-
.../algorithm/builtin/multiLogRegPredict.py| 3 +-
.../operator/algorithm/builtin/normalizeApply.py | 4 +-
.../operator/algorithm/builtin/scaleMinMax.py | 2 +
.../python/tests/algorithms/test_multiLogReg.py| 2 +-
.../python/tests/examples/tutorials/test_adult.py | 2 +-
.../python/tests/examples/tutorials/test_mnist.py | 4 +-
.../python/tests/federated/test_federated_mnist.py | 2 +-
.../tests/manual_tests/multi_log_reg_mnist.py | 2 +-
.../federated/algorithms/FederatedLmPipeline.java | 9 +-
src/test/scripts/functions/builtin/dbscan.dml | 2 +-
src/test/scripts/functions/builtin/dbscanApply.dml | 2 +-
50 files changed, 514 insertions(+), 522 deletions(-)
copy src/main/python/systemds/operator/algorithm/builtin/{intersect.py =>
differenceStatistics.py} (71%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_brightness.py =>
img_brightness_linearized.py} (72%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_crop.py =>
img_crop_linearized.py} (63%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_cutout.py =>
img_cutout_linearized.py} (70%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_invert.py =>
img_invert_linearized.py} (78%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_mirror.py =>
img_mirror_linearized.py} (56%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_posterize.py =>
img_posterize_linearized.py} (78%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_transform.py =>
img_transform_linearized.py} (60%)
copy src/main/python/systemds/operator/algorithm/builtin/{img_translate.py =>
img_translate_linearized.py} (62%)
[systemds] 01/02: [MINOR] Various Builtin Algorithm Cleanups
This is an automated email from the ASF dual-hosted git repository. baunsgaard pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/systemds.git commit 23177b7779f1868fd51c7af8f98e2ab8221b9011 Author: Sebastian Baunsgaard AuthorDate: Thu Oct 19 12:03:45 2023 +0200 [MINOR] Various Builtin Algorithm Cleanups This commit modifies our LM built-in, to no longer do a prediction and accuracy test if set to verbose. The modification removes a matrix multiplication of the bias matrix trained with the X input, reducing the overall execution time of our linear models when verbose is true. The logic of the statistics printing is now in lmPredictStats. Also modified is our normalizeApply, which now does not divide by zero in edge cases of constant columns. I also fixed the spelling in various other built-in scripts. Closes #1921 --- docs/site/builtins-reference.md| 34 scripts/builtin/confusionMatrix.dml| 2 +- scripts/builtin/csplineCG.dml | 4 +- scripts/builtin/dbscan.dml | 77 +++ scripts/builtin/decisionTree.dml | 2 +- scripts/builtin/dist.dml | 2 +- scripts/builtin/l2svmPredict.dml | 21 +- scripts/builtin/lm.dml | 18 +- scripts/builtin/lmCG.dml | 225 + scripts/builtin/lmDS.dml | 156 +- scripts/builtin/lmPredictStats.dml | 59 -- scripts/builtin/multiLogRegPredict.dml | 17 +- scripts/builtin/normalizeApply.dml | 10 +- scripts/builtin/scaleApply.dml | 8 +- scripts/builtin/scaleMinMax.dml| 14 +- .../java/org/apache/sysds/parser/DMLProgram.java | 1 + .../sysds/parser/FunctionStatementBlock.java | 14 +- .../federated/algorithms/FederatedLmPipeline.java | 9 +- src/test/scripts/functions/builtin/dbscan.dml | 2 +- src/test/scripts/functions/builtin/dbscanApply.dml | 2 +- 20 files changed, 311 insertions(+), 366 deletions(-) diff --git a/docs/site/builtins-reference.md b/docs/site/builtins-reference.md index 6977dadfd1..22b335866c 100644 --- a/docs/site/builtins-reference.md +++ b/docs/site/builtins-reference.md @@ -400,40 +400,6 @@ y = X %*% rand(rows = ncol(X), cols = 1) [predict, beta] = cvlm(X = X, y = y, k = 4) ``` - -## `DBSCAN`-Function - -The dbscan() implements the DBSCAN Clustering algorithm using Euclidian distance. - -### Usage - -```r -Y = dbscan(X = X, eps = 2.5, minPts = 5) -``` - -### Arguments - -| Name | Type| Default| Description | -| :- | :-- | :- | :-- | -| X | Matrix[Double] | required | The input Matrix to do DBSCAN on. | -| eps| Double | `0.5` | Maximum distance between two points for one to be considered reachable for the other. | -| minPts | Int | `5`| Number of points in a neighborhood for a point to be considered as a core point (includes the point itself). | - -### Returns - -| Type| Description | -| :---| :-- | -| Matrix[Integer] | The mapping of records to clusters | -| Matrix[Double] | The coordinates of all points considered part of a cluster | - -### Example - -```r -X = rand(rows=1780, cols=180, min=1, max=20) -[indices, model] = dbscan(X = X, eps = 2.5, minPts = 360) -``` - - ## `decisionTree`-Function The `decisionTree()` implements the classification tree with both scale and categorical diff --git a/scripts/builtin/confusionMatrix.dml b/scripts/builtin/confusionMatrix.dml index 652f04076e..18228d14c2 100644 --- a/scripts/builtin/confusionMatrix.dml +++ b/scripts/builtin/confusionMatrix.dml @@ -57,6 +57,6 @@ m_confusionMatrix = function(Matrix[Double] P, Matrix[Double] Y) dim = max(max(Y),max(P)) confusionSum = table(P, Y, dim, dim) - # max to avoid devision by 0, in case a colum contain no entries. + # max to avoid division by 0, in case a colum contain no entries. confusionAvg = confusionSum / max(1,colSums(confusionSum)) } diff --git a/scripts/builtin/csplineCG.dml b/scripts/builtin/csplineCG.dml index 37d557b8a1..a6e8b2077e 100644 --- a/scripts/builtin/csplineCG.dml +++ b/scripts/builtin/csplineCG.dml @@ -27,7 +27,7 @@ # monotonically increasing and there is no duplicates points in X # Y 1-column matrix of corresponding y values knots # inp_x the given input x, for which the cspline will find predicted y. -# tolTolerance (epsilon); conjugate graduent procedure terminates early if +# tolTolerance (epsilon); conjugate gradient procedure terminates early if #L2 norm of the beta-residual is less than tolerance * its initial norm # maxi Maximum number of conjugate gradient
