[
https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589478#comment-15589478
]
Niketan Pansare commented on SYSTEMML-880:
------------------------------------------
Design decisions of PR https://github.com/apache/incubator-systemml/pull/270:
1. In the initial version, I decided not to parse and walk the Python AST. The
key assumption is that the code to be pushed-down has to map to PyDML. This
will help us to identify the gaps in PyDML as well (for example: dot(H,
H.transpose()) instead of H.dot(H.transpose)). Also, cases such as sml.matrix
is not handled.
2. The inputs and outputs need to be specified. There is a tradeoff here
between simplicity of the API+Implementation and optimization scope.
- Alternative to the current implementation is that inputs+outputs are not
specified and the function to be pushed-down has access to all the variables in
the global scope. In this case, we walk down the AST and identify all the
variables on RHS and treat them as outputs and all other variables not created
as input (hand-waving lil bit here).
- Another alternative is to not require providing variable name in string
format: 'H', but this was difficult to implement due to pass-by-value and point
1.
Also, I could not find a way to enable the decorator (@parallelize) on
statement block instead of function.
import systemml as sml
from systemml import random
sml.setSparkContext(sc)
m, n = 100, 20
k = 40
V = sml.random.uniform(size=(m, n))
W = sml.random.uniform(size=(m, k))
H = sml.random.uniform(size=(k, n))
@sml.parallelize
def push_down_loop(inputs, outputs):
max_iters = 200
for i in range(0, max_iters):
H = H * dot(W.transpose(), V)/dot(W.transpose(), dot(W, H))
W = W * dot(V, H.transpose())/dot(W, dot(H, H.transpose()))
# The returned values as Matrix class
ret = push_down_loop(inputs={'H': H, 'W':W, 'V': V}, outputs=['H', 'W'])
H = ret[0].toNumPy()
W = ret[1].toNumPy()
> Push-down loop structures in Python DSL
> ---------------------------------------
>
> Key: SYSTEMML-880
> URL: https://issues.apache.org/jira/browse/SYSTEMML-880
> Project: SystemML
> Issue Type: Task
> Reporter: Niketan Pansare
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
