Baunsgaard commented on pull request #1303: URL: https://github.com/apache/systemds/pull/1303#issuecomment-857614373
> I noticed that you are transmitting the whole tree, via an integer ID, to the instruction runtime. I am not well aware of the intention, but can we not follow the conventional path: optimize the tree/DAG in the optimizer, extract information, and transmit via the instructions strings of the corresponding instructions; and/or do the same during recompilation? That way you can stick to the supposed separation of compiler and runtime, and can easily scale to distributed runtimes such as Spark and federated. Excellent question. The instruction strings then would have to be able to contain nested structures, since the workload tree provide means of separating different loops into sub tree nodes in the structure. If a string instruction containing this should be constructed it means that the string ends up with different lengths depending on how many sub loops are executed. Providing an id to the structure and storing the tree somewhere else seams more scalable and flexible to future modifications where new types of instructions are added to the WTree. Also note that currently we look up our matrix objects the same way via an id provided to the instruction string, (just with another back-end that we have worked with much much longer). The reason why I'm not using that back-end is because it require specific data types contained, like Data, MatrixValue, MatrixBlock, FrameBlock, etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
