ywcb00 opened a new pull request, #1863: URL: https://github.com/apache/systemds/pull/1863
Hi, This PR adds the functionality of propagating the sparsity from the federated workers to the coordinator when executing an instruction without subsequent data consolidation at the worker. To achieve this, the workers respond to a EXEC_INST request with the number of non-zeros of their respective federated partition. Thereby, we can obtain and aggregate the number of non-zeros and subsequently set the number of non-zeros of the (still federated) matrix object. In this PR, several instructions have been extended to support the before mentioned sparsity propagation. I also added a respective unit test which compares the propagated number of non-zeros to a reference. Unfortunately, I had to copy parts of the _DMLScript.java_ code into the test class because the test needs direct access to the execution context of the script execution. Additionally, I left three types of FIXME comments in the test dml script. In two of these comments, the underlying local operator (_ifelse_ and _cumsum_) returns an incorrect number of non-zeros for some reason that I haven't discovered yet. The other FIXME comment relates to an incorrect computation of instructions like _sum(X!=0)_ which is introduced by rewriting the constructed HOP. NOTE: This PR is based on #1862 . Hence, we should wait with the merge until the other PR has been resolved. Thanks for review :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@systemds.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
