tobiasrieger opened a new pull request #1075: URL: https://github.com/apache/systemds/pull/1075
If a federated input Matrix is passed to the parameter server builtin instruction, the parameter server will detect it. **Data Partitioning** for now only row federated matrices are supported. The federated data partitioner extracts the information about the workers. **Setup** The setup is handled by a "User Defined Function" (UDF). To send all necessary data (model, gradient and aggregation functions) to the federated worker it was required to serialise ListObjects and parts of the program. As the parameter server uses unoptimised functions, it was further necessary to extend the program serialisation functionality of the ProgramConverter. **Training** There are two different update types and frequencies for the federated workers. Updates are possible "batch synchronised" or "asynchronous". An update can be sent each batch or epoch. Each computation of an update is handled by a UDF which is sent along with the current model. In the epoch case, the federated worker can also use the aggregation function to update its local model. **Cleanup** Most cleanup is done on the fly and making sure to clear the caches for "Cacheable Objects". After the federated worker completes its task, a "teardown UDF" is sent to restore the execution context to the state before the computations. **Testing** The included test case, tests all different supported parameter combinations on a randomly generated "dummy MNIST" dataset. **Problems** At the moment the tests only work for the Exec Mode "Single Node". To fix this the program is recompiled on call of the parameter server builtin instruction, but this process creates invalid CP°rmvar instructions when passing SPARK instructions ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
