lachezar-n opened a new pull request, #2108: URL: https://github.com/apache/systemds/pull/2108
Here are many significant changes to the existing implementation and logic of the Cost Estimator and some basic tests to increase the coverage. State of the Cost Estimator: * Supports more CP instructions * Supports now a basic spectrum of Spark instructions * Supports inferring of variables's dimensions for several instructions (CP and Spark) * Implements lineage-like logic for getting the estimated cost of Spark instructions * Extended the Cost Estimator framework to include resource metrics not relevant for the compilation but relevant for the cost (mem/disk/network bandwidth, CPU NLFOPS) About the tests: * Not many tests were added, but some general test classes with examples created which can be extended to increase the coverage * Currently supported algorithms: L2SVM, LinReg, PCA, PNMF; Means excluded <- includes parser + many unknowns * Further tests for the enumerators should follow soon with the finalised version of the enumerators themselves *Note: it took longer than discussed but I also added a simplified logic to account for disk spill-overs reading at Spark operations to cover the potential scenarios we discussed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@systemds.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
