[
https://issues.apache.org/jira/browse/SYSTEMML-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952133#comment-15952133
]
Matthias Boehm commented on SYSTEMML-1451:
------------------------------------------
1) Perftest: In general this is a good list. Just to set the expectations
right, step 1, i.e., generating data and running train/predict is already
automated - it requires a single shell script invocation and spits out a
summary of runtimes and all logs of individual runs. However, the analysis is
indeed a tedious, manual process right now. This perftest also covers only a
subset of algorithms (x datasizes x sparse/dense x binomial/multinomial x
algorithm parameters) and hence, should be extended. Additionally, we might
want to consider a broader range of data characteristics (e.g., different
numbers of features, ultra-sparse data, etc), proper repetitions, summary
statistics, and the ability to run certain scale factors without a need to
change the scripts.
2) Algorithms: In general, it would be possible to structure the algorithms
like that, given that the main program would run either train or predict in
order to allow invocation through command line interfaces like spark_submit.
One thing to be aware of, however, is that there is an N:1 relationship between
training and prediction scripts. For example, for all regression algorithms
(LinregCG, LinregDS, GLM, stepwise LinregDS), we call GLM-predict. Hence, we
would need imports of common scripts to avoid duplication.
> Automate performance testing and reporting
> ------------------------------------------
>
> Key: SYSTEMML-1451
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1451
> Project: SystemML
> Issue Type: Improvement
> Components: Infrastructure, Test
> Reporter: Nakul Jindal
> Labels: gsoc2017, mentor, performance, reporting, testing
>
> As part of a release (and in general), performance tests are run for SystemML.
> Currently, running and reporting on these performance tests are a manual
> process. There are helper scripts, but largely the process is manual.
> The aim of this GSoC 2017 project is to automate performance testing and its
> reporting.
> These are the tasks that this entails
> 1. Automate running of the performance tests, including generation of test
> data
> 2. Detect errors and report if any
> 3. Record performance benchmarking information
> 4. Automatically compare this performance to previous version to check for
> performance regressions
> 5. Automatically compare to Spark MLLib, R?, Julia?
> 6. Prepare report with all the information about failed jobs, performance
> information, perf info against other comparable projects/algorithms
> (plotted/in plain text in CSV, PDF or other common format)
> 7. Create scripts to automatically run this process on a cloud provider that
> spins up machines, runs the test, saves the reports and spins down the
> machines.
> 8. Create a web application to do this interactively without dropping down
> into a shell.
> As part of this project, the student will need to know scripting (in Bash,
> Python, etc). It may also involve changing error reporting and performance
> reporting code in SystemML.
> Rating - Medium (for the amount of work)
> Mentor - [~nakul02] (Other co-mentors will join in)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
