For now, I have updated our python mllearn tests to compare the prediction of our algorithm to that of scikit-learn: https://github.com/apache/incubator-systemml/blob/master/src/main/python/tests/test_mllearn_numpy.py#L81
The test now uses scikit-learn predictions as the baseline and computes the scores (accuracy score for classifiers and r2 score for regressors). If the score is greater than 95%, the test pass. Though using this approach, we do not measure the generalization capability of our algorithm, we at least ensure that our algorithm performs no worse than scikit-learn under default setting. We can make the testing even more rigorous later. The next step would be to enable these python tests through jenkins. Thanks, Niketan Pansare IBM Almaden Research Center E-mail: npansar At us.ibm.com http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar From: Matthias Boehm <mboe...@googlemail.com> To: dev@systemml.incubator.apache.org Date: 02/17/2017 11:54 AM Subject: Re: Proposal to add 'accuracy test suite' before 1.0 release Yes, this has been discussed a couple of times now, most recently in SYSTEMML-546. It takes quite some effort though to create a sophisticated algorithm-level test suite as done for GLM. So by all means, please, go ahead and add these tests. However, I would not impose any constraints on the contribution of new algorithms in that regard, or similarly on tests with simplified algorithms because it would raise the bar to high. Regards, Matthias On 2/17/2017 10:48 AM, Niketan Pansare wrote: > > > Hi all, > > We currently test the correctness of individual runtime operators using our > integration tests but not the "released" algorithms. To be fair, we do test > a subset of "simplified" algorithms on synthetic datasets and compare the > accuracy with R. Also, we are testing subset of released algorithms using > our Python tests, but it's intended purpose is to only test the integration > of the APIs: > Simplified algorithms: > https://github.com/apache/incubator-systemml/tree/master/src/test/scripts/applications > Released algorithms: > https://github.com/apache/incubator-systemml/tree/master/scripts/algorithms > Python tests: > https://github.com/apache/incubator-systemml/tree/master/src/main/python/tests > > Though the released algorithm is tested when it is initially introduced, > other artifacts (spark versions, API changes, engine improvements, etc) > could cause them to return incorrect results over a period of time. > Therefore, similar to our performance test suite ( > https://github.com/apache/incubator-systemml/tree/master/scripts/perftest ), > I propose we create another test suite ("accuracy test suite" for lack of a > better term) that compares the accuracy (or some other metric) of our > released algorithms on standard datasets. Making it a requirement to add > tests to accuracy test suite when adding the new algorithm will greatly > improve the production-readiness of SystemML as well as serve as a usage > guide too. This implies we run both the performance as well as accuracy > test suite before our release. Alternative is to replace simplified > algorithms with our released algorithms. > > Advantages of accuracy test suite approach: > 1. No increase the running time of integration tests on Jenkins. > 2. Accuracy test suite could use much larger datasets. > 3. Accuracy test suite could include algorithms that take longer to > converge (for example: Deep Learning algorithms). > > Advantage of replacing simplified algorithms: > 1. No commit breaks any of the existing algorithms. > > Thanks, > > Niketan Pansare > IBM Almaden Research Center > E-mail: npansar At us.ibm.com > http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar >
