Re: [scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-27 Thread Steffen Herbold
Hi Andy, I now have results for LinearDiscriminantAnalysis and the SGDClassifier. I updated the results online. The LinearDiscriminantAnalysis had * an infinity of NaN for data that approaches MAXDOUBLE and * problems with an internal array size computation for data for several tests, i.

Re: [scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-23 Thread Steffen Herbold
Hi Andy, thanks for your detailed feedback. The random states are fixed, and set immediately before calling the fit function. Here is a gist with the code for one smoke tests and a metamorphic test [1]. I will run the tests for LinearDiscriminantAnalysis and the SGDClassifier. I somehow mis

Re: [scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-22 Thread Andreas Mueller
Hi Steffen. Thanks for sharing your analysis. We really need more work in this direction. I assume you fixed the random states everywhere? I consider these tests helpful but not all your expectations are warranted depending on the model. If you add one to each feature, there is no expectati

[scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-22 Thread Steffen Herbold
Dear developers, I am writing you because I applied an approach for the automated testing of classification algorithms to scikit-learn and would like to forward the results to you. The approach is a combination of smoke testing and metamorphic testing. The smoke tests try to find problems by