[scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-22 Thread Steffen Herbold
Dear developers, I am writing you because I applied an approach for the automated testing of classification algorithms to scikit-learn and would like to forward the results to you. The approach is a combination of smoke testing and metamorphic testing. The smoke tests try to find problems by

Re: [scikit-learn] Smoke and Metamorphic Testing of scikit-learn

2018-08-22 Thread Andreas Mueller
Hi Steffen. Thanks for sharing your analysis. We really need more work in this direction. I assume you fixed the random states everywhere? I consider these tests helpful but not all your expectations are warranted depending on the model. If you add one to each feature, there is no expectati