Hi all, Here is the second proposal for improving SINGA after the code quality proposal [1]. This discussion is about adding code coverage to Travis CI [2].
To enable code coverage, I think we need to fix some stuff first: 1. We need to make the Travis build fails if the test cases fail. Currently, the test cases fail (as can be seen in the log, for example [3]) but the Travis build does not show this problem and finish with "Done. Your build exited with 0." [4]. 2. Some unit tests fail because they run on the wrong architecture, not because there is a problem in the code. So we need to improve the test design. For example, in [5] there is an error "AssertionError: SINGA has not been compiled with CUDA enabled.". I think it is better to run CUDA tests only if the code is compiled for CUDA. This should be the same for all other compilation options and their corresponding test cases. 3. The python tests should run in python machines in Travis not in C++ machines [6]. Currently, the test cases uses C++ machines and install python (with conda) on them to run the python tests, instead of running python machines. 4. The python test for 3.6 [7] and 3.7 [8] should be done on different machines in Travis where the native python packages of the operating system (e.g. ubuntu or Mac) can be used. Currently, both tests happen on one C++ machine and conda is used instead of the python packages of the operating system. This only tests SINGA build with conda, but not with python alone. 5. We need to run the C++ tests [9]. Currently only the python tests are executed in Travis. After these five issues are fixed, we can proceed to 6. Add C++ coverage 7. Add python coverage 8. Show the coverage results 9. Improve the coverage (i.e. add more C++ and python unit tests to cover the code which is not tested) If everyone agrees, I will proceed by creating sub tasks for these issues under SINGA-485. Then we can implement them one by one. Thanks, Moaz [1] http://mail-archives.apache.org/mod_mbox/singa-dev/201909.mbox/%3CCAH%3DWUUbgvL62dZOCRJeEDA-a-JWNEyGvvK1Dn3ZL1YDvLXWktQ%40mail.gmail.com%3E [2] https://issues.apache.org/jira/projects/SINGA/issues/SINGA-485 [3] https://travis-ci.org/apache/singa/jobs/613933817#L3702 [4] https://travis-ci.org/apache/singa/jobs/613933817#L3849 [5] https://travis-ci.org/apache/singa/jobs/613933817#L2879 [6] https://github.com/apache/singa/blob/master/.travis.yml#L19 [7] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L33 [8] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L34 [9] https://github.com/apache/singa/tree/master/test/singa
