[GitHub] [incubator-mxnet] marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds
marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds URL: https://github.com/apache/incubator-mxnet/pull/17889#issuecomment-615851885 You have to take into account that additionally to randomness in our tests, a stable coverage report requires that the base commit (in the master branch) actually had a successful run. Otherwise, the report is based on partial data. So I'd say there are two or three action items: 1. Introduce tests for flaky lines 2. Add a retry mechanism for failed CI runs on the master branch 3. Stabilize CI runs to "never". This goal is reached when the master branch CI run rate is close to 100% successful. Wrt to 3), I think we're quite far away unfortunately - so 2) is certainly the first thing that should be done:  I'd recommend to spend time on stabilizing CI as highest priority over all the other optimizations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds
marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds URL: https://github.com/apache/incubator-mxnet/pull/17889#issuecomment-615851885 You have to take into account that additionally to randomness in our tests, a stable coverage report requires that the base commit (in the master branch) actually had a successful run. Otherwise, the report is based on partial data. So I'd say there are two or three action items: 1. Introduce tests for flaky lines 2. Add a retry mechanism for failed CI runs on the master branch 3. Stabilize CI runs to "never". This goal is reached when the master branch CI run rate is close to 100% successful. Wrt to 3), I think we're quite far away unfortunately - so 2) is certainly the first thing that should be done:  I'd recommend to spend time on stabilizing CI as highest priority over all the other cost optimizations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds
marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds URL: https://github.com/apache/incubator-mxnet/pull/17889#issuecomment-615850612 Actually the coverage is not stable at all. Look at this example report https://codecov.io/gh/apache/incubator-mxnet/pull/17905/changes which is based on #17905. The PR adds one new operator, but 132 files have coverage changes with a reported line changes in a four digit space. That example is not ideal (although the lines changed have a coverage of >90%, so the author did a good job of adding test coverage for their new code), but I couldn't find a recent PR which had a successful CI run and was a NO-OP. Maybe just create a few empty PRs, make the CI runs succeed and then you will have a few clear reports which show you exactly which lines are not stable. Generally we can only consider the coverage report stable if that page reports 0 changes for runs which do not change anything. Otherwise, these numbers are misleading and will confuse the PR authors if we enable the publishing.  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-mxnet] marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds
marcoabreu edited a comment on issue #17889: Re-enable coverage on CI for GCC CPU builds URL: https://github.com/apache/incubator-mxnet/pull/17889#issuecomment-604192321 Sure, please feel free to make a poc once publishing is enabled again and then we can evaluate if the data is helpful or not. Generally I'm a big fan of test coverage diff - hence the motivation to introduce the publish in the first place This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
