On 17/09/2020 19:51, Dumitru Ceara wrote: > On 3/27/20 7:24 PM, Mark Michelson wrote: >> Hi everyone, >> >> I've taken some time recently to look into the current CI for OVS/OVN, >> and I think there is room for improvement. Here I will outline a 4 part >> plan for improving CI for OVN. >> >> >> Part 1: Report travis build failures as replies to patch submissions. >> >> Currently, 0-day robot will respond to patches that fail to apply or >> fail to pass checkpatch.py. However, builds are handled by travis and >> reported out-of-band to the ovs-build list. By having build failures >> reported in response to the patch, it makes it much easier to correlate >> results to a patch series and does not require signing up for a separate >> mailing list. >> >> I suspect this will lead to some noise at first as some tests may need >> to be stabilized. If we are diligent about fixing such volatile tests, >> then this should reduce the noise after a while. Once we reach a point >> where the signal-to-noise ratio is acceptable, we can move on to...
+1 - I find this useful on the OVS list >> >> >> Part 2: Expand testing done by travis. >> >> Travis runs `make distcheck` now. This could be expanded to run the >> following: >> * system tests >> * ovn-kubernetes tests >> * Run tests against all supported versions of OVS >> CI could also be expanded to run some scale tests, but that likely would >> need to be done outside of travis. Any idea how long it would take to run all these? I would prefer that I ran them before I submitted a patch but if they took too long, I may not do that which would mean there would be more noise on the list. >> >> Like with part 1, this likely will be noisy as more tests are added. >> Again, though, as we fix these problems, it will lead to a more stable >> suite of tests. We can then move on to... What about coverage tests? >> > > Bumping this thread. As mentioned during the IRC meeting today [1], I > think it might be useful to spend some time in this direction. > > One of the suggestions during the meeting was to move to github actions > so that we reuse the ovn-k8s CI infrastructure. > > There was also the question whether there's any missing feature in > TravisCI that wouldn't allow us to run the same ovn-k8s tests in Travis. > One advantage of this is, I guess, the fact that the ovsrobot already > triggers Travis runs for every patch submission (result emails go to > ovs-build ML). > > Another direction, orthogonal I would say, could be to harden the > existing OVN tests, ovn*.at. A potential first step could be to > investigate the coverage results (make check-lcov) and see if there are > areas where we need more tests. > > Looking forward to hearing more thoughts on this matter :) What about badges on "README.md" to report failing/succeeding builds or coverage or something like that. > > Thanks, > Dmitru > > [1] > http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-09-17-17.20.log.html > >> >> Part 3: Correlate tests with patchwork checks >> >> Patchwork has "checks" that can be set on submissions which correspond >> to external tests. The tests we add in parts 1 and 2 can correspond to >> patchwork checks. The checks can provide an easily-referenced way to >> determine the status of tests for a given patch series. Adding checks to >> patchwork also lays the groundwork for Part 4. >> >> >> (Note: It may make sense to switch parts 2 and 3. We may want to >> establish patchwork checks prior to adding more tests) Sounds good >> >> >> Part 4: Gating >> >> If we have a stable and extensive testsuite, then we can attempt to >> enforce gating on patch submissions. In other words, patches that do not >> pass all checks cannot be pushed. This can be enforced in a few ways >> >> * Convention: We regulate ourselves. We agree as a community to only >> push changes if tests have passed, and we refuse to push patches that >> have failed CI. >> * Automation: The bot that monitors patchwork for new submissions and >> starts CI could potentially merge patches that have received ACKs and >> have passed CI. Humans would not push patches themselves. Sound a little dangerous - don't know if I want the robots taking over! >> * Push hook: When pushing patches, if patchwork CI checks have not >> passed (and if the patch has not been ACKed), then the hook would reject >> the push. >> * Pull Requests: Switching from mailing list patches to pull requests >> would allow for github to enforce gating. >> >> In any of the above cases, there will be ways for committers to override >> the system if absolutely necessary (e.g. CI false positives)>> >> >> What are your thoughts on this expansion of CI for OVN? Do you have >> ideas for other areas where it can be expanded? What are your thoughts >> on gating in particular? >> >> Thanks for your feedback, Thanks for this Mark >> Mark Michelson >> > > _______________________________________________ > dev mailing list > [email protected] > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
