All, Progress on the initiative:
* After a round of hit-and-trial and exploring options, the sonarcloud + bot is configured for apache/cloudstack repo. The quality builds are done using Github actions [1]. * Commits pushed to main branch will get static analysis and coverage statistics: https://sonarcloud.io/summary/overall?id=apache_cloudstack * All new PRs opened against main branch will get static analysis, code coverage stats as seen on some new PRs: https://sonarcloud.io/project/pull_requests_list?id=apache_cloudstack * Anyone can generate coverage/quality reports using "-P quality" passed to maven, for example: mvn -P quality -Dnoredist -Dsimulator clean install clover:aggregate clover:clover This will generate html output at client/target/site/ directory * The "@acs-robot" bot makes comments such as kicks UI build if PR has UI changes (currently pkging isn't done when java changes are detected) * The "boring-cyborg" bot adds labels to PR based on file/paths and changes introduced by the PR The sonarcloud analysis on per-PR basis helps PR authors learn more about the PR changes in an automated rule-based manner: * robustness/reliability of code (java issues such as potential NPE, memory and file leaks etc) * vulnerabilities and security hotspots (code causing security issues or can potentially lead to vulnerability) * code smell - rule-based maintenance/debt issues (for example, complex java code) * unit-test coverage percentage (of the PR and also the effective % post merging the PR) * code duplication When the built-in rules of sonarcloud for java code isn't honour it will show the PR analysis as failed (for ex. by default 80% is limit on coverage). Some of the rules/issues may be non-issues which can be marked as closed on sonarcloud or the rules used for analysis can be changed (for ex. include/excluse files, paths etc). I also welcome if any PMCs and committers want to join the sonarcloud project for this access (currently I'm the admin per https://issues.apache.org/jira/browse/INFRA-23223). For example, here's a random PR where the bot detected 69% coverage as the PR also adds unit tests: (personally I like PRs like this that add unit tests and help increase overall code coverage of the codebase) https://github.com/apache/cloudstack/pull/6358#issuecomment-1117848257 (https://sonarcloud.io/summary/new_code?id=apache_cloudstack&pullRequest=6358) All, pl share your views: * if we're happy with the default thresholds of sonarcloud, what rules may be changed * should code coverage be made a requirement (after agreeing on some min. % coverage) at least for new features if not bugfix/small changes Other things to explore and investigate: * Explore and measure coverage of smoketests * CI/CD improvements: * Build/rekick failed Travis smoketests cases/jobs * Speedup smoketests * Explore weekly run of component tests * (semi-)automated upgrade tests (may assist for release efforts) * PR/participation changes: * Assign reviewers based on sub-system/codebase experts based on file/paths * Assign PR labels by size changes * Explore triaging of Github issues (labels, milestone, assignees etc) [1] https://github.com/apache/cloudstack/tree/main/.github/workflows Regards. ________________________________ From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: Monday, April 18, 2022 16:12 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: [DISCUSS] Community CI/CD/Triage Bot All, I've some updates on the initiative: * I've added some initial support of @acs-robot which along with Github actions is used to (a) detect UI changes and kick UI QA builds, (b) detect java/xml changes and kick quality check and report coverage report on the PR * Enabled coverage/quality, example the coverage reports can be seen here: http://qa.cloudstack.cloud/coverage/ (I'll set up a daily job, on main branch) * Quality metrics/analysis re-setup with Sonar: https://sonarcloud.io/project/overview?id=apachecloudstack (will set up a daily job on main branch) To explore and investigate: (based on feedback) * Build/re-kick failed Travis jobs * Assign (latest) milestones to new issues and PRs * Detect if PR needs packaging and running smoketests, build/re-kick failed pkg jobs * Assign PR approvers based on file-path matching * Explore automatic triaging of issues (label, assign to maintainers) * Label PRs by size of the PR changes (similar to k8s, s/m/l/xl/xxl/...) * Per PR Sonar-cloud integration (not sure if this is necessary unless contributors want this) Please do share your ideas and thoughts. Thanks. Regards. ________________________________ From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: Thursday, April 7, 2022 13:37 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: [DISCUSS] Community CI/CD/Triage Bot Thanks for the feedback and suggestions Nicolas. I've been exploring the ASF infra wiki and docs to see what's already out there we can adapt and use. So far I've report progress [1][2]: * boring-cyborg: this probot based Github app/bot approved by ASF-infra and used by other projects such as Airflow. This allows automatic tagging of PRs by filepath changes [1], and can help put welcome messages and encouraging msg to a new contributor's first PR merge * merge-conflict checker: based on Nicolas's feedback I put together this Github Action workflow [2] which labels PRs with merge conflicts and comments to the PR author to address the conflict * dependabot: not enabled by me, but it appears ASF-infra is enabling this asf-repo wide (https://github.com/apache/cloudstack/pulls/app%2Fdependabot) I'll be exploring the following: * Jacoco/code coverage stats on a PR (unit test coverage for changes) * Explore automatic changes in the following to assist and reduce release management effort: * assign milestone to new PRs and issues (or maybe a triage label?) * re-kick of failed Travis tests (for ex. try 2-3 times before giving) * detect if a PR needs smoke tests or not; kick UI builds if needed * Assign reviewers/approvers on PRs based on file paths (or last commit changes?) - this may need some thinking, as most projects such as k8s keep an owner file in directories to keep track of who's best to review changes * Explore how issues can be triaged and automatically labeled * Explore other things mentioned in this thread and report findings to dev@ [1] https://github.com/apache/cloudstack/blob/main/.github/boring-cyborg.yml [2] https://github.com/apache/cloudstack/blob/main/.github/workflows/merge-conflict-checker.yml [3] https://github.com/apache/cloudstack/pulls?q=is%3Apr+is%3Aopen+label%3Astatus%3Ahas-conflicts Regards. ________________________________ From: Nicolas Vazquez <nicolas.vazq...@shapeblue.com> Sent: Wednesday, April 6, 2022 19:29 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: [DISCUSS] Community CI/CD/Triage Bot Thanks Rohit for this initiative, I think it is very useful. From the list of tasks I’m not really sure about the automatic merging of PRs – but the rest could be very useful, specially the merge conflicts detection which is a very common case near the releases freeze date. I would like to propose a few more tasks: * For new issues/PRs - assign the milestone in case it is not set * Close/reopen a PR in case of failures on the checks * Detect whether to build packages and run tests or build only UI based on the files changed/labels Regards, Nicolas Vazquez From: Rohit Yadav <rohit.ya...@shapeblue.com> Date: Tuesday, 5 April 2022 at 16:37 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: [DISCUSS] Community CI/CD/Triage Bot All, I'm trying to investigate and experiment with a few things and new ideas around our QA/automation and CI/CD. I want to build initial support around this and ask if anybody has any feedback, objections on this; Taking inspiration from kubernetes and other apache projects [2][3], I'm trying to explore the following use-cases and requirements that are initiated by a new community github bot [1]: * Triage issues and PRs: * Add labels on issues and PRs * Assign reviewers and approvers for PRs * Perform any other checks and misc things (for ex. welcome new contributors?) * Have ability to remind people on issues and PRs (for ex. /remind in 2 weeks... etc) * Kick tests, lints etc using apache approved/integrated automation hooks, Github Actions etc. * Generally, explore use of Github Actions, marketplace apps/bots or existing opensource systems such as prow * Report code coverage, stats, and other relevant/related advice And, if ASF infra permits, eventually the bot may perform the following mundane tasks which can help our RMs and contributors who end of spending a lot of time on: * Keep track of latest CI/CD reports (for ex. smoketests reports) and remove old reports/comments * Help re-kick failed jobs (builds and tests, for example try to build 3 time before giving up, or rekick failed test) or notify relevant people (maybe an external test-runner dashboard?) * Remind people when their PRs are outdated, have merge-conflicts etc. * Automated merging of PRs that pass community merge guidelines, tests; also ability for RM/contributors to put a /hold on a PR * For docs, website repos: build docs/website and publish on staging; on another command merge staging (Apache Airflow and others have some good examples of how they do it) * Discuss and automate other things to assist RMs and contributors in their day-to-day activities (take feedback for some use-cases, examples) (Note: the above things are largely beyond the scope of and additional features to the current blueorangutan bot which is currently limited to pkging+smoketests with different hypervisors and is gated due to limited lab capacities) For PoC/investigation purposes, I've created a new community github bot [1] which assuming it works and meets the goals can be moved and managed under the ACS PMC wing. I hope to have your support in the initial phase of investigation and see what PoC integrations we can enable to begin with on the apache/cloudstack, apache/cloudstack-documentation, and apache/cloudstack-www repos. [1] https://github.com/acs-robot [2] https://github.com/apache/airflow/blob/main/CI_DIAGRAMS.md [3] https://github.com/apache/airflow/pull/22753 [4] https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features [5] https://infra.apache.org/github-actions-secrets.html Regards.
