An issue that affected us several times was our CI setup: we have a build_and_test.yml on the master branch that is used for CI for all branches, containing branch-specific logic (if branch == 'branch-3.5'). This introduces a complicated coupling between our master branch and maintenance branches.
Ideally, the master workflow file should not impact any maintenance branch CIs. The maintenance branch CIs (3.5, 4.0, 4.1 etc) should only rely on workflow files in their respective branches. When 4.1 fails, we fix things in branch-4.1 - that's the natural thing to do. It would be much easier for us to pin dependencies and logic for released branches, our CI code would be much cleaner and more manageable. The current CI is likely configured this way because scheduled GitHub Actions only run on the default branch (master in this case). We have two types of CI triggers: 1. push-triggered CIs, including our PR checks (run on forked repo), and master and branch-x commit checks. These actually run on the corresponding branches. So a commit on branch-4.1 will trigger the workflow file on branch-4.1. 2. scheduled CIs, including all of our nightly builds. These rely on the master workflow file, which is the source of evil. As you can probably tell, we not only have coupling issues, but these two types of CIs also produce inconsistent results. A post-merge CI might yield different results than the nightly CI on the same branch, because they use workflow files from different branches. My proposal is to do a workaround: for scheduled CIs, instead of using the workflow file directly from the master branch, we will call for `gh workflow run --ref "the-branch-to-test"`. This will trigger the workflow using the file from that specific branch. We should be able to remove all the branch specific logic in `build_and_test.yml` and put them in corresponding branches. Changing the workflow file on master will no longer break the nightly release branch CI. More importantly, managing dependencies outside our workflow and Dockerfiles would be much easier for us. We can potentially manage all (python) dependencies in a single place on the specific branch. I will first create a prototype for a specific branch, then gradually modify our CIs, but I'd like to share this plan and ask the community for feedback on this idea first. Tian
