Hi everyone, I spent some time over the last few days to update our CI environment for base and ports. The changes in base are already merged, and I have a PR for you below to look at the upcoming changes for ports.
## Changes that apply to base and ports First, I've removed the Travis CI. See earlier posts in this thread on the rationale. I added macOS 11 and macOS 10.15 support using GitHub Actions. However, GitHub Actions does not have a 10.14 image, so I've also made sure the same jobs run on Azure Pipelines with a 10.14 image. This wasn't too hard as the two systems are obviously based on the same codebase with minor differences in configuration, output, formatting, etc. ## CI for macports-base For builds in base, you now get build & test execution in all pull requests and for all changes merged. I've also backported the CI changes to the release-2.6 branch, so you'll get the same experience there. Pushing new tags that start with "v" or pushing to the travis-ci branch triggers automatic deployment of the tarball we use to set up the build environment for the macports-ports CI. This was previously a manual step during the release process, but is now automated. I have not actually tested this with a new tag, so we'll have to keep an eye on this after the next release. See https://github.com/macports/macports-base/pull/220/checks for the user experience you get for base builds now. ## CI for macports-ports For the ports CI, I have an open pull request that showcases how I envision the new setup: https://github.com/macports/macports-ports/pull/9398 Please look at that and provide feedback. I grouped the various things we do during the CI build into steps with human-readable names, optimized the speed of the setup and have added extensive usage of folding groups, so that you can get a quick overview, but still dive in where needed. ### Lint Warnings lint warnings will now show up as a comment in the first line of the affected Portfile. Open the "Files Changed" view and scroll to the changed Portfiles to see them. You may get those comments multiple times, once for each OS on GitHub Actions, and once for every subport that's affected by this problem. This can lead to a bit of an explosion if you trigger a lint warning on every OS in every subport of a Portfile, but I'm hoping it won't be too much of an issue for now (we can always improve and deduplicate later). Unfortunately, while Azure Pipelines does support adding comments to files, it does not seem as if those will be posted back to GitHub, so that particular feature is 10.15 and 11 only for now. ### Build Progress & Results I also separated dependency installation and the port's installation for each subport into collapsible groups, so that you can quickly find the port that failed to build for you. Note that by default, only the progress information is shown for dependencies – I am assuming that most people will not be interested in the details if those fail. For the ports that where changed in this PR, you'll see the entire debugging output. At the end of the build (regardless of whether it failed or not), the logfiles will be uploaded as an artifact. This includes - the full build log of the dependencies - the full stdout of the port's build - the main.log for the port - the files list and statistics we also gather on the buildbot On Azure Pipelines, you can download the generated ZIP file from the build results view on the summary page by following the "1 published; 1 consumed" link in the third column titled "Related" of the first row that container the metadata of your build. For GitHub Actions, a button with access to the artifacts will show up at the top right on the "Checks" tab as soon as *all* builds have finished. As long as some builds are running, you will not be able to download the artifacts (at least I haven't found a way to do so). If you're in a hurry to get the logs, you can always cancel the workflow, and logs as far as available should show up. ### Bugs The detection of changed ports needs improvement. It occasionally picks up ports that have been merged on master since the PR was created. I believe this happens because the default on GitHub Actions is to perform a merge and then build the merged state, but I have not investigated the details yet. The command I use to list the changed files is git diff --name-only --diff-filter=AM macports/master @ where macports/master is, of course macports/macports-ports. Maybe that command needs to change and diff from $(git merge-base macports/master @), or maybe we should prevent the merge from happening and build the PR state directly, or always rebase the PR on top of macports/macports-ports before building. ## Feedback Please look at the current state of the PR, and leave feedback either here via email or directly in the pull request. I'm particularly interested in - whether you quickly found what you were looking for, - whether the output is simple to understand - whether the logging output is concise enough so you can quickly navigate, but detailed enough so you can debug your problem See also the last few builds on https://github.com/macports/macports-ports/actions?query=branch%3Acal-test-ci They may have subtle bugs that I've fixed in the latest version, but they also built different ports and showcase the artifacts as well. ## Timeline I think the current state is a net improvement over our current set of Travis CI jobs that we mostly ignore because they're slow or time out, and the currently completely broken GitHub Actions jobs, which occasionally get canceled because the macOS 11 one fails early and stops the 10.15 one. Consequently I'm inclined to merge this change rather sooner than later, i.e. probably early next week. -- Clemens
