The following is a summary of changes, status, and suggestions to our community CI, ci-cassandra.apache.org Please reply with questions, as well as any input on CircleCI status anyone has to offer.
This post will touch on… * Upgrade Tests * Build Times and Improvements * Pre Commit Builds * Stand-alone Pipeline Runs * Standardising our CI Build Scripts * JDK11 * Nightly Build Artefacts * CI Documentation * 4.0 QA Status ** Upgrade Tests Both in-jvm and normal upgrade tests have been added to ci-cassandra.apache.org Those in-jvm upgrade tests have been included in the pipeline builds. The normal upgrade tests currently remain stand-alone. Trunk’s version is found here https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest-upgrade/ ** Build Times and Improvements DTests have been parallelised. By default DTest jobs are divided into 64 splits now, with the exception of the Large DTest jobs which due to having far fewer tests have only 8 splits. This brings dtest runs from ~12 hours down to ~45 minutes. It brings whole pipeline builds from ~14 hours down to ~2.5 hours. Some patch (devbranch) builds have completed in 90 minutes. For more speed we can split more, but ci-cassandra currently has 36 agents (72 executors) and is now often saturated and build queues large. We can also look into Unit Tests which are only ever using one runner on both ci-cassandra and circleci. A problem here is that using more runners breaks some of the unit tests. Another possible improvement is to avoid the compiling in all the test stages, by re-using the built artefacts in the beginning of each pipeline run. ** Pre Commit Builds ci-cassandra.apache.org is less frequently used for pre-commit builds, for reasons of resource limits and access reserved to committers. More information on this can be found in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=153815764 Some trusted contributors have been given Jenkins API tokens, along with using the script found here https://the-asf.slack.com/archives/C0162JU2CKY/p1595703740008400 These tokens do rotate, and how/when still remains a little unclear. Committers can generate tokens via their Jenkins profile pages. There’s also investigation as to whether we can create jenkins accounts with build permissions for trusted contributors. Again with the resources available, especially in contrast to additional testing that will probably appear in the 4.0 beta testing phase, we are restricted to what is feasible here. Notifications for all devbranch pipeline builds now go to #cassandra-builds-patches. These report the build url, the commit SHA and message, and the patch repository. ** Stand-alone Pipeline Runs It has been raised a number of times that it would be great if users (companies) could run the complete pipeline on their own resources/cloud, from a single command line, including the setup and teardown of the jenkins platform. This would be a big win for the community with standardised testing and test reports to share, and helping to test all the possible configuration combinations possible. There is some work involved to do this but we appear to be moving in that direction anyway. For it to happen the all stage jobs inside the pipeline need to be moved, from being generated in the dsl script, to being defined in the in-tree Jenkinsfile. ** Standardising our CI Build Scripts Today we have a lot of duplication of build scripts. Those in cassandra-builds/build-scripts/ and those embedded into each of the circleci config files in-tree. I would like to suggest we move the build-scripts in-tree, and start migrating circleci to re-use the same build scripts. There are differences from how test lists are split (round-robin `split` to circleci timings based splitting) to how parallelisation works (circle’s containers vs the jenkins matrix plugin), but I suspect by focusing on the easy stuff there’s a lot that can be standardised. ** JDK11 JDK11 builds have been contributed, thanks to Shylaja. Trunk’s pipeline now builds both JDK 8 and 11 artefacts. Adding JDK11 test runs hit a hurdle with how the JDK labels are named and our tests are not friendly with directory names containing spaces. ** Nightly Build Artefacts Build artefacts of Tarballs, as well as Debian and RedHat packages, are attached to the artefact stages inside each pipeline, for both JDK8 and JDK11 builds. To download the latest successful JDK8 build of these, use the following links. Trunk: https://ci-cassandra.apache.org/job/Cassandra-trunk-artifacts/jdk=JDK%201.8%20(latest),label=cassandra/lastSuccessfulBuild/artifact/ 3.11: https://ci-cassandra.apache.org/job/Cassandra-3.11-artifacts/jdk=JDK%201.8%20(latest),label=cassandra/lastSuccessfulBuild/artifact/ 3.0: https://ci-cassandra.apache.org/job/Cassandra-3.0-artifacts/jdk=JDK%201.8%20(latest),label=cassandra/lastSuccessfulBuild/artifact/ 2.2: https://ci-cassandra.apache.org/job/Cassandra-2.2-artifacts/jdk=JDK%201.8%20(latest),label=cassandra/lastSuccessfulBuild/artifact/ Please note that these artefacts come with no guarantees at all, and should only be used for development purposes. ** CI Documentation There’s a draft page put together for our CI systems at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=153815764 It is work-in-progress, and it needs input from CircleCI users. Further discussion and validation is required before this page can come out of draft status. ** 4.0 QA Status Trunk is currently hovering around 10 failures for each pipeline run. There has been a doubling of failures in the past week. Maybe a bad commit, maybe much higher utilisation of ci-cassandra exposing more flakies. This is still an amazing improvement and effort from everyone considering we have ~18k tests.