Re: Cassandra CI Status – 2020-08-08
This is awesome! Thanks Mick for all the great work. ~Charles On Mon, Aug 10, 2020 at 11:41 AM Ekaterina Dimitrova wrote: > > Thank you Mick for all the work you are doing for the C* CI and not only! > I am especially excited about the addition of the upgrade tests. > It’s also great that things got documented. > > Best regards, > Ekaterina > > On Sat, 8 Aug 2020 at 6:41, Mick Semb Wever wrote: > > > The following is a summary of changes, status, and suggestions to our > > community CI, ci-cassandra.apache.org > > Please reply with questions, as well as any input on CircleCI status anyone > > has to offer. > > > > This post will touch on… > > * Upgrade Tests > > * Build Times and Improvements > > * Pre Commit Builds > > * Stand-alone Pipeline Runs > > * Standardising our CI Build Scripts > > * JDK11 > > * Nightly Build Artefacts > > * CI Documentation > > * 4.0 QA Status > > > > > > ** Upgrade Tests > > > > Both in-jvm and normal upgrade tests have been added to > > ci-cassandra.apache.org > > > > Those in-jvm upgrade tests have been included in the pipeline builds. The > > normal upgrade tests currently remain stand-alone. Trunk’s version is found > > here https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest-upgrade/ > > > > > > ** Build Times and Improvements > > > > DTests have been parallelised. By default DTest jobs are divided into 64 > > splits now, with the exception of the Large DTest jobs which due to having > > far fewer tests have only 8 splits. > > > > This brings dtest runs from ~12 hours down to ~45 minutes. It brings whole > > pipeline builds from ~14 hours down to ~2.5 hours. Some patch (devbranch) > > builds have completed in 90 minutes. > > > > For more speed we can split more, but ci-cassandra currently has 36 agents > > (72 executors) and is now often saturated and build queues large. We can > > also look into Unit Tests which are only ever using one runner on both > > ci-cassandra and circleci. A problem here is that using more runners breaks > > some of the unit tests. Another possible improvement is to avoid the > > compiling in all the test stages, by re-using the built artefacts in the > > beginning of each pipeline run. > > > > > > ** Pre Commit Builds > > > > ci-cassandra.apache.org is less frequently used for pre-commit builds, for > > reasons of resource limits and access reserved to committers. More > > information on this can be found in > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=153815764 > > > > Some trusted contributors have been given Jenkins API tokens, along with > > using > > the script found here > > https://the-asf.slack.com/archives/C0162JU2CKY/p1595703740008400 > > > > These tokens do rotate, and how/when still remains a little unclear. > > Committers can generate tokens via their Jenkins profile pages. There’s > > also investigation as to whether we can create jenkins accounts with build > > permissions for trusted contributors. Again with the resources available, > > especially in contrast to additional testing that will probably appear in > > the 4.0 beta testing phase, we are restricted to what is feasible here. > > > > Notifications for all devbranch pipeline builds now go to > > #cassandra-builds-patches. These report the build url, the commit SHA and > > message, and the patch repository. > > > > > > ** Stand-alone Pipeline Runs > > > > It has been raised a number of times that it would be great if users > > (companies) could run the complete pipeline on their own resources/cloud, > > from a single command line, including the setup and teardown of the jenkins > > platform. This would be a big win for the community with standardised > > testing and test reports to share, and helping to test all the possible > > configuration combinations possible. There is some work involved to do this > > but we appear to be moving in that direction anyway. For it to happen the > > all stage jobs inside the pipeline need to be moved, from being generated > > in the dsl script, to being defined in the in-tree Jenkinsfile. > > > > > > ** Standardising our CI Build Scripts > > > > Today we have a lot of duplication of build scripts. Those in > > cassandra-builds/build-scripts/ and those embedded into each of the > > circleci config files in-tree. > > > > I would like to suggest we move the build-scripts in-tree, and start > > migrating circleci to re-use the same build scripts. There are differences > > from how test lists are split (round-robin `split` to circleci timings > > based splitting) to how parallelisation works (circle’s containers vs the > > jenkins matrix plugin), but I suspect by focusing on the easy stuff there’s > > a lot that can be standardised. > > > > > > ** JDK11 > > > > JDK11 builds have been contributed, thanks to Shylaja. Trunk’s pipeline now > > builds both JDK 8 and 11 artefacts. > > > > Adding JDK11 test runs hit a hurdle with how the JDK labels are named and > > our tests are
Re: Cassandra CI Status – 2020-08-08
Thank you Mick for all the work you are doing for the C* CI and not only! I am especially excited about the addition of the upgrade tests. It’s also great that things got documented. Best regards, Ekaterina On Sat, 8 Aug 2020 at 6:41, Mick Semb Wever wrote: > The following is a summary of changes, status, and suggestions to our > community CI, ci-cassandra.apache.org > Please reply with questions, as well as any input on CircleCI status anyone > has to offer. > > This post will touch on… > * Upgrade Tests > * Build Times and Improvements > * Pre Commit Builds > * Stand-alone Pipeline Runs > * Standardising our CI Build Scripts > * JDK11 > * Nightly Build Artefacts > * CI Documentation > * 4.0 QA Status > > > ** Upgrade Tests > > Both in-jvm and normal upgrade tests have been added to > ci-cassandra.apache.org > > Those in-jvm upgrade tests have been included in the pipeline builds. The > normal upgrade tests currently remain stand-alone. Trunk’s version is found > here https://ci-cassandra.apache.org/job/Cassandra-trunk-dtest-upgrade/ > > > ** Build Times and Improvements > > DTests have been parallelised. By default DTest jobs are divided into 64 > splits now, with the exception of the Large DTest jobs which due to having > far fewer tests have only 8 splits. > > This brings dtest runs from ~12 hours down to ~45 minutes. It brings whole > pipeline builds from ~14 hours down to ~2.5 hours. Some patch (devbranch) > builds have completed in 90 minutes. > > For more speed we can split more, but ci-cassandra currently has 36 agents > (72 executors) and is now often saturated and build queues large. We can > also look into Unit Tests which are only ever using one runner on both > ci-cassandra and circleci. A problem here is that using more runners breaks > some of the unit tests. Another possible improvement is to avoid the > compiling in all the test stages, by re-using the built artefacts in the > beginning of each pipeline run. > > > ** Pre Commit Builds > > ci-cassandra.apache.org is less frequently used for pre-commit builds, for > reasons of resource limits and access reserved to committers. More > information on this can be found in > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=153815764 > > Some trusted contributors have been given Jenkins API tokens, along with > using > the script found here > https://the-asf.slack.com/archives/C0162JU2CKY/p1595703740008400 > > These tokens do rotate, and how/when still remains a little unclear. > Committers can generate tokens via their Jenkins profile pages. There’s > also investigation as to whether we can create jenkins accounts with build > permissions for trusted contributors. Again with the resources available, > especially in contrast to additional testing that will probably appear in > the 4.0 beta testing phase, we are restricted to what is feasible here. > > Notifications for all devbranch pipeline builds now go to > #cassandra-builds-patches. These report the build url, the commit SHA and > message, and the patch repository. > > > ** Stand-alone Pipeline Runs > > It has been raised a number of times that it would be great if users > (companies) could run the complete pipeline on their own resources/cloud, > from a single command line, including the setup and teardown of the jenkins > platform. This would be a big win for the community with standardised > testing and test reports to share, and helping to test all the possible > configuration combinations possible. There is some work involved to do this > but we appear to be moving in that direction anyway. For it to happen the > all stage jobs inside the pipeline need to be moved, from being generated > in the dsl script, to being defined in the in-tree Jenkinsfile. > > > ** Standardising our CI Build Scripts > > Today we have a lot of duplication of build scripts. Those in > cassandra-builds/build-scripts/ and those embedded into each of the > circleci config files in-tree. > > I would like to suggest we move the build-scripts in-tree, and start > migrating circleci to re-use the same build scripts. There are differences > from how test lists are split (round-robin `split` to circleci timings > based splitting) to how parallelisation works (circle’s containers vs the > jenkins matrix plugin), but I suspect by focusing on the easy stuff there’s > a lot that can be standardised. > > > ** JDK11 > > JDK11 builds have been contributed, thanks to Shylaja. Trunk’s pipeline now > builds both JDK 8 and 11 artefacts. > > Adding JDK11 test runs hit a hurdle with how the JDK labels are named and > our tests are not friendly with directory names containing spaces. > > > ** Nightly Build Artefacts > > Build artefacts of Tarballs, as well as Debian and RedHat packages, are > attached to the artefact stages inside each pipeline, for both JDK8 and > JDK11 builds. > > To download the latest successful JDK8 build of these, use the following > links. > Trunk: > >
Re: [Vote] Remove Windows support from 4.0+
There's some significant performance differences between bare metal linux and WAL 2 as per phoronix' recent testing: https://www.phoronix.com/scan.php?page=article=windows10-may2020-wsl2=2; I'd be wary of recommending it for *production* workloads but it certainly seems to work fine for dev. (context for those that don't know: I did the initial full windows support work back in 2014-2015) - my impression is that it's not a ton of work to keep things up to date / working with windows fwiw, but we as a community don't have anyone committed to maintaining support in 4.0 for Windows. It seems our general impression as a dev community is that there's minimal appetite for production usage of C* on Windows but we're lacking empirical evidence for or against. reached out with an explicit request for donation of support, making it > explicit that we will be removing support if none is forthcoming Seems like a reasonable enough approach to me. On Mon, Aug 10, 2020 at 1:02 PM Benedict Elliott Smith wrote: > I think it would seem a bit more legitimate to remove support without a > normal deprecation cycle if we've reached out with an explicit request for > donation of support, making it explicit that we will be removing support if > none is forthcoming. But just my 2c; like most people on this list, even > discussing Windows support barely registers on my priority list. > > On 10/08/2020, 17:53, "Jordan West" wrote: > > It wasn't directly regarding removing support but we did reach out to > cassandra-users@ for testing 4.0 on Windows and got no response: > https://www.mail-archive.com/user@cassandra.apache.org/msg60234.html > > Jordan > > > On Mon, Aug 10, 2020 at 4:16 AM Benedict Elliott Smith < > bened...@apache.org> > wrote: > > > Have we considered first asking the user list if there's anyone > willing to > > donate resources to maintain compatibility? > > > > I know I have in the (distant) past handled Jira filed by > (production) > > Windows users. I don’t know how prevalent they are, but perhaps we > should > > offer them a chance to step up before cutting them off? I understand > > nobody presently involved has the resources or inclination to > maintain > > them, but if the effort is low it is not infeasible that somebody > else > > might. > > > > On 10/08/2020, 12:11, "Aleksey Yeshchenko" > > > wrote: > > > > +1 > > > > > On 10 Aug 2020, at 04:14, Yuki Morishita > wrote: > > > > > > As per the discussion(*), I propose to remove Windows support > from > > 4.0 > > > release and onward. > > > > > > Windows scripts are not maintained and we lack windows test > > > environments. WIndows users can use docker or cloud > environments to > > > set up Cassandra application development. > > > > > > If the vote pass, I will create the following tickets to > officially > > > remove Windows support from 4.0: > > > > > > - Remove Windows scripts and add notice to NEWS.txt > > > - Update "Getting Started" documents for Windows users (to > direct > > them > > > to use docker or cloud) > > > > > > Regards, > > > Yuki > > > > > > -- > > > *: > > > https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > > > > > > - > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [Vote] Remove Windows support from 4.0+
I think it would seem a bit more legitimate to remove support without a normal deprecation cycle if we've reached out with an explicit request for donation of support, making it explicit that we will be removing support if none is forthcoming. But just my 2c; like most people on this list, even discussing Windows support barely registers on my priority list. On 10/08/2020, 17:53, "Jordan West" wrote: It wasn't directly regarding removing support but we did reach out to cassandra-users@ for testing 4.0 on Windows and got no response: https://www.mail-archive.com/user@cassandra.apache.org/msg60234.html Jordan On Mon, Aug 10, 2020 at 4:16 AM Benedict Elliott Smith wrote: > Have we considered first asking the user list if there's anyone willing to > donate resources to maintain compatibility? > > I know I have in the (distant) past handled Jira filed by (production) > Windows users. I don’t know how prevalent they are, but perhaps we should > offer them a chance to step up before cutting them off? I understand > nobody presently involved has the resources or inclination to maintain > them, but if the effort is low it is not infeasible that somebody else > might. > > On 10/08/2020, 12:11, "Aleksey Yeshchenko" > wrote: > > +1 > > > On 10 Aug 2020, at 04:14, Yuki Morishita wrote: > > > > As per the discussion(*), I propose to remove Windows support from > 4.0 > > release and onward. > > > > Windows scripts are not maintained and we lack windows test > > environments. WIndows users can use docker or cloud environments to > > set up Cassandra application development. > > > > If the vote pass, I will create the following tickets to officially > > remove Windows support from 4.0: > > > > - Remove Windows scripts and add notice to NEWS.txt > > - Update "Getting Started" documents for Windows users (to direct > them > > to use docker or cloud) > > > > Regards, > > Yuki > > > > -- > > *: > https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [Vote] Remove Windows support from 4.0+
It wasn't directly regarding removing support but we did reach out to cassandra-users@ for testing 4.0 on Windows and got no response: https://www.mail-archive.com/user@cassandra.apache.org/msg60234.html Jordan On Mon, Aug 10, 2020 at 4:16 AM Benedict Elliott Smith wrote: > Have we considered first asking the user list if there's anyone willing to > donate resources to maintain compatibility? > > I know I have in the (distant) past handled Jira filed by (production) > Windows users. I don’t know how prevalent they are, but perhaps we should > offer them a chance to step up before cutting them off? I understand > nobody presently involved has the resources or inclination to maintain > them, but if the effort is low it is not infeasible that somebody else > might. > > On 10/08/2020, 12:11, "Aleksey Yeshchenko" > wrote: > > +1 > > > On 10 Aug 2020, at 04:14, Yuki Morishita wrote: > > > > As per the discussion(*), I propose to remove Windows support from > 4.0 > > release and onward. > > > > Windows scripts are not maintained and we lack windows test > > environments. WIndows users can use docker or cloud environments to > > set up Cassandra application development. > > > > If the vote pass, I will create the following tickets to officially > > remove Windows support from 4.0: > > > > - Remove Windows scripts and add notice to NEWS.txt > > - Update "Getting Started" documents for Windows users (to direct > them > > to use docker or cloud) > > > > Regards, > > Yuki > > > > -- > > *: > https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [Vote] Remove Windows support from 4.0+
+1 On Sun, Aug 9, 2020, 10:15 PM Yuki Morishita wrote: > As per the discussion(*), I propose to remove Windows support from 4.0 > release and onward. > > Windows scripts are not maintained and we lack windows test > environments. WIndows users can use docker or cloud environments to > set up Cassandra application development. > > If the vote pass, I will create the following tickets to officially > remove Windows support from 4.0: > > - Remove Windows scripts and add notice to NEWS.txt > - Update "Getting Started" documents for Windows users (to direct them > to use docker or cloud) > > Regards, > Yuki > > -- > *: > https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [Vote] Remove Windows support from 4.0+
Have we considered first asking the user list if there's anyone willing to donate resources to maintain compatibility? I know I have in the (distant) past handled Jira filed by (production) Windows users. I don’t know how prevalent they are, but perhaps we should offer them a chance to step up before cutting them off? I understand nobody presently involved has the resources or inclination to maintain them, but if the effort is low it is not infeasible that somebody else might. On 10/08/2020, 12:11, "Aleksey Yeshchenko" wrote: +1 > On 10 Aug 2020, at 04:14, Yuki Morishita wrote: > > As per the discussion(*), I propose to remove Windows support from 4.0 > release and onward. > > Windows scripts are not maintained and we lack windows test > environments. WIndows users can use docker or cloud environments to > set up Cassandra application development. > > If the vote pass, I will create the following tickets to officially > remove Windows support from 4.0: > > - Remove Windows scripts and add notice to NEWS.txt > - Update "Getting Started" documents for Windows users (to direct them > to use docker or cloud) > > Regards, > Yuki > > -- > *: https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [Vote] Remove Windows support from 4.0+
+1 > On 10 Aug 2020, at 04:14, Yuki Morishita wrote: > > As per the discussion(*), I propose to remove Windows support from 4.0 > release and onward. > > Windows scripts are not maintained and we lack windows test > environments. WIndows users can use docker or cloud environments to > set up Cassandra application development. > > If the vote pass, I will create the following tickets to officially > remove Windows support from 4.0: > > - Remove Windows scripts and add notice to NEWS.txt > - Update "Getting Started" documents for Windows users (to direct them > to use docker or cloud) > > Regards, > Yuki > > -- > *: > https://mail-archives.apache.org/mod_mbox/cassandra-dev/202007.mbox/%3CCAGM0Up_3GoPucCP-U18L1akzBXS1eJoKbui997%3DajcCfKJQdng%40mail.gmail.com%3E > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
