I recall hearing that Apache HAWQ already runs Concourse in a similar approach.
I also know of at least three other open source projects are running Concourse: 1. Concourse.ci <https://ci.concourse.ci/> itself 2. Greenplum Database <http://gpdb.ci.pivotalci.info/> 3. RabbitMQ <https://ci-aws.rabbitmq.com/> On Fri, Oct 6, 2017 at 2:08 PM, Anthony Baker <aba...@pivotal.io> wrote: > Hi all, > > I’d like to propose the following that we switch our continuous > integration (CI) system from Jenkins [1] to Concourse [2]. I suggest > this because we continue to experience a significant number of > environmental-related test failures. > > These issues include CPU interference from other Jenkins jobs on the > same host, running out of disk space, port conflicts, and other > gremlins. The net effect is that we are only getting 1-2 successful > builds per month. Certainly not all test failures can be traced back > to environmental issues. However, internal testing on isolated VM’s > shows a combined success rate of about 3X higher compared to ASF > Jenkins for the same tests. This is still definitely NotAwesome, but > removing environmental factors will let us focus on stabilizing flaky > tests. > > Concourse is an Apache-licensed open source CI system based on > pipelines. The pipelines are defined in a YML file containing job > definitions—inputs, outputs, resources, and tasks. A task is simply a > bash script that returns 0/1 for success/failure. A web UI displays > build status. Importantly, each job runs inside an isolated > container. The containers are load-balanced across a pool of workers. > For an example of a build pipeline, see [3] for the pipeline used to > build concourse itself. > > A Concourse environment is deployed and managed in cloud environments > through bosh [4]. Pivotal has agreed to donate AWS and/or GCP compute > and storage resources as well as manage the infrastructure. These > project resources would be available for use by all committers and > community members regardless of corporate affiliations. Note that > AFAIK there is no explicit requirement to host CI on ASF > infrastructure—unlike for critical project resources such as source > code, mailing lists, and issue tracking. > > The source for the pipeline and job scripts would reside within the > geode-* repos. Geode committers would be able to modify those, same > as with our .travis.yml scripts. All test results and build artifacts > would be publicly viewable just like with our Jenkins build output > today. Requests for admin assistance would go through the dev@geode > mailing list. > > Thoughts? As a first step we could run both CI systems side-by-side > and see how the Concourse approach works for our project. > > Thanks, > Anthony > > > [1] https://builds.apache.org/job/Geode-nightly/ > [2] https://concourse.ci > [3] https://ci.concourse.ci > [4] https://bosh.io > -- Greg Chase Product team, Pivotal Cloud Foundry Services https://pivotal.io/platform/services Pivotal Software http://www.pivotal.io/ 650-215-0477 @GregChase
