[DISCUSS] CodeClimate analysis of Geode

[Alexander Murmann]

Hi everyone,

I set up my fork of Geode with CodeClimate to see if the tool might get us
interesting information. You can see the results here


The current configuration of CodeClimate uses PMD (https://pmd.github.io/)
with the *Basic, Code Size* and *Coupling* rulesets
.
Geode has six *critical* issues

that we should take a look at as soon as possible. I have not yet
identified if these are false positives or not.

The biggest value we can get out of this tool right now seems to be the *churn
vs. code quality*
 graph.
We have 65,000+ issues in total. It's not feasible to address all of these.
We can however use the *churn vs. code quality* graph to guide our
decisions in what to refactor. Code that changes frequently and is of poor
quality is a prime refactoring target. Michael Feathers explains this much
better than I can:
https://www.stickyminds.com/article/getting-empirical-about-refactoring

Do you find this analysis interesting? Are there any rules in the PMD tool
you'd like to see added or removed?

Thanks for all feedback and input on this!

Failed: jinmeiliao/geode#30 (gem-1691 - 274b439)

[Travis CI]

Build Update for jinmeiliao/geode
-

Build: #30
Status: Failed

Duration: 12 minutes and 46 seconds
Commit: 274b439 (gem-1691)
Author: Jinmei Liao
Message: GEODE-3785: correctly update the schema version, namespace and 
lcoation when importing old schema

View the changeset: 
https://github.com/jinmeiliao/geode/compare/2284ea6978e5^...274b439d060a

View the full build log and details: 
https://travis-ci.org/jinmeiliao/geode/builds/284434318?utm_source=email_medium=notification

--

You can configure recipients for build notifications in your .travis.yml file. 
See https://docs.travis-ci.com/user/notifications

Broken: apache/geode#4200 (develop - d1ef591)

[Travis CI]

Build Update for apache/geode
-

Build: #4200
Status: Broken

Duration: 7 minutes and 16 seconds
Commit: d1ef591 (develop)
Author: Jens Deppe
Message: GEODE-3746: FunctionCommandsDUnitTest uses http as transport

View the changeset: 
https://github.com/apache/geode/compare/3dd324452d58...d1ef59150748

View the full build log and details: 
https://travis-ci.org/apache/geode/builds/284409064?utm_source=email_medium=notification

--

You can configure recipients for build notifications in your .travis.yml file. 
See https://docs.travis-ci.com/user/notifications

Build failed in Jenkins: Geode-nightly-flaky #141

[Apache Jenkins Server]

See 


Changes:

[jdeppe] GEODE-3542: Add descriptive error message for gfsh show metrics

[jdeppe] GEODE-3542: Increase await timeout to 2 minutes in case of long GCs on

[boglesby] GEODE-3730: Moved retry logic to receiver when

[jdeppe] GEODE-3542: Refactor test for show metrics command

[jdeppe] GEODE-3542: Spotless changes

[jdeppe] GEODE-3723: Remove Optional from getRequiredPermissions

[jdeppe] GEODE-3542: Add license headers

[jdeppe] GEODE-3723: Correct documentation for getRequiredPermissions

[github] Fix the path error in the Javadoc

[boglesby] GEODE-3730: Moved retry loop around switch statement

[kohlmu-pivotal] GEODE-3732: Amended the test to better handle timeout scenarios

[jinmeiliao] GEODE-3621: revert the change to maintain backward compatibility

[kohlmu-pivotal] GEODE-3749: 
DeltaPropagationDUnitTest.testBug40165ClientReconnects

[github] GEODE-3723 Revise javadoc defn of getRequiredPermissions parameter

[eshu] GEODE-3744: Local region will not participate in a transaction hosted

[jstewart] GEODE-3685: MBean wrappers are properly invoked over http

[kohlmu-pivotal] GEODE-3750: Adding exception logging in auto reconnect

[github] GEODE-3703 Document JAR resource becomes DEPLOY (#860)

[bschuchardt] GEODE-3695 Remove unused protobuf messages from new protocol

[khowe] GEODE-3539: Enhanced test coverage for alter runtime command options

[khowe] GEODE-3539: refactored to reduce code duplication

[jdeppe] GEODE-3542: Add null guards in the case of invalid commands

[jdeppe] GEODE-3542: Make sure that gfsh is stopped properly at the end of tests

[klund] GEODE-3747: add SharedErrorCollector rule

[jinmeiliao] GEODE-3567: correctly set the serverPortSetByUser flag

[kohlmu-pivotal] GEODE-3733: Added Flaky test to the test

[jinmeiliao] Consolidate Http request

[dbarnes] Update user guide build environment to v1.3

[github] GEODE-2817 Document user-defined function authorization levels (#855)

[dsmith] GEODE-3687: GatewayReceiverImpl constructor and create method updates

[bschuchardt] GEODE-3695 Remove unused protobuf messages from new protocol

[bschuchardt] GEODE-3760 uncaught exception in shunned member removal thread

[huynhja] GEODE-3736: CompactRangeIndex getSizeEstimate should not throw

[huynhja] SpotlessApply and added comment into test describing when

[huynhja] Organized imports

[eshu] GEODE-3679 Forward client member id to other peers in transaction

[github] GEODE-3096: Unify gfsh Http clients (#875)

[kirk.lund] GEODE-3755: make resilient to variable number of dunit VMs

[github] GEODE-3742: Added logging to help identify this issue

[jdeppe] GEODE-3775: Fix typo 'jmxManger' should be 'jmxManager'

[klund] GEODE-3755: apply review feedback to tests

[kirk.lund] GEODE-3753: move CacheRule into dunit.rules package

[klund] GEODE-3777: delete useless broken test

[dbarnes] GEODE-3378: Document Snapshot Improvements. Incorporate reviewer

[kohlmu-pivotal] GEODE-3779: Cleaned up some error handling code in new Protocol

--
[...truncated 119.53 KB...]
:geode-web-api:processResources
:geode-web-api:classes
:geode-assembly:docs
:geode-assembly:gfshDepsJar
:geode-common:javadocJar
:geode-common:sourcesJar
:geode-common:signArchives SKIPPED
:geode-core:javadocJar
:geode-core:raJar
:geode-core:jcaJar
:geode-core:sourcesJar
:geode-core:signArchives SKIPPED
:geode-core:webJar
:geode-cq:jar
:geode-cq:javadoc
:geode-cq:javadocJar
:geode-cq:sourcesJar
:geode-cq:signArchives SKIPPED
:geode-json:javadocJar
:geode-json:sourcesJar
:geode-json:signArchives SKIPPED
:geode-lucene:jar
:geode-lucene:javadoc
:geode-lucene:javadocJar
:geode-lucene:sourcesJar
:geode-lucene:signArchives SKIPPED
:geode-old-client-support:jar
:geode-old-client-support:javadoc
:geode-old-client-support:javadocJar
:geode-old-client-support:sourcesJar
:geode-old-client-support:signArchives SKIPPED
:geode-protobuf:jar
:geode-protobuf:javadoc
:geode-protobuf:javadocJar
:geode-protobuf:sourcesJar
:geode-protobuf:signArchives SKIPPED
:geode-pulse:javadoc
:geode-pulse:javadocJar
:geode-pulse:sourcesJar
:geode-pulse:war
:geode-pulse:signArchives SKIPPED
:geode-rebalancer:jar
:geode-rebalancer:javadoc
:geode-rebalancer:javadocJar
:geode-rebalancer:sourcesJar
:geode-rebalancer:signArchives SKIPPED
:geode-wan:jar
:geode-wan:javadoc
:geode-wan:javadocJar
:geode-wan:sourcesJar
:geode-wan:signArchives SKIPPED
:geode-web:javadoc NO-SOURCE
:geode-web:javadocJar
:geode-web:sourcesJar
:geode-web:war
:geode-web:signArchives SKIPPED
:geode-web-api:javadoc
:geode-web-api:javadocJar
:geode-web-api:sourcesJar
:geode-web-api:war
:geode-web-api:signArchives SKIPPED
:geode-assembly:installDist
:geode-pulse:jar
:geode-assembly:compileTestJava
Download 
https://repo1.maven.org/maven2/org/codehaus/cargo/cargo-core-uberjar/1.6.3/cargo-core-uberjar-1.6.3.pom
Download 

Re: [DISCUSS] CI improvements

[Anthony Baker]

Comments inline

> On Oct 6, 2017, at 11:51 AM, Mark Bretl  wrote:
> 
> Correct, there is no requirement from ASF about where to run CI or even to
> run CI. I am all for the best tools and stable (and repeatable)
> environments. I would be open to seeing how Concourse could work for Geode.
> 
> Once the Concourse environment is setup, probably would be best to update
> the Jenkins job to a pipeline job type as well.
> 
> There a few questions I have, which can be discussed as we go:
> - What does 'donate' mean for Pivotal? Is there a, for lack of a better
> term, contract or is this a 'good faith' effort? I am hesitant to go to any
> corporately controlled infrastructure, especially since the entity could
> decide to stop funding.

Pivotal is offering to fund a cloud account with resources for the Geode 
project.  I can’t predict the future but I wouldn’t be surprised to see more 
ASF projects receiving this kind of support in the future given the increasing 
popularity of cloud computing (not to mention the complexity of donating 
capital assets).  If at some point Pivotal does stop funding this effort and we 
can’t find another donor, we can always fall back to ASF Jenkins :-)

> - What types of activities will require admin assistance?

Admin support should be fairly minimal and would include activities like 
initial deployment and configuration, upgrades to bosh / concourse, adding more 
compute resources, or troubleshooting occasional cloud hiccups.

> - Who will be watching the dev list for requests?

Assuming we reach consensus to move forward with this proposal, expect to see 
some introductions on the dev list.  Does that help?

> 
> --Mark

Build failed in Jenkins: Geode-nightly #976

[Apache Jenkins Server]

See 


Changes:

[boglesby] GEODE-3730: Moved retry logic to receiver when

[boglesby] GEODE-3730: Moved retry loop around switch statement

[eshu] GEODE-3744: Local region will not participate in a transaction hosted

[bschuchardt] GEODE-3695 Remove unused protobuf messages from new protocol

[jinmeiliao] GEODE-3567: correctly set the serverPortSetByUser flag

[kohlmu-pivotal] GEODE-3733: Added Flaky test to the test

[jinmeiliao] Consolidate Http request

[dbarnes] Update user guide build environment to v1.3

[github] GEODE-2817 Document user-defined function authorization levels (#855)

[dsmith] GEODE-3687: GatewayReceiverImpl constructor and create method updates

[bschuchardt] GEODE-3695 Remove unused protobuf messages from new protocol

[bschuchardt] GEODE-3760 uncaught exception in shunned member removal thread

[huynhja] GEODE-3736: CompactRangeIndex getSizeEstimate should not throw

[huynhja] SpotlessApply and added comment into test describing when

[huynhja] Organized imports

[eshu] GEODE-3679 Forward client member id to other peers in transaction

--
[...truncated 253.68 KB...]
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl$1.create(MemoryAllocatorImpl.java:93)
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl.create(MemoryAllocatorImpl.java:122)
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl.create(MemoryAllocatorImpl.java:89)
at 
org.apache.geode.internal.offheap.OffHeapStorage.basicCreateOffHeapStorage(OffHeapStorage.java:219)
at 
org.apache.geode.internal.offheap.OffHeapStorage.createOffHeapStorage(OffHeapStorage.java:207)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:700)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:350)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:336)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:330)
at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:205)
at 
org.apache.geode.test.dunit.internal.JUnit4DistributedTestCase.getSystem(JUnit4DistributedTestCase.java:180)
at 
org.apache.geode.internal.cache.wan.WANTestBase.createFirstLocatorWithDSId(WANTestBase.java:283)
at 
org.apache.geode.internal.cache.wan.concurrent.ConcurrentWANPropagation_2_DUnitTest.lambda$testReplicatedSerialPropagationWithParallelThreads$a2e85a4$1(ConcurrentWANPropagation_2_DUnitTest.java:266)

org.apache.geode.internal.cache.wan.concurrent.ConcurrentWANPropagation_2_DUnitTest
 > testReplicatedSerialPropagationWithConflation FAILED
org.apache.geode.test.dunit.RMIException: While invoking 
org.apache.geode.internal.cache.wan.concurrent.ConcurrentWANPropagation_2_DUnitTest$$Lambda$1309/1808798357.call
 in VM 0 running on Host 6a4d7f99f695 with 8 VMs
at org.apache.geode.test.dunit.VM.invoke(VM.java:393)
at org.apache.geode.test.dunit.VM.invoke(VM.java:363)
at org.apache.geode.test.dunit.VM.invoke(VM.java:331)
at 
org.apache.geode.internal.cache.wan.concurrent.ConcurrentWANPropagation_2_DUnitTest.testReplicatedSerialPropagationWithConflation(ConcurrentWANPropagation_2_DUnitTest.java:228)

Caused by:
java.lang.OutOfMemoryError: Failed creating 314572800 bytes of off-heap 
memory during cache creation.
at 
org.apache.geode.internal.offheap.AddressableMemoryManager.allocate(AddressableMemoryManager.java:64)
at 
org.apache.geode.internal.offheap.SlabImpl.(SlabImpl.java:35)
at 
org.apache.geode.internal.offheap.SlabImpl.(SlabImpl.java:27)
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl$1.create(MemoryAllocatorImpl.java:93)
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl.create(MemoryAllocatorImpl.java:122)
at 
org.apache.geode.internal.offheap.MemoryAllocatorImpl.create(MemoryAllocatorImpl.java:89)
at 
org.apache.geode.internal.offheap.OffHeapStorage.basicCreateOffHeapStorage(OffHeapStorage.java:219)
at 
org.apache.geode.internal.offheap.OffHeapStorage.createOffHeapStorage(OffHeapStorage.java:207)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:700)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:350)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:336)
at 

Re: [DISCUSS] CI improvements

[Mark Bretl]

Correct, there is no requirement from ASF about where to run CI or even to
run CI. I am all for the best tools and stable (and repeatable)
environments. I would be open to seeing how Concourse could work for Geode.

Once the Concourse environment is setup, probably would be best to update
the Jenkins job to a pipeline job type as well.

There a few questions I have, which can be discussed as we go:
- What does 'donate' mean for Pivotal? Is there a, for lack of a better
term, contract or is this a 'good faith' effort? I am hesitant to go to any
corporately controlled infrastructure, especially since the entity could
decide to stop funding.
- What types of activities will require admin assistance?
- Who will be watching the dev list for requests?

--Mark

On Fri, Oct 6, 2017 at 10:05 AM, Kenneth Howe  wrote:

> +1 - Reduce the noise level for analyzing CI results
>
> > On Oct 6, 2017, at 9:49 AM, Udo Kohlmeyer  wrote:
> >
> > +1 Switch... parallel runs would be safest
> >
> > On Fri, Oct 6, 2017 at 9:42 AM, Jianxia Chen  wrote:
> >
> >> +1 to switch to Concourse
> >> +1 As a first step we could run both CI systems side-by-side and see how
> >> the Concourse approach works for our project.
> >>
> >> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker 
> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I’d like to propose the following that we switch our continuous
> >>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> >>> this because we continue to experience a significant number of
> >>> environmental-related test failures.
> >>>
> >>> These issues include CPU interference from other Jenkins jobs on the
> >>> same host, running out of disk space, port conflicts, and other
> >>> gremlins.  The net effect is that we are only getting 1-2 successful
> >>> builds per month.  Certainly not all test failures can be traced back
> >>> to environmental issues.  However, internal testing on isolated VM’s
> >>> shows a combined success rate of about 3X higher compared to ASF
> >>> Jenkins for the same tests.  This is still definitely NotAwesome, but
> >>> removing environmental factors will let us focus on stabilizing flaky
> >>> tests.
> >>>
> >>> Concourse is an Apache-licensed open source CI system based on
> >>> pipelines.  The pipelines are defined in a YML file containing job
> >>> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> >>> bash script that returns 0/1 for success/failure.  A web UI displays
> >>> build status.  Importantly, each job runs inside an isolated
> >>> container.  The containers are load-balanced across a pool of workers.
> >>> For an example of a build pipeline, see [3] for the pipeline used to
> >>> build concourse itself.
> >>>
> >>> A Concourse environment is deployed and managed in cloud environments
> >>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> >>> and storage resources as well as manage the infrastructure.  These
> >>> project resources would be available for use by all committers and
> >>> community members regardless of corporate affiliations.  Note that
> >>> AFAIK there is no explicit requirement to host CI on ASF
> >>> infrastructure—unlike for critical project resources such as source
> >>> code, mailing lists, and issue tracking.
> >>>
> >>> The source for the pipeline and job scripts would reside within the
> >>> geode-* repos.  Geode committers would be able to modify those, same
> >>> as with our .travis.yml scripts.  All test results and build artifacts
> >>> would be publicly viewable just like with our Jenkins build output
> >>> today.  Requests for admin assistance would go through the dev@geode
> >>> mailing list.
> >>>
> >>> Thoughts?  As a first step we could run both CI systems side-by-side
> >>> and see how the Concourse approach works for our project.
> >>>
> >>> Thanks,
> >>> Anthony
> >>>
> >>>
> >>> [1] https://builds.apache.org/job/Geode-nightly/
> >>> [2] https://concourse.ci
> >>> [3] https://ci.concourse.ci
> >>> [4] https://bosh.io
> >>>
> >>
> >
> >
> >
> > --
> > Kindest Regards
> > -
> > *Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal*
> > *Mobile:* +61 409-279-160 | ukohlme...@pivotal.io
> > 
> > www.pivotal.io
>
>

Re: [DISCUSS] CI improvements

[Kenneth Howe]

+1 - Reduce the noise level for analyzing CI results

> On Oct 6, 2017, at 9:49 AM, Udo Kohlmeyer  wrote:
> 
> +1 Switch... parallel runs would be safest
> 
> On Fri, Oct 6, 2017 at 9:42 AM, Jianxia Chen  wrote:
> 
>> +1 to switch to Concourse
>> +1 As a first step we could run both CI systems side-by-side and see how
>> the Concourse approach works for our project.
>> 
>> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker  wrote:
>> 
>>> Hi all,
>>> 
>>> I’d like to propose the following that we switch our continuous
>>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
>>> this because we continue to experience a significant number of
>>> environmental-related test failures.
>>> 
>>> These issues include CPU interference from other Jenkins jobs on the
>>> same host, running out of disk space, port conflicts, and other
>>> gremlins.  The net effect is that we are only getting 1-2 successful
>>> builds per month.  Certainly not all test failures can be traced back
>>> to environmental issues.  However, internal testing on isolated VM’s
>>> shows a combined success rate of about 3X higher compared to ASF
>>> Jenkins for the same tests.  This is still definitely NotAwesome, but
>>> removing environmental factors will let us focus on stabilizing flaky
>>> tests.
>>> 
>>> Concourse is an Apache-licensed open source CI system based on
>>> pipelines.  The pipelines are defined in a YML file containing job
>>> definitions—inputs, outputs, resources, and tasks.  A task is simply a
>>> bash script that returns 0/1 for success/failure.  A web UI displays
>>> build status.  Importantly, each job runs inside an isolated
>>> container.  The containers are load-balanced across a pool of workers.
>>> For an example of a build pipeline, see [3] for the pipeline used to
>>> build concourse itself.
>>> 
>>> A Concourse environment is deployed and managed in cloud environments
>>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
>>> and storage resources as well as manage the infrastructure.  These
>>> project resources would be available for use by all committers and
>>> community members regardless of corporate affiliations.  Note that
>>> AFAIK there is no explicit requirement to host CI on ASF
>>> infrastructure—unlike for critical project resources such as source
>>> code, mailing lists, and issue tracking.
>>> 
>>> The source for the pipeline and job scripts would reside within the
>>> geode-* repos.  Geode committers would be able to modify those, same
>>> as with our .travis.yml scripts.  All test results and build artifacts
>>> would be publicly viewable just like with our Jenkins build output
>>> today.  Requests for admin assistance would go through the dev@geode
>>> mailing list.
>>> 
>>> Thoughts?  As a first step we could run both CI systems side-by-side
>>> and see how the Concourse approach works for our project.
>>> 
>>> Thanks,
>>> Anthony
>>> 
>>> 
>>> [1] https://builds.apache.org/job/Geode-nightly/
>>> [2] https://concourse.ci
>>> [3] https://ci.concourse.ci
>>> [4] https://bosh.io
>>> 
>> 
> 
> 
> 
> -- 
> Kindest Regards
> -
> *Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal*
> *Mobile:* +61 409-279-160 | ukohlme...@pivotal.io
> 
> www.pivotal.io

Re: [DISCUSS] CI improvements

[Udo Kohlmeyer]

+1 Switch... parallel runs would be safest

On Fri, Oct 6, 2017 at 9:42 AM, Jianxia Chen  wrote:

> +1 to switch to Concourse
> +1 As a first step we could run both CI systems side-by-side and see how
> the Concourse approach works for our project.
>
> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker  wrote:
>
> > Hi all,
> >
> > I’d like to propose the following that we switch our continuous
> > integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> > this because we continue to experience a significant number of
> > environmental-related test failures.
> >
> > These issues include CPU interference from other Jenkins jobs on the
> > same host, running out of disk space, port conflicts, and other
> > gremlins.  The net effect is that we are only getting 1-2 successful
> > builds per month.  Certainly not all test failures can be traced back
> > to environmental issues.  However, internal testing on isolated VM’s
> > shows a combined success rate of about 3X higher compared to ASF
> > Jenkins for the same tests.  This is still definitely NotAwesome, but
> > removing environmental factors will let us focus on stabilizing flaky
> > tests.
> >
> > Concourse is an Apache-licensed open source CI system based on
> > pipelines.  The pipelines are defined in a YML file containing job
> > definitions—inputs, outputs, resources, and tasks.  A task is simply a
> > bash script that returns 0/1 for success/failure.  A web UI displays
> > build status.  Importantly, each job runs inside an isolated
> > container.  The containers are load-balanced across a pool of workers.
> > For an example of a build pipeline, see [3] for the pipeline used to
> > build concourse itself.
> >
> > A Concourse environment is deployed and managed in cloud environments
> > through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> > and storage resources as well as manage the infrastructure.  These
> > project resources would be available for use by all committers and
> > community members regardless of corporate affiliations.  Note that
> > AFAIK there is no explicit requirement to host CI on ASF
> > infrastructure—unlike for critical project resources such as source
> > code, mailing lists, and issue tracking.
> >
> > The source for the pipeline and job scripts would reside within the
> > geode-* repos.  Geode committers would be able to modify those, same
> > as with our .travis.yml scripts.  All test results and build artifacts
> > would be publicly viewable just like with our Jenkins build output
> > today.  Requests for admin assistance would go through the dev@geode
> > mailing list.
> >
> > Thoughts?  As a first step we could run both CI systems side-by-side
> > and see how the Concourse approach works for our project.
> >
> > Thanks,
> > Anthony
> >
> >
> > [1] https://builds.apache.org/job/Geode-nightly/
> > [2] https://concourse.ci
> > [3] https://ci.concourse.ci
> > [4] https://bosh.io
> >
>



-- 
Kindest Regards
-
*Udo Kohlmeyer* | *Snr Solutions Architect* |*Pivotal*
*Mobile:* +61 409-279-160 | ukohlme...@pivotal.io

www.pivotal.io

Re: [DISCUSS] CI improvements

[Jianxia Chen]

+1 to switch to Concourse
+1 As a first step we could run both CI systems side-by-side and see how
the Concourse approach works for our project.

On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker  wrote:

> Hi all,
>
> I’d like to propose the following that we switch our continuous
> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> this because we continue to experience a significant number of
> environmental-related test failures.
>
> These issues include CPU interference from other Jenkins jobs on the
> same host, running out of disk space, port conflicts, and other
> gremlins.  The net effect is that we are only getting 1-2 successful
> builds per month.  Certainly not all test failures can be traced back
> to environmental issues.  However, internal testing on isolated VM’s
> shows a combined success rate of about 3X higher compared to ASF
> Jenkins for the same tests.  This is still definitely NotAwesome, but
> removing environmental factors will let us focus on stabilizing flaky
> tests.
>
> Concourse is an Apache-licensed open source CI system based on
> pipelines.  The pipelines are defined in a YML file containing job
> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> bash script that returns 0/1 for success/failure.  A web UI displays
> build status.  Importantly, each job runs inside an isolated
> container.  The containers are load-balanced across a pool of workers.
> For an example of a build pipeline, see [3] for the pipeline used to
> build concourse itself.
>
> A Concourse environment is deployed and managed in cloud environments
> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> and storage resources as well as manage the infrastructure.  These
> project resources would be available for use by all committers and
> community members regardless of corporate affiliations.  Note that
> AFAIK there is no explicit requirement to host CI on ASF
> infrastructure—unlike for critical project resources such as source
> code, mailing lists, and issue tracking.
>
> The source for the pipeline and job scripts would reside within the
> geode-* repos.  Geode committers would be able to modify those, same
> as with our .travis.yml scripts.  All test results and build artifacts
> would be publicly viewable just like with our Jenkins build output
> today.  Requests for admin assistance would go through the dev@geode
> mailing list.
>
> Thoughts?  As a first step we could run both CI systems side-by-side
> and see how the Concourse approach works for our project.
>
> Thanks,
> Anthony
>
>
> [1] https://builds.apache.org/job/Geode-nightly/
> [2] https://concourse.ci
> [3] https://ci.concourse.ci
> [4] https://bosh.io
>

Re: [DISCUSS] CI improvements

[Jared Stewart]

+1 I think this will be a huge improvement to the reliability of our test 
infrastructure.

- Jared

> On Oct 6, 2017, at 9:26 AM, Kirk Lund  wrote:
> 
> +1 no thoughts other than make it so!
> 
> On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker  wrote:
> 
>> Hi all,
>> 
>> I’d like to propose the following that we switch our continuous
>> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
>> this because we continue to experience a significant number of
>> environmental-related test failures.
>> 
>> These issues include CPU interference from other Jenkins jobs on the
>> same host, running out of disk space, port conflicts, and other
>> gremlins.  The net effect is that we are only getting 1-2 successful
>> builds per month.  Certainly not all test failures can be traced back
>> to environmental issues.  However, internal testing on isolated VM’s
>> shows a combined success rate of about 3X higher compared to ASF
>> Jenkins for the same tests.  This is still definitely NotAwesome, but
>> removing environmental factors will let us focus on stabilizing flaky
>> tests.
>> 
>> Concourse is an Apache-licensed open source CI system based on
>> pipelines.  The pipelines are defined in a YML file containing job
>> definitions—inputs, outputs, resources, and tasks.  A task is simply a
>> bash script that returns 0/1 for success/failure.  A web UI displays
>> build status.  Importantly, each job runs inside an isolated
>> container.  The containers are load-balanced across a pool of workers.
>> For an example of a build pipeline, see [3] for the pipeline used to
>> build concourse itself.
>> 
>> A Concourse environment is deployed and managed in cloud environments
>> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
>> and storage resources as well as manage the infrastructure.  These
>> project resources would be available for use by all committers and
>> community members regardless of corporate affiliations.  Note that
>> AFAIK there is no explicit requirement to host CI on ASF
>> infrastructure—unlike for critical project resources such as source
>> code, mailing lists, and issue tracking.
>> 
>> The source for the pipeline and job scripts would reside within the
>> geode-* repos.  Geode committers would be able to modify those, same
>> as with our .travis.yml scripts.  All test results and build artifacts
>> would be publicly viewable just like with our Jenkins build output
>> today.  Requests for admin assistance would go through the dev@geode
>> mailing list.
>> 
>> Thoughts?  As a first step we could run both CI systems side-by-side
>> and see how the Concourse approach works for our project.
>> 
>> Thanks,
>> Anthony
>> 
>> 
>> [1] https://builds.apache.org/job/Geode-nightly/
>> [2] https://concourse.ci
>> [3] https://ci.concourse.ci
>> [4] https://bosh.io
>> 

Re: [DISCUSS] CI improvements

[Kirk Lund]

+1 no thoughts other than make it so!

On Fri, Oct 6, 2017 at 7:08 AM, Anthony Baker  wrote:

> Hi all,
>
> I’d like to propose the following that we switch our continuous
> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> this because we continue to experience a significant number of
> environmental-related test failures.
>
> These issues include CPU interference from other Jenkins jobs on the
> same host, running out of disk space, port conflicts, and other
> gremlins.  The net effect is that we are only getting 1-2 successful
> builds per month.  Certainly not all test failures can be traced back
> to environmental issues.  However, internal testing on isolated VM’s
> shows a combined success rate of about 3X higher compared to ASF
> Jenkins for the same tests.  This is still definitely NotAwesome, but
> removing environmental factors will let us focus on stabilizing flaky
> tests.
>
> Concourse is an Apache-licensed open source CI system based on
> pipelines.  The pipelines are defined in a YML file containing job
> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> bash script that returns 0/1 for success/failure.  A web UI displays
> build status.  Importantly, each job runs inside an isolated
> container.  The containers are load-balanced across a pool of workers.
> For an example of a build pipeline, see [3] for the pipeline used to
> build concourse itself.
>
> A Concourse environment is deployed and managed in cloud environments
> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> and storage resources as well as manage the infrastructure.  These
> project resources would be available for use by all committers and
> community members regardless of corporate affiliations.  Note that
> AFAIK there is no explicit requirement to host CI on ASF
> infrastructure—unlike for critical project resources such as source
> code, mailing lists, and issue tracking.
>
> The source for the pipeline and job scripts would reside within the
> geode-* repos.  Geode committers would be able to modify those, same
> as with our .travis.yml scripts.  All test results and build artifacts
> would be publicly viewable just like with our Jenkins build output
> today.  Requests for admin assistance would go through the dev@geode
> mailing list.
>
> Thoughts?  As a first step we could run both CI systems side-by-side
> and see how the Concourse approach works for our project.
>
> Thanks,
> Anthony
>
>
> [1] https://builds.apache.org/job/Geode-nightly/
> [2] https://concourse.ci
> [3] https://ci.concourse.ci
> [4] https://bosh.io
>

Re: [DISCUSS] CI improvements

[Jinmei Liao]

+1 for switching concourse and running it side by side first to see how it
works first.

On Fri, Oct 6, 2017 at 7:57 AM, Gregory Chase  wrote:

> I recall hearing that Apache HAWQ already runs Concourse in a similar
> approach.
>
> I also know of at least three other open source projects are running
> Concourse:
>
> 1. Concourse.ci  itself
> 2. Greenplum Database 
> 3. RabbitMQ 
>
> On Fri, Oct 6, 2017 at 2:08 PM, Anthony Baker  wrote:
>
> > Hi all,
> >
> > I’d like to propose the following that we switch our continuous
> > integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> > this because we continue to experience a significant number of
> > environmental-related test failures.
> >
> > These issues include CPU interference from other Jenkins jobs on the
> > same host, running out of disk space, port conflicts, and other
> > gremlins.  The net effect is that we are only getting 1-2 successful
> > builds per month.  Certainly not all test failures can be traced back
> > to environmental issues.  However, internal testing on isolated VM’s
> > shows a combined success rate of about 3X higher compared to ASF
> > Jenkins for the same tests.  This is still definitely NotAwesome, but
> > removing environmental factors will let us focus on stabilizing flaky
> > tests.
> >
> > Concourse is an Apache-licensed open source CI system based on
> > pipelines.  The pipelines are defined in a YML file containing job
> > definitions—inputs, outputs, resources, and tasks.  A task is simply a
> > bash script that returns 0/1 for success/failure.  A web UI displays
> > build status.  Importantly, each job runs inside an isolated
> > container.  The containers are load-balanced across a pool of workers.
> > For an example of a build pipeline, see [3] for the pipeline used to
> > build concourse itself.
> >
> > A Concourse environment is deployed and managed in cloud environments
> > through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> > and storage resources as well as manage the infrastructure.  These
> > project resources would be available for use by all committers and
> > community members regardless of corporate affiliations.  Note that
> > AFAIK there is no explicit requirement to host CI on ASF
> > infrastructure—unlike for critical project resources such as source
> > code, mailing lists, and issue tracking.
> >
> > The source for the pipeline and job scripts would reside within the
> > geode-* repos.  Geode committers would be able to modify those, same
> > as with our .travis.yml scripts.  All test results and build artifacts
> > would be publicly viewable just like with our Jenkins build output
> > today.  Requests for admin assistance would go through the dev@geode
> > mailing list.
> >
> > Thoughts?  As a first step we could run both CI systems side-by-side
> > and see how the Concourse approach works for our project.
> >
> > Thanks,
> > Anthony
> >
> >
> > [1] https://builds.apache.org/job/Geode-nightly/
> > [2] https://concourse.ci
> > [3] https://ci.concourse.ci
> > [4] https://bosh.io
> >
>
>
>
> --
> Greg Chase
>
> Product team, Pivotal Cloud Foundry Services
> https://pivotal.io/platform/services
>
> Pivotal Software
> http://www.pivotal.io/
>
> 650-215-0477
> @GregChase
>



-- 
Cheers

Jinmei

Re: [DISCUSS] CI improvements

[Gregory Chase]

I recall hearing that Apache HAWQ already runs Concourse in a similar
approach.

I also know of at least three other open source projects are running
Concourse:

1. Concourse.ci  itself
2. Greenplum Database 
3. RabbitMQ 

On Fri, Oct 6, 2017 at 2:08 PM, Anthony Baker  wrote:

> Hi all,
>
> I’d like to propose the following that we switch our continuous
> integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
> this because we continue to experience a significant number of
> environmental-related test failures.
>
> These issues include CPU interference from other Jenkins jobs on the
> same host, running out of disk space, port conflicts, and other
> gremlins.  The net effect is that we are only getting 1-2 successful
> builds per month.  Certainly not all test failures can be traced back
> to environmental issues.  However, internal testing on isolated VM’s
> shows a combined success rate of about 3X higher compared to ASF
> Jenkins for the same tests.  This is still definitely NotAwesome, but
> removing environmental factors will let us focus on stabilizing flaky
> tests.
>
> Concourse is an Apache-licensed open source CI system based on
> pipelines.  The pipelines are defined in a YML file containing job
> definitions—inputs, outputs, resources, and tasks.  A task is simply a
> bash script that returns 0/1 for success/failure.  A web UI displays
> build status.  Importantly, each job runs inside an isolated
> container.  The containers are load-balanced across a pool of workers.
> For an example of a build pipeline, see [3] for the pipeline used to
> build concourse itself.
>
> A Concourse environment is deployed and managed in cloud environments
> through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
> and storage resources as well as manage the infrastructure.  These
> project resources would be available for use by all committers and
> community members regardless of corporate affiliations.  Note that
> AFAIK there is no explicit requirement to host CI on ASF
> infrastructure—unlike for critical project resources such as source
> code, mailing lists, and issue tracking.
>
> The source for the pipeline and job scripts would reside within the
> geode-* repos.  Geode committers would be able to modify those, same
> as with our .travis.yml scripts.  All test results and build artifacts
> would be publicly viewable just like with our Jenkins build output
> today.  Requests for admin assistance would go through the dev@geode
> mailing list.
>
> Thoughts?  As a first step we could run both CI systems side-by-side
> and see how the Concourse approach works for our project.
>
> Thanks,
> Anthony
>
>
> [1] https://builds.apache.org/job/Geode-nightly/
> [2] https://concourse.ci
> [3] https://ci.concourse.ci
> [4] https://bosh.io
>



-- 
Greg Chase

Product team, Pivotal Cloud Foundry Services
https://pivotal.io/platform/services

Pivotal Software
http://www.pivotal.io/

650-215-0477
@GregChase

[DISCUSS] CI improvements

[Anthony Baker]

Hi all,

I’d like to propose the following that we switch our continuous
integration (CI) system from Jenkins [1] to Concourse [2].  I suggest
this because we continue to experience a significant number of
environmental-related test failures.

These issues include CPU interference from other Jenkins jobs on the
same host, running out of disk space, port conflicts, and other
gremlins.  The net effect is that we are only getting 1-2 successful
builds per month.  Certainly not all test failures can be traced back
to environmental issues.  However, internal testing on isolated VM’s
shows a combined success rate of about 3X higher compared to ASF
Jenkins for the same tests.  This is still definitely NotAwesome, but
removing environmental factors will let us focus on stabilizing flaky
tests.

Concourse is an Apache-licensed open source CI system based on
pipelines.  The pipelines are defined in a YML file containing job
definitions—inputs, outputs, resources, and tasks.  A task is simply a
bash script that returns 0/1 for success/failure.  A web UI displays
build status.  Importantly, each job runs inside an isolated
container.  The containers are load-balanced across a pool of workers.
For an example of a build pipeline, see [3] for the pipeline used to
build concourse itself.

A Concourse environment is deployed and managed in cloud environments
through bosh [4].  Pivotal has agreed to donate AWS and/or GCP compute
and storage resources as well as manage the infrastructure.  These
project resources would be available for use by all committers and
community members regardless of corporate affiliations.  Note that
AFAIK there is no explicit requirement to host CI on ASF
infrastructure—unlike for critical project resources such as source
code, mailing lists, and issue tracking.

The source for the pipeline and job scripts would reside within the
geode-* repos.  Geode committers would be able to modify those, same
as with our .travis.yml scripts.  All test results and build artifacts
would be publicly viewable just like with our Jenkins build output
today.  Requests for admin assistance would go through the dev@geode
mailing list.

Thoughts?  As a first step we could run both CI systems side-by-side
and see how the Concourse approach works for our project.

Thanks,
Anthony


[1] https://builds.apache.org/job/Geode-nightly/
[2] https://concourse.ci
[3] https://ci.concourse.ci
[4] https://bosh.io