So, we need to disable running them in parallel and improve the tearDown to make sure these tests don't leave an orphaned tests behind.
On Tue, Aug 21, 2018 at 11:11 AM, Sai Boorlagadda <sai.boorlaga...@gmail.com > wrote: > jdbc-connector acceptance tests need docker-in-docker (also docker-compose) > to spin up mysql and postgres. > > On Tue, Aug 21, 2018 at 11:04 AM Dan Smith <dsm...@pivotal.io> wrote: > > > Actually, it looks like the problem is that we are *not* using docker > > containers for the acceptance tests. Check this out, in > > gradle/docker.gradle. Since acceptance tests use the default port, this > > means the test are guaranteed to be flaky, especially since we are > running > > them in parallel: > > > > // ACCEPTANCE TEST NEEDS DOCKER-COMPOSE TO WORK WITHIN DOCKER FIRST > > // acceptanceTest.configure(dockerConfig) > > > > I'm not sure what changed that is causing the tests to fail more often > now, > > but maybe a test ordering change? > > > > -Dan > > > > > > > > On Tue, Aug 21, 2018 at 10:52 AM, Kenneth Howe <kh...@pivotal.io> wrote: > > > > > > > > > > > > On Aug 21, 2018, at 10:44 AM, Kirk Lund <kl...@apache.org> wrote: > > > > > > > > GEODE-5590 would seem to imply that GfshRule does not have an > adequate > > > safe > > > > guard? If it spawns a server process which binds to the default > server > > > port > > > > and that process persists after the test then we need better > tearDown. > > > > > > > Yes, that does appear to be the case. The current failures are > apparently > > > due to incomplete > > > teardown between tests within a test class. > > > > > > I am attempting to reproduce the failures on a consistent basis for > > > debugging the problem. > > > > > > > > > > Actually I thought we were using Docker to run each AcceptanceTest in > > > > isolation. Then when the test finishes the Docker instances goes > away. > > > Did > > > > we stop using Docker for these? > > > > > > > > On Tue, Aug 21, 2018 at 10:25 AM, Sai Boorlagadda < > > > sai.boorlaga...@gmail.com > > > >> wrote: > > > > > > > >> DeployWithLargeJarTest & PutCommandWithJsonTest are flaky on > Develop. > > > >> > > > >> DeployWithLargeJarTest - > > > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > > > >> develop/jobs/AcceptanceTest/builds/335 > > > >> PutCommandWithJsonTest - > > > >> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > > > >> develop/jobs/AcceptanceTest/builds/334 > > > >> > > > >> On Tue, Aug 21, 2018 at 10:18 AM Sai Boorlagadda < > > > >> sai.boorlaga...@gmail.com> > > > >> wrote: > > > >> > > > >>> The metrics job themselves will be green (as they complete to > > success) > > > >> but > > > >>> you can expand the get_metrics task output and see that build#20 > > > started > > > >>> reporting these failures, so probably these are due to recent > changes > > > on > > > >>> develop. I believe these metrics are from develop CI test runs. > > > >>> > > > >>> On Tue, Aug 21, 2018 at 10:15 AM Kirk Lund <kl...@apache.org> > wrote: > > > >>> > > > >>>> Those metrics show AcceptanceTests consistently GREEN. Do these > > > metrics > > > >>>> include test failures from pull request precheckin runs like mine? > > Or > > > >> does > > > >>>> it just cover CI test runs? > > > >>>> > > > >>>> On Tue, Aug 21, 2018 at 10:09 AM, Sai Boorlagadda < > > > >>>> sai.boorlaga...@gmail.com > > > >>>>> wrote: > > > >>>> > > > >>>>> Metrics show these started failing recently. > > > >>>>> > > > >>>>> > > > >>>> https://concourse.apachegeode-ci.info/teams/main/pipelines/ > > > >> metrics/jobs/ > > > >>>>> GeodeAcceptanceTestMetrics/builds/20 > > > >>>>> > > > >>>>> On Tue, Aug 21, 2018 at 10:07 AM Kirk Lund <kl...@apache.org> > > wrote: > > > >>>>> > > > >>>>>> Are PutCommandWithJsonTest and DeployWithLargeJarTest known to > be > > > >>>> flaky? > > > >>>>>> > > > >>>>>> My latest pull request failed with these two failures and all I > > did > > > >>>> was > > > >>>>>> extract LocalRegion.validateRegionName and improve unit testing > of > > > >>>>>> RegionNameValidation. No other tests failed for me. > > > >>>>>> > > > >>>>>>> Task :geode-assembly:acceptanceTest > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:619 > > > > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:620 > > > > > > >>>>>> > > > >>>> org.apache.geode.management.internal.cli.commands. > > > >> PutCommandWithJsonTest > > > >>>>>>> putWithJsonString FAILED > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:621 > > > > > > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:622 > > > > > > >>>>>> at > > > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > >>>>>> Method) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:623 > > > > > > >>>>>> at > > > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( > > > >>>>> NativeConstructorAccessorImpl.java:62) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:624 > > > > > > >>>>>> at > > > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > > >>>>> DelegatingConstructorAccessorImpl.java:45) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:625 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. > > > >>>>> awaitIfNecessary(GfshScript.java:117) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:626 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( > > > >>>>> GfshRule.java:135) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:627 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( > > > >>>>> GfshScript.java:106) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:628 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.management.internal.cli.commands. > > > >>>>> PutCommandWithJsonTest.putWithJsonString( > > > PutCommandWithJsonTest.java: > > > >> 55) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:629 > > > > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:630 > > > > > > >>>>>> > > > >>>> org.apache.geode.management.internal.cli.commands. > > > >> DeployWithLargeJarTest > > > >>>>>>> deployLargeSetOfJars FAILED > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:631 > > > > > > >>>>>> org.junit.ComparisonFailure: expected:<[0]> but was:<[1]> > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:632 > > > > > > >>>>>> at > > > >>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > >>>>>> Method) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:633 > > > > > > >>>>>> at > > > >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance( > > > >>>>> NativeConstructorAccessorImpl.java:62) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:634 > > > > > > >>>>>> at > > > >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > > >>>>> DelegatingConstructorAccessorImpl.java:45) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:635 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript. > > > >>>>> awaitIfNecessary(GfshScript.java:117) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:636 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshRule.execute( > > > >>>>> GfshRule.java:135) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:637 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.test.junit.rules.gfsh.GfshScript.execute( > > > >>>>> GfshScript.java:106) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:638 > > > > > > >>>>>> at > > > >>>>>> org.apache.geode.management.internal.cli.commands. > > > >>>>> DeployWithLargeJarTest.deployLargeSetOfJars( > > > >> DeployWithLargeJarTest.java: > > > >>>>> 41) > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:639 > > > > > > >>>>>> <https://concourse.apachegeode-ci.info/builds/ > 19680#L5b60bc1a:640 > > > > > > >>>>>>> Task :geode-assembly:acceptanceTest FAILED > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > >
