Geode tests completed in pipeline with non-zero exit code

[apachegeodeci]

Pipeline results can be found at:

Concourse: 
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/32

Re: Debugging intermittent dunit failures

[Patrick Rhomberg]

Are the AcceptanceTests currently set to fork after every one?  It might be
apples to oranges, but we could try to look at the cost difference there.

On Thu, Dec 14, 2017 at 4:24 PM, Kirk Lund  wrote:

> Thanks Jens! Let us know if you end up running dunit in parallel with
> forkevery 1 or if it's completely serial (one at a time). I think parallel
> with docker with forkevery 1 is going to give us the best results but I'm
> sure you already know that since you're the docker king.
>
> This change would eliminate memory pollution. Any test pollution remaining
> after this should be caused by dunit writing to non-TemporaryFolder
> directories such as current working directory or user's home directory.
>
> On Thu, Dec 14, 2017 at 4:08 PM, Jens Deppe  wrote:
>
> > I'll do that.
> >
> > On Thu, Dec 14, 2017 at 2:02 PM, Kirk Lund  wrote:
> >
> > > Someone needs to try it out. If there are any build-engineer types
> > working
> > > on geode, then my suggestion is for them to try this change and report
> > the
> > > timing difference.
> > >
> > > On Tue, Dec 12, 2017 at 2:22 PM, Alexander Murmann <
> amurm...@pivotal.io>
> > > wrote:
> > >
> > > > Do we have a rough idea how forking every time would impact how long
> > > tests
> > > > run?
> > > >
> > > > On Tue, Dec 12, 2017 at 1:39 PM, Kirk Lund  wrote:
> > > >
> > > > > We should just change to fork every 1 instead of 30. Wasting time
> > > trying
> > > > to
> > > > > debug statics is well... it's a waste of time. We should be focused
> > on
> > > > > other things.
> > > > >
> > > > > On Mon, Dec 11, 2017 at 9:05 PM, Jinmei Liao 
> > > wrote:
> > > > >
> > > > > > It doesn't call as much static methods as
> > JUnit4DistributedTestCase.
> > > > > > tearDownVM,
> > > > > > see MemberStarterRule.after().
> > > > > >
> > > > > > On Mon, Dec 11, 2017 at 4:36 PM, Dan Smith 
> > > wrote:
> > > > > >
> > > > > > > I don't think we are trying to reuse the distributed system  -
> it
> > > > gets
> > > > > > > disconnected after each test. See JUnit4DistributedTestCase.
> > > > > tearDownVM.
> > > > > > >
> > > > > > > Are the new junit rules also cleaning things up?
> > > > > > >
> > > > > > > -Dan
> > > > > > >
> > > > > > > On Mon, Dec 11, 2017 at 4:16 PM, Kirk Lund 
> > > wrote:
> > > > > > >
> > > > > > > > Is there a reason we can't change DistributedTestCase and
> > > > subclasses
> > > > > to
> > > > > > > use
> > > > > > > > TemporaryFolder for all artifacts?
> > > > > > > >
> > > > > > > > We could also disconnectAllFromDS in @AfterClass (or even
> > @After)
> > > > to
> > > > > > get
> > > > > > > > things a bit more separate between dunit test classes.
> > > > > > > >
> > > > > > > > Running dunit tests in parallel is much more important than
> > > trying
> > > > to
> > > > > > > reuse
> > > > > > > > distributed system across multiple dunit tests. The latter
> just
> > > > isn't
> > > > > > > worth
> > > > > > > > the headache and trouble that it causes when static vars or
> > > > constants
> > > > > > or
> > > > > > > > disk artifacts pollute later tests.
> > > > > > > >
> > > > > > > > On Mon, Dec 11, 2017 at 1:42 PM, Dan Smith <
> dsm...@pivotal.io>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > One other thing you can do is look for the below line in
> the
> > > logs
> > > > > of
> > > > > > > your
> > > > > > > > > failure. These are the tests that ran in the same JVM as
> your
> > > > > tests.
> > > > > > > This
> > > > > > > > > won't help if your tests are getting messed up by disk
> > > artifacts
> > > > or
> > > > > > > port
> > > > > > > > > issues, but if it is some JVM state left by a previous test
> > it
> > > > > would
> > > > > > be
> > > > > > > > in
> > > > > > > > > this list.
> > > > > > > > >
> > > > > > > > > Previously run tests: [ClientServerMiscSelectorDUnitTest,
> > > > > > > > > ClientConflationDUnitTest, ReliableMessagingDUnitTest]
> > > > > > > > >
> > > > > > > > > On Mon, Dec 11, 2017 at 1:14 PM, Jens Deppe <
> > > > jensde...@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I've recently debugged various distributed tests which
> fail
> > > as
> > > > a
> > > > > > > result
> > > > > > > > > of
> > > > > > > > > > prior tests not cleaning up enough. It's somewhat painful
> > and
> > > > > this
> > > > > > is
> > > > > > > > my
> > > > > > > > > > usual debug process:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >- Examine the progress.txt file to determine which
> tests
> > > ran
> > > > > > > before
> > > > > > > > my
> > > > > > > > > >failing test.
> > > > > > > > > >- I pick 20-25 of these tests and create a Suite
> > > (including
> > > > my
> > > > > > > > failing
> > > > > > > > > >test) - as these tests may have run in parallel, it's
> > not
> > > > > clear
> > > > > > > > which
> > > > > > > > > > tests
> > > > > > > > > >would 

Re: Debugging intermittent dunit failures

[Kirk Lund]

Thanks Jens! Let us know if you end up running dunit in parallel with
forkevery 1 or if it's completely serial (one at a time). I think parallel
with docker with forkevery 1 is going to give us the best results but I'm
sure you already know that since you're the docker king.

This change would eliminate memory pollution. Any test pollution remaining
after this should be caused by dunit writing to non-TemporaryFolder
directories such as current working directory or user's home directory.

On Thu, Dec 14, 2017 at 4:08 PM, Jens Deppe  wrote:

> I'll do that.
>
> On Thu, Dec 14, 2017 at 2:02 PM, Kirk Lund  wrote:
>
> > Someone needs to try it out. If there are any build-engineer types
> working
> > on geode, then my suggestion is for them to try this change and report
> the
> > timing difference.
> >
> > On Tue, Dec 12, 2017 at 2:22 PM, Alexander Murmann 
> > wrote:
> >
> > > Do we have a rough idea how forking every time would impact how long
> > tests
> > > run?
> > >
> > > On Tue, Dec 12, 2017 at 1:39 PM, Kirk Lund  wrote:
> > >
> > > > We should just change to fork every 1 instead of 30. Wasting time
> > trying
> > > to
> > > > debug statics is well... it's a waste of time. We should be focused
> on
> > > > other things.
> > > >
> > > > On Mon, Dec 11, 2017 at 9:05 PM, Jinmei Liao 
> > wrote:
> > > >
> > > > > It doesn't call as much static methods as
> JUnit4DistributedTestCase.
> > > > > tearDownVM,
> > > > > see MemberStarterRule.after().
> > > > >
> > > > > On Mon, Dec 11, 2017 at 4:36 PM, Dan Smith 
> > wrote:
> > > > >
> > > > > > I don't think we are trying to reuse the distributed system  - it
> > > gets
> > > > > > disconnected after each test. See JUnit4DistributedTestCase.
> > > > tearDownVM.
> > > > > >
> > > > > > Are the new junit rules also cleaning things up?
> > > > > >
> > > > > > -Dan
> > > > > >
> > > > > > On Mon, Dec 11, 2017 at 4:16 PM, Kirk Lund 
> > wrote:
> > > > > >
> > > > > > > Is there a reason we can't change DistributedTestCase and
> > > subclasses
> > > > to
> > > > > > use
> > > > > > > TemporaryFolder for all artifacts?
> > > > > > >
> > > > > > > We could also disconnectAllFromDS in @AfterClass (or even
> @After)
> > > to
> > > > > get
> > > > > > > things a bit more separate between dunit test classes.
> > > > > > >
> > > > > > > Running dunit tests in parallel is much more important than
> > trying
> > > to
> > > > > > reuse
> > > > > > > distributed system across multiple dunit tests. The latter just
> > > isn't
> > > > > > worth
> > > > > > > the headache and trouble that it causes when static vars or
> > > constants
> > > > > or
> > > > > > > disk artifacts pollute later tests.
> > > > > > >
> > > > > > > On Mon, Dec 11, 2017 at 1:42 PM, Dan Smith 
> > > > wrote:
> > > > > > >
> > > > > > > > One other thing you can do is look for the below line in the
> > logs
> > > > of
> > > > > > your
> > > > > > > > failure. These are the tests that ran in the same JVM as your
> > > > tests.
> > > > > > This
> > > > > > > > won't help if your tests are getting messed up by disk
> > artifacts
> > > or
> > > > > > port
> > > > > > > > issues, but if it is some JVM state left by a previous test
> it
> > > > would
> > > > > be
> > > > > > > in
> > > > > > > > this list.
> > > > > > > >
> > > > > > > > Previously run tests: [ClientServerMiscSelectorDUnitTest,
> > > > > > > > ClientConflationDUnitTest, ReliableMessagingDUnitTest]
> > > > > > > >
> > > > > > > > On Mon, Dec 11, 2017 at 1:14 PM, Jens Deppe <
> > > jensde...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I've recently debugged various distributed tests which fail
> > as
> > > a
> > > > > > result
> > > > > > > > of
> > > > > > > > > prior tests not cleaning up enough. It's somewhat painful
> and
> > > > this
> > > > > is
> > > > > > > my
> > > > > > > > > usual debug process:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >- Examine the progress.txt file to determine which tests
> > ran
> > > > > > before
> > > > > > > my
> > > > > > > > >failing test.
> > > > > > > > >- I pick 20-25 of these tests and create a Suite
> > (including
> > > my
> > > > > > > failing
> > > > > > > > >test) - as these tests may have run in parallel, it's
> not
> > > > clear
> > > > > > > which
> > > > > > > > > tests
> > > > > > > > >would have run immediately prior to your test
> > > > > > > > >- Run the whole suite to see if I can get my test to
> fail
> > > > > > > > >- Bisect or manually iterate through the tests to see
> > which
> > > > one
> > > > > is
> > > > > > > > >causing the problem.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The last step is painful, so I've updated SuiteRunner to
> use
> > a
> > > > > > > > 'candidate'
> > > > > > > > > test class and run this class after every other class in
> the
> > > list
> > 

Re: Debugging intermittent dunit failures

[Jens Deppe]

I'll do that.

On Thu, Dec 14, 2017 at 2:02 PM, Kirk Lund  wrote:

> Someone needs to try it out. If there are any build-engineer types working
> on geode, then my suggestion is for them to try this change and report the
> timing difference.
>
> On Tue, Dec 12, 2017 at 2:22 PM, Alexander Murmann 
> wrote:
>
> > Do we have a rough idea how forking every time would impact how long
> tests
> > run?
> >
> > On Tue, Dec 12, 2017 at 1:39 PM, Kirk Lund  wrote:
> >
> > > We should just change to fork every 1 instead of 30. Wasting time
> trying
> > to
> > > debug statics is well... it's a waste of time. We should be focused on
> > > other things.
> > >
> > > On Mon, Dec 11, 2017 at 9:05 PM, Jinmei Liao 
> wrote:
> > >
> > > > It doesn't call as much static methods as JUnit4DistributedTestCase.
> > > > tearDownVM,
> > > > see MemberStarterRule.after().
> > > >
> > > > On Mon, Dec 11, 2017 at 4:36 PM, Dan Smith 
> wrote:
> > > >
> > > > > I don't think we are trying to reuse the distributed system  - it
> > gets
> > > > > disconnected after each test. See JUnit4DistributedTestCase.
> > > tearDownVM.
> > > > >
> > > > > Are the new junit rules also cleaning things up?
> > > > >
> > > > > -Dan
> > > > >
> > > > > On Mon, Dec 11, 2017 at 4:16 PM, Kirk Lund 
> wrote:
> > > > >
> > > > > > Is there a reason we can't change DistributedTestCase and
> > subclasses
> > > to
> > > > > use
> > > > > > TemporaryFolder for all artifacts?
> > > > > >
> > > > > > We could also disconnectAllFromDS in @AfterClass (or even @After)
> > to
> > > > get
> > > > > > things a bit more separate between dunit test classes.
> > > > > >
> > > > > > Running dunit tests in parallel is much more important than
> trying
> > to
> > > > > reuse
> > > > > > distributed system across multiple dunit tests. The latter just
> > isn't
> > > > > worth
> > > > > > the headache and trouble that it causes when static vars or
> > constants
> > > > or
> > > > > > disk artifacts pollute later tests.
> > > > > >
> > > > > > On Mon, Dec 11, 2017 at 1:42 PM, Dan Smith 
> > > wrote:
> > > > > >
> > > > > > > One other thing you can do is look for the below line in the
> logs
> > > of
> > > > > your
> > > > > > > failure. These are the tests that ran in the same JVM as your
> > > tests.
> > > > > This
> > > > > > > won't help if your tests are getting messed up by disk
> artifacts
> > or
> > > > > port
> > > > > > > issues, but if it is some JVM state left by a previous test it
> > > would
> > > > be
> > > > > > in
> > > > > > > this list.
> > > > > > >
> > > > > > > Previously run tests: [ClientServerMiscSelectorDUnitTest,
> > > > > > > ClientConflationDUnitTest, ReliableMessagingDUnitTest]
> > > > > > >
> > > > > > > On Mon, Dec 11, 2017 at 1:14 PM, Jens Deppe <
> > jensde...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > I've recently debugged various distributed tests which fail
> as
> > a
> > > > > result
> > > > > > > of
> > > > > > > > prior tests not cleaning up enough. It's somewhat painful and
> > > this
> > > > is
> > > > > > my
> > > > > > > > usual debug process:
> > > > > > > >
> > > > > > > >
> > > > > > > >- Examine the progress.txt file to determine which tests
> ran
> > > > > before
> > > > > > my
> > > > > > > >failing test.
> > > > > > > >- I pick 20-25 of these tests and create a Suite
> (including
> > my
> > > > > > failing
> > > > > > > >test) - as these tests may have run in parallel, it's not
> > > clear
> > > > > > which
> > > > > > > > tests
> > > > > > > >would have run immediately prior to your test
> > > > > > > >- Run the whole suite to see if I can get my test to fail
> > > > > > > >- Bisect or manually iterate through the tests to see
> which
> > > one
> > > > is
> > > > > > > >causing the problem.
> > > > > > > >
> > > > > > > >
> > > > > > > > The last step is painful, so I've updated SuiteRunner to use
> a
> > > > > > > 'candidate'
> > > > > > > > test class and run this class after every other class in the
> > list
> > > > of
> > > > > > > > SuiteClasses. For example:
> > > > > > > >
> > > > > > > > @Suite.SuiteClasses(value = {
> > > > > > > > org.apache.geode.cache.snapshot.
> > SnapshotByteArrayDUnitTest.
> > > > > class,
> > > > > > > > org.apache.geode.cache.query.dunit.
> > > > > QueryDataInconsistencyDUnitTes
> > > > > > > > t.class,
> > > > > > > > org.apache.geode.cache.query.internal.index.
> > > > > > > > MultiIndexCreationDUnitTest.class,
> > > > > > > > })
> > > > > > > >  @SuiteRunner.Candidate(org.apache.geode.management.
> > > > > > > > internal.configuration.ClusterConfigDistributionDUnit
> > Test.class)
> > > > > > > > @RunWith(SuiteRunner.class)
> > > > > > > > public class DebugSuite {
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > > > > > > The Candidate is optional, but this would run the following
> > > tests:
> 

Re: Debugging intermittent dunit failures

[Kirk Lund]

Someone needs to try it out. If there are any build-engineer types working
on geode, then my suggestion is for them to try this change and report the
timing difference.

On Tue, Dec 12, 2017 at 2:22 PM, Alexander Murmann 
wrote:

> Do we have a rough idea how forking every time would impact how long tests
> run?
>
> On Tue, Dec 12, 2017 at 1:39 PM, Kirk Lund  wrote:
>
> > We should just change to fork every 1 instead of 30. Wasting time trying
> to
> > debug statics is well... it's a waste of time. We should be focused on
> > other things.
> >
> > On Mon, Dec 11, 2017 at 9:05 PM, Jinmei Liao  wrote:
> >
> > > It doesn't call as much static methods as JUnit4DistributedTestCase.
> > > tearDownVM,
> > > see MemberStarterRule.after().
> > >
> > > On Mon, Dec 11, 2017 at 4:36 PM, Dan Smith  wrote:
> > >
> > > > I don't think we are trying to reuse the distributed system  - it
> gets
> > > > disconnected after each test. See JUnit4DistributedTestCase.
> > tearDownVM.
> > > >
> > > > Are the new junit rules also cleaning things up?
> > > >
> > > > -Dan
> > > >
> > > > On Mon, Dec 11, 2017 at 4:16 PM, Kirk Lund  wrote:
> > > >
> > > > > Is there a reason we can't change DistributedTestCase and
> subclasses
> > to
> > > > use
> > > > > TemporaryFolder for all artifacts?
> > > > >
> > > > > We could also disconnectAllFromDS in @AfterClass (or even @After)
> to
> > > get
> > > > > things a bit more separate between dunit test classes.
> > > > >
> > > > > Running dunit tests in parallel is much more important than trying
> to
> > > > reuse
> > > > > distributed system across multiple dunit tests. The latter just
> isn't
> > > > worth
> > > > > the headache and trouble that it causes when static vars or
> constants
> > > or
> > > > > disk artifacts pollute later tests.
> > > > >
> > > > > On Mon, Dec 11, 2017 at 1:42 PM, Dan Smith 
> > wrote:
> > > > >
> > > > > > One other thing you can do is look for the below line in the logs
> > of
> > > > your
> > > > > > failure. These are the tests that ran in the same JVM as your
> > tests.
> > > > This
> > > > > > won't help if your tests are getting messed up by disk artifacts
> or
> > > > port
> > > > > > issues, but if it is some JVM state left by a previous test it
> > would
> > > be
> > > > > in
> > > > > > this list.
> > > > > >
> > > > > > Previously run tests: [ClientServerMiscSelectorDUnitTest,
> > > > > > ClientConflationDUnitTest, ReliableMessagingDUnitTest]
> > > > > >
> > > > > > On Mon, Dec 11, 2017 at 1:14 PM, Jens Deppe <
> jensde...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > I've recently debugged various distributed tests which fail as
> a
> > > > result
> > > > > > of
> > > > > > > prior tests not cleaning up enough. It's somewhat painful and
> > this
> > > is
> > > > > my
> > > > > > > usual debug process:
> > > > > > >
> > > > > > >
> > > > > > >- Examine the progress.txt file to determine which tests ran
> > > > before
> > > > > my
> > > > > > >failing test.
> > > > > > >- I pick 20-25 of these tests and create a Suite (including
> my
> > > > > failing
> > > > > > >test) - as these tests may have run in parallel, it's not
> > clear
> > > > > which
> > > > > > > tests
> > > > > > >would have run immediately prior to your test
> > > > > > >- Run the whole suite to see if I can get my test to fail
> > > > > > >- Bisect or manually iterate through the tests to see which
> > one
> > > is
> > > > > > >causing the problem.
> > > > > > >
> > > > > > >
> > > > > > > The last step is painful, so I've updated SuiteRunner to use a
> > > > > > 'candidate'
> > > > > > > test class and run this class after every other class in the
> list
> > > of
> > > > > > > SuiteClasses. For example:
> > > > > > >
> > > > > > > @Suite.SuiteClasses(value = {
> > > > > > > org.apache.geode.cache.snapshot.
> SnapshotByteArrayDUnitTest.
> > > > class,
> > > > > > > org.apache.geode.cache.query.dunit.
> > > > QueryDataInconsistencyDUnitTes
> > > > > > > t.class,
> > > > > > > org.apache.geode.cache.query.internal.index.
> > > > > > > MultiIndexCreationDUnitTest.class,
> > > > > > > })
> > > > > > >  @SuiteRunner.Candidate(org.apache.geode.management.
> > > > > > > internal.configuration.ClusterConfigDistributionDUnit
> Test.class)
> > > > > > > @RunWith(SuiteRunner.class)
> > > > > > > public class DebugSuite {
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > The Candidate is optional, but this would run the following
> > tests:
> > > > > > >
> > > > > > > - SnapshotByteArrayDUnitTest
> > > > > > > - *ClusterConfigDistributionDUnitTest*
> > > > > > > - QueryDataInconsistencyDUnitTest
> > > > > > > - *ClusterConfigDistributionDUnitTest*
> > > > > > > - MultiIndexCreationDUnitTest
> > > > > > > - *ClusterConfigDistributionDUnitTest*
> > > > > > >
> > > > > > > --Jens
> > > > > > >
> > > > > >
> > > > >
> > > >
> > 

Geode tests completed in pipeline with non-zero exit code

[apachegeodeci]

Pipeline results can be found at:

Concourse: 
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/30

how to get a good commit comment when using gitbox to merge a pull request

[Darrel Schneider]

I noticed that the git log message for a pull request I just merged was
lacking what I expected it to pick up from the single revision in the pull
request. It ended up with a message like this:

Merge pull request #1165 from dschneider-pivotal/feature/GEODE-4091



GEODE-4091: add ThreadFactory for evictor



The threads created to do background eviction scanning

will now be named "LRUListWithAsyncSortingThreadNNN".

The will now be daemon threads.

They now have an unhandled exception handler.

Their group is named "LRUListWithAsyncSorting Threads".


I wanted it to look like this:

GEODE-4091: add ThreadFactory for evictor



The threads created to do background eviction scanning

will now be named "LRUListWithAsyncSortingThreadNNN".

The will now be daemon threads.

They now have an unhandled exception handler.

Their group is named "LRUListWithAsyncSorting Threads".

I added the last five lines to it myself because I saw they were missing.
Did my adding those mess this up?
Am I correct that we don't want that first line ("Merge pull request #...")?

Thanks for any help

Re: Permissions to Add Content in the Wiki

[Juan José Ramos]

Thanks Dan!.
Cheers.

On Wed, Dec 13, 2017 at 2:30 PM, Dan Smith  wrote:

> Hi Juan,
>
> You should have permissions now!
>
> Thanks,
> -Dan
>
> On Wed, Dec 13, 2017 at 3:33 AM, Juan José Ramos 
> wrote:
>
> > Hello team,
> >
> > Can I get permissions to create content on the wiki?. My username is
> > *jujoramos*.
> > Best regards.
> >
> > --
> > Juan José Ramos Cassella
> > Senior Technical Support Engineer
> > Email: jra...@pivotal.io
> > Office#: +353 21 4238611
> > Mobile#: +353 87 2074066
> > After Hours Contact#: +1 877 477 2269
> > Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
> > How to upload artifacts:
> > https://support.pivotal.io/hc/en-us/articles/204369073
> > How to escalate a ticket:
> > https://support.pivotal.io/hc/en-us/articles/203809556
> >
> > [image: support]  [image: twitter]
> >  [image: linkedin]
> >  [image: facebook]
> >  [image: google plus]
> >  [image: youtube]
> >  eSPScpj2J50ErtzR9ANSzv3kl>
> >
>



-- 
Juan José Ramos Cassella
Senior Technical Support Engineer
Email: jra...@pivotal.io
Office#: +353 21 4238611
Mobile#: +353 87 2074066
After Hours Contact#: +1 877 477 2269
Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
How to upload artifacts:
https://support.pivotal.io/hc/en-us/articles/204369073
How to escalate a ticket:
https://support.pivotal.io/hc/en-us/articles/203809556

[image: support]  [image: twitter]
 [image: linkedin]
 [image: facebook]
 [image: google plus]
 [image: youtube]