Re: Unit tests are hanging?

Thanks Owen! On Sat, Aug 10, 2019 at 1:53 PM Owen Nichols wrote: > Done (increased from 10 minutes to 20 minutes for PR pipeline, and changed > from none to 20 minutes for develop pipeline). > > -Owen > > > On Aug 10, 2019, at 11:01 AM, Kirk Lund wrote: > > > > I just saw the Build job exceed t

Re: Unit tests are hanging?

Done (increased from 10 minutes to 20 minutes for PR pipeline, and changed from none to 20 minutes for develop pipeline). -Owen > On Aug 10, 2019, at 11:01 AM, Kirk Lund wrote: > > I just saw the Build job exceed timeout during pmdMain so I think we need > to increase its timeout as well. > ht

Re: Unit tests are hanging?

I just saw the Build job exceed timeout during pmdMain so I think we need to increase its timeout as well. https://concourse.apachegeode-ci.info/builds/85560 On Thu, Aug 8, 2019 at 3:53 PM Ryan McMahon wrote: > I have a PR up for this now. > https://github.com/apache/geode/pull/3900 > > It bump

Re: Unit tests are hanging?

I have a PR up for this now. https://github.com/apache/geode/pull/3900 It bumps the timeout to 20 minutes but also changes the CALL_STACK_TIMEOUT to be 1140 seconds (19 minutes). The latter configuration parameter controls when we declare the task "hung" and dump stacks. We were not dumping stac

Re: Unit tests are hanging?

Looks like we have a general consensus from the community. I'll go ahead and make a PR for the changes. Thanks, Ryan On Thu, Aug 8, 2019 at 11:03 AM Juan José Ramos wrote: > +1 > > On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund wrote: > > > +1 > > > > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith wrot

Re: Unit tests are hanging?

+1 On Thu, Aug 8, 2019 at 6:55 PM Kirk Lund wrote: > +1 > > On Thu, Aug 8, 2019 at 10:14 AM Dan Smith wrote: > > > > With all that, I propose we permanently bump the timeouts on UnitTestX > > jobs > > > across the board (main pipeline, PR pipeline, etc) from 10 to 20 > minutes > > to > > > be m

Re: Unit tests are hanging?

+1 On Thu, Aug 8, 2019 at 10:14 AM Dan Smith wrote: > > With all that, I propose we permanently bump the timeouts on UnitTestX > jobs > > across the board (main pipeline, PR pipeline, etc) from 10 to 20 minutes > to > > be more tolerant of these types of degradations. > > > > +1 > > -Dan >

Re: Unit tests are hanging?

> With all that, I propose we permanently bump the timeouts on UnitTestX jobs > across the board (main pipeline, PR pipeline, etc) from 10 to 20 minutes to > be more tolerant of these types of degradations. > +1 -Dan

Re: Unit tests are hanging?

So we temporarily bumped the timeout from 10 minutes to 2 hours on the UnitTestOpenJDK11 execute_tests Concourse task, originally with the intention of logging into the heavy lifter to debug further. However, after doing that we see that the jobs are all succeeding in roughly the same amount of ti

Re: Unit tests are hanging?

Still trying to identify the cause of the unit test hangs.stay tuned. For other people who might not know this, the reason the entire CI job fails rather than the hanging test is because we don't have any global test-level timeouts, so a hanging test will run up until the Concourse timeout is r

Re: Unit tests are hanging?

Yeah, that test passed on my branch in unit tests and stress tests but for some reason is hanging after the merge to develop. I've pushed an @Ignore for the test that you should pick up. On 8/7/19 10:38 AM, Kirk Lund wrote: Yep, that's the same test I'm seeing in the callstacks of my dunit tgz

Re: Unit tests are hanging?

Yep, that's the same test I'm seeing in the callstacks of my dunit tgz from concourse... Started @ 2019-08-07 07:25:18.494 + 2019-08-07 07:51:25.252 + org.apache.geode.cache30.DistributedMulticastRegionDUnitTest testMulticastAfterReconnect Ended @ 2019-08-07 08:28:16.591 + Thanks for

Re: Unit tests are hanging?

I think the reflection and PowerMock warnings here are probably a red herring. We pulled down the artifacts and found that the DUnit job is hanging due to stuck threads in a newer DUnit test. I am not sure why it isn't failing the test but rather failing the entire job. I believe Bruce Schuchart

Re: Unit tests are hanging?

That makes sense…I’m running 11 and I see them > On Aug 7, 2019, at 1:11 PM, Helena Bales wrote: > > Kirk, as Lynn said in slack, these warnings are showing up with JDK11. So > you'd probably see them locally if you used 11 instead of 8. > > On Wed, Aug 7, 2019, 10:05 AM Kirk Lund wrote: > >>

Re: Unit tests are hanging?

Kirk, as Lynn said in slack, these warnings are showing up with JDK11. So you'd probably see them locally if you used 11 instead of 8. On Wed, Aug 7, 2019, 10:05 AM Kirk Lund wrote: > We previously decided to stop using PowerMock in any Geode tests and remove > it. Ryan and I created https://iss

Re: Unit tests are hanging?

We previously decided to stop using PowerMock in any Geode tests and remove it. Ryan and I created https://issues.apache.org/jira/browse/GEODE-6143 but it's languishing because it takes a lot of time to strip out PowerMock from a test. I don't get any PowerMock warnings locally when I run unit tes

Re: Unit tests are hanging?

I'm on CIO duty today. There have been some network issues over the last 24h and all of the hangs in the CIO dashboard corresponded with network failures. The jobs have been kicked off again, so hopefully we won't see this issues again for a while. Are you consistently seeing these hangs in the PR

Re: Unit tests are hanging?

Yea. I think it might be a red herring, because I am seeing those errors in every run, passing ones two…just double checked on https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UnitTestOpenJDK11/builds/948

Re: Unit tests are hanging?

I don't know if the PowerMock warnings are related but that's the only thing interesting in the output before "timeout exceeded". On Wed, Aug 7, 2019 at 9:43 AM Murtuza Boxwala wrote: > How down know these warnings are related to the builds hanging. When I > was on CIO duty a couple days ago, I

Re: Unit tests are hanging?

How down know these warnings are related to the builds hanging. When I was on CIO duty a couple days ago, I remember seeing this warning in a failing build, but then I looked back at passing builds on saw this as well. > On Aug 7, 2019, at 12:40 PM, Kirk Lund wrote: > > The build is broken in

Unit tests are hanging?

The build is broken in CI right now (for main CI and PRs). The UnitTest jobs are timing out so I assume there's a hang of some sort. The WARNINGs appear to be related to PowerMock and begins with "An illegal reflective access operation" in geode-assembly:test. I'm running unit tests locally and h