[
https://issues.apache.org/jira/browse/SOLR-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16305639#comment-16305639
]
Steve Rowe commented on SOLR-11258:
-----------------------------------
This branch_7x nightly failure from my Jenkins reproduced for me 5/5 iterations:
{noformat}
Checking out Revision 89344ea4c5c1c1f7da2797f0e724574751723976
(refs/remotes/origin/branch_7x)
[...]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=ChaosMonkeySafeLeaderWithPullReplicasTest -Dtests.method=test
-Dtests.seed=DD16DE67F3F5708A -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true
-Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt
-Dtests.locale=fr -Dtests.timezone=US/Indiana-Starke -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] FAILURE 104s J6 | ChaosMonkeySafeLeaderWithPullReplicasTest.test
<<<
[junit4] > Throwable #1: java.lang.AssertionError: The Monkey ran for
over 60 seconds and no jetties were stopped - this is worth investigating!
[junit4] > at
__randomizedtesting.SeedInfo.seed([DD16DE67F3F5708A:5542E1BD5D091D72]:0)
[junit4] > at
org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:589)
[junit4] > at
org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:175)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
[junit4] > at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[...]
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70):
{rnd_b=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
a_t=PostingsFormat(name=Memory),
id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))},
docValues:{_version_=DocValuesFormat(name=Asserting)},
maxPointsInLeafNode=590, maxMBSortInHeap=5.448183350014092,
sim=RandomSimilarity(queryNorm=false): {}, locale=fr, timezone=US/Indiana-Starke
[junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation
1.8.0_151 (64-bit)/cpus=16,threads=6,free=170276048,total=524288000
{noformat}
> ChaosMonkeySafeLeaderWithPullReplicasTest fails a lot & reproducibly: The
> Monkey ran for over 45 seconds and no jetties were stopped - this is worth
> investigating!
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-11258
> URL: https://issues.apache.org/jira/browse/SOLR-11258
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Hoss Man
>
> Between June21 & Aug18, there have been 18 failures like this...
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=ChaosMonkeySafeLeaderWithPullReplicasTest -Dtests.method=test
> -Dtests.seed=7669B63E9E4D1685 -Dtests.nightly=true -Dtests.slow=true
> -Dtests.locale=pa-Guru -Dtests.timezone=Europe/Podgorica -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> [junit4] FAILURE 82.4s | ChaosMonkeySafeLeaderWithPullReplicasTest.test <<<
> [junit4] > Throwable #1: java.lang.AssertionError: The Monkey ran for
> over 45 seconds and no jetties were stopped - this is worth investigating!
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([7669B63E9E4D1685:FE3D89E430B17B7D]:0)
> [junit4] > at
> org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:587)
> [junit4] > at
> org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:174)
> [junit4] > at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
> [junit4] > at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In my own testing, when these failures happen, the seeds reproduce -
> suggesting the problem is logic flaw in the test that can can happen by
> chance.
> Perhaps the ChaosMonkey needs to be changed to get more aggressive about
> stopping nodes bsaed on how long it's been since hte last time it stopped a
> node?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
