Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20969/
Java: 64bit/jdk-10-ea+32 -XX:-UseCompressedOops -XX:+UseG1GC
3 tests failed.
FAILED:
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
Error Message:
3 threads leaked from SUITE scope at
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest: 1)
Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/java.lang.Thread.sleep(Native Method) at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051) 2)
Thread[id=14773, name=Connection evictor, state=TIMED_WAITING,
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at
java.base@10-ea/java.lang.Thread.sleep(Native Method) at
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@10-ea/java.lang.Thread.run(Thread.java:844) 3)
Thread[id=14775,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-EventThread,
state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/jdk.internal.misc.Unsafe.park(Native Method) at
java.base@10-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at
java.base@10-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2074)
at
java.base@10-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
at
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE
scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
1) Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/java.lang.Thread.sleep(Native Method)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
2) Thread[id=14773, name=Connection evictor, state=TIMED_WAITING,
group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/java.lang.Thread.sleep(Native Method)
at
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@10-ea/java.lang.Thread.run(Thread.java:844)
3) Thread[id=14775,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-EventThread,
state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/jdk.internal.misc.Unsafe.park(Native Method)
at
java.base@10-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at
java.base@10-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2074)
at
java.base@10-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
at
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
at __randomizedtesting.SeedInfo.seed([AC6E4B5CCD8A9F42]:0)
FAILED:
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
Error Message:
There are still zombie threads that couldn't be terminated: 1)
Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/java.lang.Thread.sleep(Native Method) at
app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
threads that couldn't be terminated:
1) Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@10-ea/java.lang.Thread.sleep(Native Method)
at
app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
at __randomizedtesting.SeedInfo.seed([AC6E4B5CCD8A9F42]:0)
FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
Error Message:
shard2 is not consistent. Got 2709 from
http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41 (previous client)
and got 3104 from http://127.0.0.1:45793/h_uz/s/collection1_shard2_replica_n45
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 2709 from
http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41 (previous client)
and got 3104 from http://127.0.0.1:45793/h_uz/s/collection1_shard2_replica_n45
at
__randomizedtesting.SeedInfo.seed([AC6E4B5CCD8A9F42:243A74866376F2BA]:0)
at org.junit.Assert.fail(Assert.java:93)
at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1322)
at
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:274)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 13248 lines...]
[junit4] Suite:
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
[junit4] 2> Creating dataDir:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/init-core-data-001
[junit4] 2> 1406955 WARN
(SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[AC6E4B5CCD8A9F42]-worker)
[ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 1406955 INFO
(SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[AC6E4B5CCD8A9F42]-worker)
[ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1406956 INFO
(SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[AC6E4B5CCD8A9F42]-worker)
[ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via:
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="https://issues.apache.org/jira/browse/SOLR-5776";)
[junit4] 2> 1406956 INFO
(SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[AC6E4B5CCD8A9F42]-worker)
[ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1406956 INFO
(SUITE-ChaosMonkeyNothingIsSafeWithPullReplicasTest-seed#[AC6E4B5CCD8A9F42]-worker)
[ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system
property: /h_uz/s
[junit4] 2> 1406956 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ChaosMonkeyNothingIsSafeWithPullReplicasTest Starting
ChaosMonkey test with 2 shards and 6 nodes
[junit4] 2> 1406957 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1406958 INFO (Thread-3469) [ ] o.a.s.c.ZkTestServer client
port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1406958 INFO (Thread-3469) [ ] o.a.s.c.ZkTestServer
Starting server
[junit4] 2> 1406959 ERROR (Thread-3469) [ ] o.a.z.s.ZooKeeperServer
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action
on ERROR or SHUTDOWN server state changes
[junit4] 2> 1407058 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkTestServer start zk server on port:40871
[junit4] 2> 1407069 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
to /configs/conf1/solrconfig.xml
[junit4] 2> 1407070 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
to /configs/conf1/schema.xml
[junit4] 2> 1407071 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1407072 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
to /configs/conf1/stopwords.txt
[junit4] 2> 1407073 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
to /configs/conf1/protwords.txt
[junit4] 2> 1407074 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
to /configs/conf1/currency.xml
[junit4] 2> 1407075 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
to /configs/conf1/enumsConfig.xml
[junit4] 2> 1407076 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1407077 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1407077 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
to /configs/conf1/old_synonyms.txt
[junit4] 2> 1407078 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractZkTestCase put
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
to /configs/conf1/synonyms.txt
[junit4] 2> 1407079 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless
explicitly asked otherwise
[junit4] 2> 1407160 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1407160 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@7c3b68f7{/h_uz/s,null,AVAILABLE}
[junit4] 2> 1407161 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.AbstractConnector Started
ServerConnector@419b02fe{HTTP/1.1,[http/1.1]}{127.0.0.1:44417}
[junit4] 2> 1407161 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server Started @1408545ms
[junit4] 2> 1407161 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/h_uz/s,
solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/tempDir-001/control/data,
hostPort=44417,
coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/control-001/cores,
replicaType=NRT}
[junit4] 2> 1407162 ERROR
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging
may be missing or incomplete.
[junit4] 2> 1407162 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™
version 8.0.0
[junit4] 2> 1407162 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on
port null
[junit4] 2> 1407162 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1407162 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-11-22T05:13:24.829286Z
[junit4] 2> 1407163 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1407163 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Loading container configuration from
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/control-001/solr.xml
[junit4] 2> 1407165 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1407165 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1407165 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1407167 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40871/solr
[junit4] 2> 1407190 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1407190 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.OverseerElectionContext I am going to be the leader
127.0.0.1:44417_h_uz%2Fs
[junit4] 2> 1407190 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer
(id=99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000) starting
[junit4] 2> 1407193 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:44417_h_uz%2Fs
[junit4] 2> 1407194 INFO (zkCallback-2259-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1407216 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1407222 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1407222 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1407223 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/control-001/cores
[junit4] 2> 1407233 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1407234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at
127.0.0.1:40871/solr ready
[junit4] 2> 1407235 INFO (qtp753744363-14736) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44417_h_uz%252Fs&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1407236 INFO (OverseerThreadFactory-7349-thread-1) [ ]
o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 1407339 INFO (qtp753744363-14738) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1407339 INFO (qtp753744363-14738) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 1407441 INFO (zkCallback-2259-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1408362 INFO (qtp753744363-14738) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1408376 INFO (qtp753744363-14738) [ ] o.a.s.s.IndexSchema
[control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 1408423 INFO (qtp753744363-14738) [ ] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1408429 INFO (qtp753744363-14738) [ ]
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1'
using configuration from collection control_collection, trusted=true
[junit4] 2> 1408430 INFO (qtp753744363-14738) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.control_collection.shard1.replica_n1' (registry
'solr.core.control_collection.shard1.replica_n1') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1408430 INFO (qtp753744363-14738) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 1408430 INFO (qtp753744363-14738) [ ] o.a.s.c.SolrCore
[[control_collection_shard1_replica_n1] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/control-001/cores/control_collection_shard1_replica_n1],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 1408432 INFO (qtp753744363-14738) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=1.0]
[junit4] 2> 1408450 WARN (qtp753744363-14738) [ ]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 1408469 INFO (qtp753744363-14738) [ ]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 1408469 INFO (qtp753744363-14738) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1408477 INFO (qtp753744363-14738) [ ]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1408477 INFO (qtp753744363-14738) [ ]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1408478 INFO (qtp753744363-14738) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.6828181195476832]
[junit4] 2> 1408478 INFO (qtp753744363-14738) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@18637cb8[control_collection_shard1_replica_n1] main]
[junit4] 2> 1408479 INFO (qtp753744363-14738) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 1408479 INFO (qtp753744363-14738) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1408480 INFO (qtp753744363-14738) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1408481 INFO (searcherExecutor-7352-thread-1) [ ]
o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher
Searcher@18637cb8[control_collection_shard1_replica_n1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1408481 INFO (qtp753744363-14738) [ ] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1584741855944245248
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ] o.a.s.c.SyncStrategy
Sync replicas to
http://127.0.0.1:44417/h_uz/s/control_collection_shard1_replica_n1/
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ] o.a.s.c.SyncStrategy
Sync Success - now sync replicas to me
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ] o.a.s.c.SyncStrategy
http://127.0.0.1:44417/h_uz/s/control_collection_shard1_replica_n1/ has no
replicas
[junit4] 2> 1408484 INFO (qtp753744363-14738) [ ]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 1408485 INFO (qtp753744363-14738) [ ]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:44417/h_uz/s/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 1408586 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1408636 INFO (qtp753744363-14738) [ ] o.a.s.c.ZkController
I am the leader, no recovery necessary
[junit4] 2> 1408638 INFO (qtp753744363-14738) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1299
[junit4] 2> 1408641 INFO (qtp753744363-14736) [ ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 1408739 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/control_collection/state.json] for collection
[control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1409237 INFO
(OverseerCollectionConfigSetProcessor-99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1409642 INFO (qtp753744363-14736) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:44417_h_uz%252Fs&wt=javabin&version=2}
status=0 QTime=2406
[junit4] 2> 1409647 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1409649 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at
127.0.0.1:40871/solr ready
[junit4] 2> 1409649 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause
connection loss:false
[junit4] 2> 1409650 INFO (qtp753744363-14736) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1409653 INFO (OverseerThreadFactory-7349-thread-2) [ ]
o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1409654 WARN (OverseerThreadFactory-7349-thread-2) [ ]
o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1)
without cores.
[junit4] 2> 1409859 INFO (qtp753744363-14736) [ ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 1409859 INFO (qtp753744363-14736) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2}
status=0 QTime=208
[junit4] 2> 1409944 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001
of type NRT
[junit4] 2> 1409945 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@bd1a982{/h_uz/s,null,AVAILABLE}
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.AbstractConnector Started
ServerConnector@42a1692e{HTTP/1.1,[http/1.1]}{127.0.0.1:45489}
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server Started @1411329ms
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/h_uz/s,
solrconfig=solrconfig.xml,
solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/tempDir-001/jetty1,
hostPort=45489,
coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001/cores,
replicaType=NRT}
[junit4] 2> 1409946 ERROR
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging
may be missing or incomplete.
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™
version 8.0.0
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on
port null
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1409946 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-11-22T05:13:27.613892Z
[junit4] 2> 1409948 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1409948 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Loading container configuration from
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001/solr.xml
[junit4] 2> 1409950 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1409950 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1409950 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1409952 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40871/solr
[junit4] 2> 1409956 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1409957 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1409957 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:45489_h_uz%2Fs
[junit4] 2> 1409958 INFO (zkCallback-2267-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1409958 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1409958 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1409984 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1409990 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1409990 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1409991 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001/cores
[junit4] 2> 1410003 INFO (qtp753744363-14737) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:45489_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1410005 INFO
(OverseerCollectionConfigSetProcessor-99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1410006 INFO (OverseerThreadFactory-7349-thread-3) [ ]
o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:45489_h_uz%2Fs for creating new
replica
[junit4] 2> 1410007 INFO (qtp884502929-14783) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1410007 INFO (qtp884502929-14783) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 1410109 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 1411033 INFO (qtp884502929-14783) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1411047 INFO (qtp884502929-14783) [ ] o.a.s.s.IndexSchema
[collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 1411105 INFO (qtp884502929-14783) [ ] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1411111 INFO (qtp884502929-14783) [ ]
o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using
configuration from collection collection1, trusted=true
[junit4] 2> 1411111 INFO (qtp884502929-14783) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard2.replica_n41' (registry
'solr.core.collection1.shard2.replica_n41') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1411111 INFO (qtp884502929-14783) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 1411111 INFO (qtp884502929-14783) [ ] o.a.s.c.SolrCore
[[collection1_shard2_replica_n41] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001/cores/collection1_shard2_replica_n41],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 1411113 INFO (qtp884502929-14783) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=1.0]
[junit4] 2> 1411127 WARN (qtp884502929-14783) [ ]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 1411147 INFO (qtp884502929-14783) [ ]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 1411147 INFO (qtp884502929-14783) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1411148 INFO (qtp884502929-14783) [ ]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1411148 INFO (qtp884502929-14783) [ ]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1411148 INFO (qtp884502929-14783) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.6828181195476832]
[junit4] 2> 1411149 INFO (qtp884502929-14783) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@682201af[collection1_shard2_replica_n41] main]
[junit4] 2> 1411149 INFO (qtp884502929-14783) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 1411149 INFO (qtp884502929-14783) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1411150 INFO (qtp884502929-14783) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1411150 INFO (searcherExecutor-7363-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher
Searcher@682201af[collection1_shard2_replica_n41]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1411150 INFO (qtp884502929-14783) [ ] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1584741858742894592
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ] o.a.s.c.SyncStrategy
Sync replicas to http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41/
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ] o.a.s.c.SyncStrategy
Sync Success - now sync replicas to me
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ] o.a.s.c.SyncStrategy
http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41/ has no replicas
[junit4] 2> 1411153 INFO (qtp884502929-14783) [ ]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 1411154 INFO (qtp884502929-14783) [ ]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41/ shard2
[junit4] 2> 1411255 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 1411305 INFO (qtp884502929-14783) [ ] o.a.s.c.ZkController
I am the leader, no recovery necessary
[junit4] 2> 1411307 INFO (qtp884502929-14783) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1300
[junit4] 2> 1411310 INFO (qtp753744363-14737) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={node=127.0.0.1:45489_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2}
status=0 QTime=1306
[junit4] 2> 1411393 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001
of type NRT
[junit4] 2> 1411393 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@547ca425{/h_uz/s,null,AVAILABLE}
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.AbstractConnector Started
ServerConnector@235ffd14{HTTP/1.1,[http/1.1]}{127.0.0.1:37673}
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server Started @1412777ms
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/h_uz/s,
solrconfig=solrconfig.xml,
solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/tempDir-001/jetty2,
hostPort=37673,
coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001/cores}
[junit4] 2> 1411394 ERROR
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging
may be missing or incomplete.
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™
version 8.0.0
[junit4] 2> 1411394 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on
port null
[junit4] 2> 1411395 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1411395 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-11-22T05:13:29.062043Z
[junit4] 2> 1411396 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1411396 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Loading container configuration from
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001/solr.xml
[junit4] 2> 1411397 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1411398 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1411398 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1411399 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40871/solr
[junit4] 2> 1411403 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1411404 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1411404 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:37673_h_uz%2Fs
[junit4] 2> 1411404 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1411404 INFO (zkCallback-2267-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1411405 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1411404 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1411434 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1411441 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1411441 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1411442 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001/cores
[junit4] 2> 1411460 INFO (qtp417876687-14812) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:37673_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1411462 INFO
(OverseerCollectionConfigSetProcessor-99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1411462 INFO (OverseerThreadFactory-7349-thread-4) [ ]
o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:37673_h_uz%2Fs for creating new
replica
[junit4] 2> 1411463 INFO (qtp417876687-14814) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1411463 INFO (qtp417876687-14814) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 1411566 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 1411566 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 1412485 INFO (qtp417876687-14814) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1412500 INFO (qtp417876687-14814) [ ] o.a.s.s.IndexSchema
[collection1_shard1_replica_n43] Schema name=test
[junit4] 2> 1412544 INFO (qtp417876687-14814) [ ] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1412549 INFO (qtp417876687-14814) [ ]
o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using
configuration from collection collection1, trusted=true
[junit4] 2> 1412550 INFO (qtp417876687-14814) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard1.replica_n43' (registry
'solr.core.collection1.shard1.replica_n43') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1412550 INFO (qtp417876687-14814) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 1412550 INFO (qtp417876687-14814) [ ] o.a.s.c.SolrCore
[[collection1_shard1_replica_n43] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001/cores/collection1_shard1_replica_n43],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
[junit4] 2> 1412552 INFO (qtp417876687-14814) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=1.0]
[junit4] 2> 1412566 WARN (qtp417876687-14814) [ ]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 1412589 INFO (qtp417876687-14814) [ ]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 1412589 INFO (qtp417876687-14814) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1412597 INFO (qtp417876687-14814) [ ]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1412597 INFO (qtp417876687-14814) [ ]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1412598 INFO (qtp417876687-14814) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.6828181195476832]
[junit4] 2> 1412598 INFO (qtp417876687-14814) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@3caf1b8[collection1_shard1_replica_n43] main]
[junit4] 2> 1412599 INFO (qtp417876687-14814) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 1412599 INFO (qtp417876687-14814) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1412599 INFO (qtp417876687-14814) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1412600 INFO (searcherExecutor-7374-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher
Searcher@3caf1b8[collection1_shard1_replica_n43]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1412600 INFO (qtp417876687-14814) [ ] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1584741860263329792
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ] o.a.s.c.SyncStrategy
Sync replicas to http://127.0.0.1:37673/h_uz/s/collection1_shard1_replica_n43/
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ] o.a.s.c.SyncStrategy
Sync Success - now sync replicas to me
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ] o.a.s.c.SyncStrategy
http://127.0.0.1:37673/h_uz/s/collection1_shard1_replica_n43/ has no replicas
[junit4] 2> 1412602 INFO (qtp417876687-14814) [ ]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 1412603 INFO (qtp417876687-14814) [ ]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:37673/h_uz/s/collection1_shard1_replica_n43/ shard1
[junit4] 2> 1412704 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 1412704 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [3])
[junit4] 2> 1412754 INFO (qtp417876687-14814) [ ] o.a.s.c.ZkController
I am the leader, no recovery necessary
[junit4] 2> 1412755 INFO (qtp417876687-14814) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1292
[junit4] 2> 1412757 INFO (qtp417876687-14812) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={node=127.0.0.1:37673_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2}
status=0 QTime=1296
[junit4] 2> 1412823 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001
of type NRT
[junit4] 2> 1412823 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@64855488{/h_uz/s,null,AVAILABLE}
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.AbstractConnector Started
ServerConnector@4d902fd0{HTTP/1.1,[http/1.1]}{127.0.0.1:45793}
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server Started @1414207ms
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/h_uz/s,
solrconfig=solrconfig.xml,
solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/tempDir-001/jetty3,
hostPort=45793,
coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001/cores}
[junit4] 2> 1412824 ERROR
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging
may be missing or incomplete.
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™
version 8.0.0
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on
port null
[junit4] 2> 1412824 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1412825 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-11-22T05:13:30.492003Z
[junit4] 2> 1412826 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1412826 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Loading container configuration from
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001/solr.xml
[junit4] 2> 1412827 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1412827 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1412828 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1412829 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40871/solr
[junit4] 2> 1412834 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1412834 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1412835 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:45793_h_uz%2Fs
[junit4] 2> 1412835 INFO (zkCallback-2267-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1412838 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1412835 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1412836 INFO (zkCallback-2284-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1412836 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1412860 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1412867 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1412867 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1412868 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001/cores
[junit4] 2> 1412892 INFO (qtp417876687-14813) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:45793_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1412894 INFO
(OverseerCollectionConfigSetProcessor-99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1412894 INFO (OverseerThreadFactory-7349-thread-5) [ ]
o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:45793_h_uz%2Fs for creating new
replica
[junit4] 2> 1412895 INFO (qtp693710932-14841) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1412895 INFO (qtp693710932-14841) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 1412997 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 1412997 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 1412997 INFO (zkCallback-2284-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [4])
[junit4] 2> 1413960 INFO (qtp693710932-14841) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1413975 INFO (qtp693710932-14841) [ ] o.a.s.s.IndexSchema
[collection1_shard2_replica_n45] Schema name=test
[junit4] 2> 1414032 INFO (qtp693710932-14841) [ ] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1414038 INFO (qtp693710932-14841) [ ]
o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using
configuration from collection collection1, trusted=true
[junit4] 2> 1414039 INFO (qtp693710932-14841) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard2.replica_n45' (registry
'solr.core.collection1.shard2.replica_n45') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1414039 INFO (qtp693710932-14841) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 1414039 INFO (qtp693710932-14841) [ ] o.a.s.c.SolrCore
[[collection1_shard2_replica_n45] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001/cores/collection1_shard2_replica_n45],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
[junit4] 2> 1414040 INFO (qtp693710932-14841) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=1.0]
[junit4] 2> 1414120 WARN (qtp693710932-14841) [ ]
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type =
requestHandler,name = /dump,class = DumpRequestHandler,attributes =
{initParams=a, name=/dump, class=DumpRequestHandler},args =
{defaults={a=A,b=B}}}
[junit4] 2> 1414163 INFO (qtp693710932-14841) [ ]
o.a.s.u.UpdateHandler Using UpdateLog implementation:
org.apache.solr.update.UpdateLog
[junit4] 2> 1414163 INFO (qtp693710932-14841) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100
maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1414164 INFO (qtp693710932-14841) [ ]
o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1414164 INFO (qtp693710932-14841) [ ]
o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1414165 INFO (qtp693710932-14841) [ ]
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648,
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
noCFSRatio=0.6828181195476832]
[junit4] 2> 1414165 INFO (qtp693710932-14841) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@1667a899[collection1_shard2_replica_n45] main]
[junit4] 2> 1414166 INFO (qtp693710932-14841) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf1
[junit4] 2> 1414167 INFO (qtp693710932-14841) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1414167 INFO (qtp693710932-14841) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1414168 INFO (searcherExecutor-7385-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher
Searcher@1667a899[collection1_shard2_replica_n45]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1414168 INFO (qtp693710932-14841) [ ] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1584741861907496960
[junit4] 2> 1414170 INFO (qtp693710932-14841) [ ] o.a.s.c.ZkController
Core needs to recover:collection1_shard2_replica_n45
[junit4] 2> 1414170 INFO (updateExecutor-2281-thread-1) [ ]
o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1414170 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1414170 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1414170 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.c.RecoveryStrategy Begin buffering updates.
core=[collection1_shard2_replica_n45]
[junit4] 2> 1414170 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE,
tlog=null}
[junit4] 2> 1414170 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.c.RecoveryStrategy Publishing state of core
[collection1_shard2_replica_n45] as recovering, leader is
[http://127.0.0.1:45489/h_uz/s/collection1_shard2_replica_n41/] and I am
[http://127.0.0.1:45793/h_uz/s/collection1_shard2_replica_n45/]
[junit4] 2> 1414170 INFO (qtp693710932-14841) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
status=0 QTime=1275
[junit4] 2> 1414171 INFO (recoveryExecutor-2282-thread-1) [ ]
o.a.s.c.RecoveryStrategy Sending prep recovery command to
[http://127.0.0.1:45489/h_uz/s]; [WaitForState:
action=PREPRECOVERY&core=collection1_shard2_replica_n41&nodeName=127.0.0.1:45793_h_uz%252Fs&coreNodeName=core_node46&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1414172 INFO (qtp884502929-14785) [ ]
o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node46, state:
recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true,
maxTime: 183 s
[junit4] 2> 1414172 INFO (qtp417876687-14813) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={node=127.0.0.1:45793_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2}
status=0 QTime=1279
[junit4] 2> 1414172 INFO (qtp884502929-14785) [ ]
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1,
shard=shard2, thisCore=collection1_shard2_replica_n41,
leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true,
currentState=down, localState=active, nodeName=127.0.0.1:45793_h_uz%2Fs,
coreNodeName=core_node46, onlyIfActiveCheckResult=false, nodeProps:
core_node46:{"core":"collection1_shard2_replica_n45","base_url":"http://127.0.0.1:45793/h_uz/s","node_name":"127.0.0.1:45793_h_uz%2Fs","state":"down","type":"NRT"}
[junit4] 2> 1414233 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-4-001
of type NRT
[junit4] 2> 1414233 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@38c86f44{/h_uz/s,null,AVAILABLE}
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.AbstractConnector Started
ServerConnector@2523f25c{HTTP/1.1,[http/1.1]}{127.0.0.1:36687}
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.e.j.s.Server Started @1415617ms
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/h_uz/s,
solrconfig=solrconfig.xml,
solr.data.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/tempDir-001/jetty4,
hostPort=36687,
coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-4-001/cores}
[junit4] 2> 1414234 ERROR
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging
may be missing or incomplete.
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™
version 8.0.0
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on
port null
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1414234 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-11-22T05:13:31.901780Z
[junit4] 2> 1414236 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
ZooKeeper)
[junit4] 2> 1414236 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Loading container configuration from
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-4-001/solr.xml
[junit4] 2> 1414237 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1414237 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig Configuration parameter
autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1414237 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.SolrXmlConfig MBean server found:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620, but no JMX reporters were
configured - adding default JMX reporter.
[junit4] 2> 1414239 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40871/solr
[junit4] 2> 1414240 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ]
o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from
client sessionid 0x15fe2251f920011, likely client has closed socket
[junit4] 2> at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 1414242 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 1414243 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1414244 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:36687_h_uz%2Fs
[junit4] 2> 1414244 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414244 INFO (zkCallback-2259-thread-2) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414244 INFO (zkCallback-2284-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414244 INFO (zkCallback-2291-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414244 INFO (zkCallback-2267-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414244 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 1414294 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1414300 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1414300 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1414301 INFO
(TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42])
[ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-4-001/cores
[junit4] 2> 1414329 INFO (qtp614814898-14874) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params
node=127.0.0.1:36687_h_uz%252Fs&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 1414330 INFO
(OverseerCollectionConfigSetProcessor-99046365896769540-127.0.0.1:44417_h_uz%2Fs-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000008 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 1414331 INFO (OverseerThreadFactory-7349-thread-5) [ ]
o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:36687_h_uz%2Fs for creating new
replica
[junit4] 2> 1414332 INFO (qtp614814898-14876) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n47&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1414332 INFO (qtp614814898-14876) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4
transient cores
[junit4] 2> 1414434 INFO (zkCallback-2272-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [5])
[junit4] 2> 1414434 INFO (zkCallback-2291-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [5])
[junit4] 2> 1414434 INFO (zkCallback-2284-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [5])
[junit4] 2> 1414434 INFO (zkCallback-2278-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [5])
[junit4] 2> 1415172 INFO (qtp884502929-14785) [ ]
o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1,
shard=shard2, thisCore=collection1_shard2_replica_n41,
leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true,
currentState=recovering, localState=active, nodeName=127.0.0.1:45793_h_uz%2Fs,
coreNodeName=core_node46, onlyIfActiveCheckResult=false, nodeProps:
core_node46:{"core":"collection1_shard2_replica_n45","base_url":"http://127.0.0.1:45793/h_uz/s","node_name":"127.0.0.1:45793_h_uz%2Fs","state":"recovering","type":"NRT"}
[junit4] 2> 1415172 INFO (qtp884502929-14785) [ ]
o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node46, state: recovering,
checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 1415172 INFO (qtp884502929-14785) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={nodeName=127.0.0.1:45793_h_uz%252Fs&onlyIfLeaderActive=true&core=collection1_shard2_replica_n41&coreNodeName=core_node46&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2}
status=0 QTime=1000
[junit4] 2> 1415355 INFO (qtp614814898-14876) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1415369 INFO (qtp614814898-14876) [ ] o.a.s.s.IndexSchema
[collection1_shard1_replica_n47] Schema name=test
[junit4] 2> 1415412 INFO (qtp614814898-14876) [ ] o.a.s.s.IndexSchema
Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1415419 INFO (qtp614814898-14876) [ ]
o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n47' using
configuration from collection collection1, trusted=true
[junit4] 2> 1415419 INFO (qtp614814898-14876) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr.core.collection1.shard1.replica_n47' (registry
'solr.core.collection1.shard1.replica_n47') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@3c642620
[junit4] 2> 1415419 INFO (qtp614814898-14876) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 1415420 INFO (qtp614814898-14876) [ ] o.a.s.c.SolrCore
[[collection1_shard1_replica_n47] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_AC6E4B5CCD8A9F42-001/shard-4-001/cores/collection1_shard1_replica_n47],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-
[...truncated too long message...]
), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] 2> at java.base@10-ea/java.lang.Thread.sleep(Native
Method)
[junit4] 2> at
app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] 2> at
app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] 2> at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] 2> NOTE: test params are:
codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY,
chunkSize=10805, maxDocsPerChunk=2, blockSize=961),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY,
chunkSize=10805, blockSize=961)),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5d8c514e),
locale=sg, timezone=Etc/GMT-8
[junit4] 2> NOTE: Linux 4.10.0-37-generic amd64/Oracle Corporation 10-ea
(64-bit)/cpus=8,threads=2,free=362485376,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [ZkFailoverTest,
ShowFileRequestHandlerTest, TestSchemaNameResource, TestFreeTextSuggestions,
TestPivotHelperCode, TestSha256AuthenticationProvider,
TestSubQueryTransformerDistrib, TestLocalFSCloudBackupRestore,
TestCloudManagedSchema, TestSchemaVersionResource,
TestSolrCloudWithHadoopAuthPlugin, TestManagedSchemaThreadSafety,
HealthCheckHandlerTest, HdfsSyncSliceTest, PathHierarchyTokenizerFactoryTest,
SolrXmlInZkTest, TestAnalyzeInfixSuggestions, TestPayloadScoreQParserPlugin,
DistributedIntervalFacetingTest, SpellingQueryConverterTest,
TestSolrQueryParser, CircularListTest, SolrJmxReporterTest,
TestPseudoReturnFields, DistributedFacetPivotLargeTest, TestLegacyField,
UninvertDocValuesMergePolicyTest, ConvertedLegacyTest, ZkSolrClientTest,
TestRestManager, SolrCmdDistributorTest, TestCursorMarkWithoutUniqueKey,
TestDocSet, TestFieldSortValues, TestWriterPerf, BlockJoinFacetDistribTest,
TestShortCircuitedRequests, DebugComponentTest, TestQuerySenderNoQuery,
TestMacroExpander, CoreAdminRequestStatusTest, SaslZkACLProviderTest,
DirectUpdateHandlerTest, TestLeaderElectionWithEmptyReplica,
TestScoreJoinQPNoScore, MBeansHandlerTest, SolrIndexConfigTest,
DistributedDebugComponentTest, AddBlockUpdateTest, EnumFieldTest,
ChaosMonkeySafeLeaderWithPullReplicasTest, SimpleCollectionCreateDeleteTest,
ResponseHeaderTest, TestPolicyCloud, PingRequestHandlerTest, TestFastWriter,
LargeFieldTest, SmileWriterTest, DeleteLastCustomShardedReplicaTest,
TestInfoStreamLogging, AssignTest, SignatureUpdateProcessorFactoryTest,
OverseerTaskQueueTest, DOMUtilTest, ClusterStateUpdateTest,
TestSolrCloudWithDelegationTokens, TestStressCloudBlindAtomicUpdates,
BadCopyFieldTest, MissingSegmentRecoveryTest, CollectionsAPIDistributedZkTest,
TestFaceting, TestJmxIntegration, TestSimpleTrackingShardHandler,
HighlighterConfigTest, NodeLostTriggerTest, DocumentAnalysisRequestHandlerTest,
NumericFieldsTest, MoreLikeThisHandlerTest, TestCloudInspectUtil,
SortSpecParsingTest, NotRequiredUniqueKeyTest,
DistributedQueryComponentOptimizationTest, LukeRequestHandlerTest,
DistribDocExpirationUpdateProcessorTest, TestReplicationHandlerBackup,
BlockDirectoryTest, ClassificationUpdateProcessorTest,
TestEmbeddedSolrServerAdminHandler, ChaosMonkeySafeLeaderTest,
OpenExchangeRatesOrgProviderTest, ConcurrentDeleteAndCreateCollectionTest,
DataDrivenBlockJoinTest, TestBlobHandler, TestInPlaceUpdatesDistrib,
CdcrReplicationHandlerTest, HLLUtilTest, TestDocBasedVersionConstraints,
TestNoOpRegenerator, DistributedVersionInfoTest,
TlogReplayBufferedWhileIndexingTest, CloudExitableDirectoryReaderTest,
BadIndexSchemaTest, TestRangeQuery, TestCollectionAPIs,
RestartWhileUpdatingTest, TestBackupRepositoryFactory, TestStressReorder,
PrimUtilsTest, HdfsLockFactoryTest, InfixSuggestersTest, TestSmileRequest,
TestRebalanceLeaders, TestRTGBase, TestNumericTerms32, TestClusterStateMutator,
TestTrieFacet, TestSchemalessBufferedUpdates, SpatialHeatmapFacetsTest,
AtomicUpdatesTest, TestFieldCacheReopen, TestMaxScoreQueryParser,
TestBM25SimilarityFactory, BasicDistributedZk2Test, UnloadDistributedZkTest,
SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest,
ShardRoutingTest, TestZkChroot, ShardRoutingCustomTest,
TestDistributedGrouping, TestRecovery, TermVectorComponentDistributedTest,
TestRealTimeGet, TestJoin, TestStressVersions, TestCoreContainer,
SpellCheckComponentTest, PeerSyncTest, TestFunctionQuery,
BasicFunctionalityTest, TestLazyCores, HighlighterTest,
DistributedQueryElevationComponentTest, SolrIndexSplitterTest,
SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest,
SuggesterTSTTest, TestStressLucene, TestTrie, SpatialFilterTest, TestCSVLoader,
PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest,
SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest,
TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest,
DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest,
StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest,
TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1,
IndexBasedSpellCheckerTest, TestSurroundQueryParser, DisMaxRequestHandlerTest,
TestQueryUtils, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions,
FileBasedSpellCheckerTest, TermsComponentTest, RequestHandlersTest,
TermVectorComponentTest, TestIndexingPerformance, RequiredFieldsTest,
LoggingHandlerTest, RegexBoostProcessorTest, SolrPluginUtilsTest,
TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest,
TestCSVResponseWriter, QueryParsingTest, TestAnalyzedSuggestions,
JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication,
CSVRequestHandlerTest, SearchHandlerTest, SOLR749Test, TestQuerySenderListener,
UpdateParamsTest, AlternateDirectoryTest, ResponseLogComponentTest,
BadComponentTest, TestStressRecovery, TestMergePolicyConfig, SampleTest,
MinimalSchemaTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest,
TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion,
SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest,
TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory,
TestPerFieldSimilarity, TestIBSimilarityFactory, TimeZoneUtilsTest,
ScriptEngineTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache,
PreAnalyzedFieldTest, ClusterStateTest, TestSolrJ, TestUtils,
TestDocumentBuilder, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest,
DistributedMLTComponentTest, SolrTestCaseJ4Test, TestDistributedMissingSort,
TestLuceneIndexBackCompat, TestEmbeddedSolrServerSchemaAPI,
TestJettySolrRunner, ActionThrottleTest, AsyncCallRequestStatusResponseTest,
ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=ChaosMonkeyNothingIsSafeWithPullReplicasTest
-Dtests.seed=AC6E4B5CCD8A9F42 -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=sg -Dtests.timezone=Etc/GMT-8 -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | ChaosMonkeyNothingIsSafeWithPullReplicasTest
(suite) <<<
[junit4] > Throwable #1:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE
scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
[junit4] > 1) Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@10-ea/java.lang.Thread.sleep(Native
Method)
[junit4] > at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
[junit4] > 2) Thread[id=14773, name=Connection evictor,
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@10-ea/java.lang.Thread.sleep(Native
Method)
[junit4] > at
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
[junit4] > at
java.base@10-ea/java.lang.Thread.run(Thread.java:844)
[junit4] > 3) Thread[id=14775,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-EventThread,
state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at
java.base@10-ea/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at
java.base@10-ea/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
[junit4] > at
java.base@10-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2074)
[junit4] > at
java.base@10-ea/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
[junit4] > at
app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
[junit4] > at
__randomizedtesting.SeedInfo.seed([AC6E4B5CCD8A9F42]:0)Throwable #2:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
threads that couldn't be terminated:
[junit4] > 1) Thread[id=14774,
name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[AC6E4B5CCD8A9F42]-SendThread(127.0.0.1:40871),
state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@10-ea/java.lang.Thread.sleep(Native
Method)
[junit4] > at
app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at
app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at
app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > at
__randomizedtesting.SeedInfo.seed([AC6E4B5CCD8A9F42]:0)
[junit4] Completed [523/750 (1!)] on J1 in 113.42s, 1 test, 1 failure, 2
errors <<< FAILURES!
[...truncated 40850 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
