Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20425/
Java: 32bit/jdk-9-ea+181 -client -XX:+UseG1GC --illegal-access=deny
1 tests failed.
FAILED: org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
Error Message:
Timeout waiting for all live and active
Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
at
__randomizedtesting.SeedInfo.seed([C9D4C7D991EAB5F7:4AA2982B4793BB56]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:184)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 11927 lines...]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] 2> 629271 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks:
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/init-core-data-001
[junit4] 2> 629271 WARN
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
[junit4] 2> 629272 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true)
w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 629272 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via:
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0,
clientAuth=0.0/0.0)
[junit4] 2> 629272 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001
[junit4] 2> 629272 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 629273 INFO (Thread-1162) [ ] o.a.s.c.ZkTestServer client
port:0.0.0.0/0.0.0.0:0
[junit4] 2> 629273 INFO (Thread-1162) [ ] o.a.s.c.ZkTestServer
Starting server
[junit4] 2> 629274 ERROR (Thread-1162) [ ] o.a.z.s.ZooKeeperServer
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action
on ERROR or SHUTDOWN server state changes
[junit4] 2> 629379 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.ZkTestServer start zk server on port:38879
[junit4] 2> 629382 INFO (jetty-launcher-946-thread-1) [ ]
o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 629382 INFO (jetty-launcher-946-thread-2) [ ]
o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 629387 INFO (jetty-launcher-946-thread-1) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@124a4b9{/solr,null,AVAILABLE}
[junit4] 2> 629387 INFO (jetty-launcher-946-thread-2) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@1d64fff{/solr,null,AVAILABLE}
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@2c0ad3{HTTP/1.1,[http/1.1]}{127.0.0.1:36179}
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.e.j.s.AbstractConnector Started
ServerConnector@1827017{HTTP/1.1,[http/1.1]}{127.0.0.1:38799}
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.e.j.s.Server Started @631017ms
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.e.j.s.Server Started @631017ms
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
hostPort=38799}
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
hostPort=36179}
[junit4] 2> 629391 ERROR (jetty-launcher-946-thread-1) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 629391 ERROR (jetty-launcher-946-thread-2) [ ]
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version
8.0.0
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config
dir: null
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config
dir: null
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-09-06T17:03:32.141830Z
[junit4] 2> 629391 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2017-09-06T17:03:32.141845Z
[junit4] 2> 629394 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 629394 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 629400 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38879/solr
[junit4] 2> 629409 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38879/solr
[junit4] 2> 629423 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 629424 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36179_solr
[junit4] 2> 629424 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.Overseer Overseer
(id=98613160314339334-127.0.0.1:36179_solr-n_0000000000) starting
[junit4] 2> 629433 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:36179_solr
[junit4] 2> 629434 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 629456 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 629457 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 629458 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:38799_solr
[junit4] 2> 629458 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 629459 INFO (zkCallback-957-thread-1) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 629497 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36179.solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629506 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36179.solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629506 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36179.solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629507 INFO (jetty-launcher-946-thread-2) [ ]
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node2/.
[junit4] 2> 629553 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38799.solr.node' (registry
'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629559 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38799.solr.jvm' (registry
'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629559 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38799.solr.jetty' (registry
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 629560 INFO (jetty-launcher-946-thread-1) [ ]
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node1/.
[junit4] 2> 629584 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 629584 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38879/solr ready
[junit4] 2> 629603 INFO (qtp22976181-5432) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 629609 INFO (OverseerThreadFactory-2736-thread-1) [ ]
o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 629719 INFO
(OverseerStateUpdate-98613160314339334-127.0.0.1:36179_solr-n_0000000000) [
] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38799/solr";,
[junit4] 2> "type":"TLOG"}
[junit4] 2> 629720 INFO
(OverseerStateUpdate-98613160314339334-127.0.0.1:36179_solr-n_0000000000) [
] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36179/solr";,
[junit4] 2> "type":"TLOG"}
[junit4] 2> 629721 INFO
(OverseerStateUpdate-98613160314339334-127.0.0.1:36179_solr-n_0000000000) [
] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38799/solr";,
[junit4] 2> "type":"TLOG"}
[junit4] 2> 629721 INFO
(OverseerStateUpdate-98613160314339334-127.0.0.1:36179_solr-n_0000000000) [
] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36179/solr";,
[junit4] 2> "type":"TLOG"}
[junit4] 2> 629923 INFO (qtp22976181-5423) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 629923 INFO (qtp22976181-5426) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 629923 INFO (qtp3430435-5422) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 629923 INFO (qtp22976181-5423) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647
transient cores
[junit4] 2> 629923 INFO (qtp3430435-5422) [ ]
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647
transient cores
[junit4] 2> 629924 INFO (qtp3430435-5431) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 630026 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 630026 INFO (zkCallback-957-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 630026 INFO (zkCallback-958-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 630026 INFO (zkCallback-957-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 630928 INFO (qtp22976181-5423) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 630931 INFO (qtp22976181-5423) [ ] o.a.s.s.IndexSchema
[collection1_shard1_replica_t1] Schema name=minimal
[junit4] 2> 630931 INFO (qtp22976181-5426) [ ] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 8.0.0
[junit4] 2> 630932 INFO (qtp22976181-5423) [ ] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 630933 INFO (qtp22976181-5423) [ ] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard1_replica_t1' using configuration from
collection collection1, trusted=true
[junit4] 2> 630973 INFO (qtp3430435-5431) [ ] o.a.s.c.SolrConfig Using
Lucene MatchVersion: 8.0.0
[junit4] 2> 630974 INFO (qtp3430435-5422) [ ] o.a.s.c.SolrConfig Using
Lucene MatchVersion: 8.0.0
[junit4] 2> 630975 INFO (qtp22976181-5423) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr_38799.solr.core.collection1.shard1.replica_t1' (registry
'solr.core.collection1.shard1.replica_t1') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 630975 INFO (qtp22976181-5423) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 630977 INFO (qtp22976181-5423) [ ] o.a.s.c.SolrCore
[[collection1_shard1_replica_t1] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node1/collection1_shard1_replica_t1],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node1/./collection1_shard1_replica_t1/data/]
[junit4] 2> 630977 INFO (qtp22976181-5426) [ ] o.a.s.s.IndexSchema
[collection1_shard2_replica_t4] Schema name=minimal
[junit4] 2> 630978 INFO (qtp22976181-5426) [ ] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 630978 INFO (qtp22976181-5426) [ ] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard2_replica_t4' using configuration from
collection collection1, trusted=true
[junit4] 2> 630979 INFO (qtp22976181-5426) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr_38799.solr.core.collection1.shard2.replica_t4' (registry
'solr.core.collection1.shard2.replica_t4') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 630979 INFO (qtp22976181-5426) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 630979 INFO (qtp22976181-5426) [ ] o.a.s.c.SolrCore
[[collection1_shard2_replica_t4] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node1/collection1_shard2_replica_t4],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node1/./collection1_shard2_replica_t4/data/]
[junit4] 2> 630979 INFO (qtp3430435-5431) [ ] o.a.s.s.IndexSchema
[collection1_shard1_replica_t2] Schema name=minimal
[junit4] 2> 630979 INFO (qtp3430435-5422) [ ] o.a.s.s.IndexSchema
[collection1_shard2_replica_t6] Schema name=minimal
[junit4] 2> 630980 INFO (qtp3430435-5431) [ ] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 630980 INFO (qtp3430435-5431) [ ] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard1_replica_t2' using configuration from
collection collection1, trusted=true
[junit4] 2> 630981 INFO (qtp3430435-5422) [ ] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 630981 INFO (qtp3430435-5422) [ ] o.a.s.c.CoreContainer
Creating SolrCore 'collection1_shard2_replica_t6' using configuration from
collection collection1, trusted=true
[junit4] 2> 630981 INFO (qtp3430435-5431) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr_36179.solr.core.collection1.shard1.replica_t2' (registry
'solr.core.collection1.shard1.replica_t2') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 630981 INFO (qtp3430435-5431) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 630981 INFO (qtp3430435-5422) [ ]
o.a.s.m.r.SolrJmxReporter JMX monitoring for
'solr_36179.solr.core.collection1.shard2.replica_t6' (registry
'solr.core.collection1.shard2.replica_t6') enabled at server:
com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5694
[junit4] 2> 630981 INFO (qtp3430435-5431) [ ] o.a.s.c.SolrCore
[[collection1_shard1_replica_t2] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node2/collection1_shard1_replica_t2],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node2/./collection1_shard1_replica_t2/data/]
[junit4] 2> 630981 INFO (qtp3430435-5422) [ ] o.a.s.c.SolrCore
solr.RecoveryStrategy.Builder
[junit4] 2> 630981 INFO (qtp3430435-5422) [ ] o.a.s.c.SolrCore
[[collection1_shard2_replica_t6] ] Opening new SolrCore at
[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node2/collection1_shard2_replica_t6],
dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001/tempDir-001/node2/./collection1_shard2_replica_t6/data/]
[junit4] 2> 631055 INFO (qtp22976181-5423) [ ] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 631055 INFO (qtp22976181-5423) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 631056 INFO (qtp22976181-5423) [ ] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 631056 INFO (qtp22976181-5423) [ ] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 631057 INFO (qtp22976181-5423) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@179e6dd[collection1_shard1_replica_t1] main]
[junit4] 2> 631058 INFO (qtp22976181-5423) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/config
[junit4] 2> 631058 INFO (qtp22976181-5423) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 631059 INFO (qtp22976181-5423) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 631059 INFO (qtp3430435-5431) [ ] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 631060 INFO (qtp3430435-5431) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 631060 INFO (searcherExecutor-2741-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher
Searcher@179e6dd[collection1_shard1_replica_t1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 631060 INFO (qtp22976181-5423) [ ] o.a.s.u.UpdateLog Could
not find max version in index or recent updates, using new clock
1577810566903234560
[junit4] 2> 631061 INFO (qtp3430435-5431) [ ] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 631061 INFO (qtp3430435-5431) [ ] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 631064 INFO (qtp3430435-5431) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@1e8bf21[collection1_shard1_replica_t2] main]
[junit4] 2> 631065 INFO (qtp3430435-5431) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/config
[junit4] 2> 631065 INFO (qtp3430435-5431) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 631066 INFO (qtp3430435-5431) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 631067 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for
shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 631067 INFO (searcherExecutor-2743-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher
Searcher@1e8bf21[collection1_shard1_replica_t2]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 631067 INFO (qtp3430435-5431) [ ] o.a.s.u.UpdateLog Could
not find max version in index or recent updates, using new clock
1577810566910574592
[junit4] 2> 631069 INFO (qtp22976181-5426) [ ] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 631069 INFO (qtp22976181-5426) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 631070 INFO (qtp22976181-5426) [ ] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 631070 INFO (qtp22976181-5426) [ ] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 631072 INFO (qtp22976181-5426) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@19e02dd[collection1_shard2_replica_t4] main]
[junit4] 2> 631072 INFO (qtp22976181-5426) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/config
[junit4] 2> 631073 INFO (qtp22976181-5426) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 631073 INFO (qtp22976181-5426) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 631074 INFO (searcherExecutor-2742-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher
Searcher@19e02dd[collection1_shard2_replica_t4]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 631074 INFO (qtp22976181-5426) [ ] o.a.s.u.UpdateLog Could
not find max version in index or recent updates, using new clock
1577810566917914624
[junit4] 2> 631077 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for
shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 631087 INFO (qtp3430435-5422) [ ] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 631087 INFO (qtp3430435-5422) [ ] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 631088 INFO (qtp3430435-5422) [ ] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 631088 INFO (qtp3430435-5422) [ ] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 631090 INFO (qtp3430435-5422) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@14f4095[collection1_shard2_replica_t6] main]
[junit4] 2> 631091 INFO (qtp3430435-5422) [ ]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/config
[junit4] 2> 631091 INFO (qtp3430435-5422) [ ]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 631091 INFO (qtp3430435-5422) [ ]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 631092 INFO (searcherExecutor-2744-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher
Searcher@14f4095[collection1_shard2_replica_t6]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 631092 INFO (qtp3430435-5422) [ ] o.a.s.u.UpdateLog Could
not find max version in index or recent updates, using new clock
1577810566936788992
[junit4] 2> 631178 INFO (zkCallback-958-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631178 INFO (zkCallback-957-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631178 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631178 INFO (zkCallback-957-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631567 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 631567 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 631567 INFO (qtp22976181-5423) [ ] o.a.s.c.SyncStrategy
Sync replicas to http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/
[junit4] 2> 631567 INFO (qtp22976181-5423) [ ] o.a.s.u.PeerSync
PeerSync: core=collection1_shard1_replica_t1 url=http://127.0.0.1:38799/solr
START replicas=[http://127.0.0.1:36179/solr/collection1_shard1_replica_t2/]
nUpdates=100
[junit4] 2> 631568 INFO (qtp3430435-5435) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t2] webapp=/solr path=/get
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
status=0 QTime=0
[junit4] 2> 631577 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 631577 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 631577 INFO (qtp22976181-5426) [ ] o.a.s.c.SyncStrategy
Sync replicas to http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/
[junit4] 2> 631578 INFO (qtp22976181-5426) [ ] o.a.s.u.PeerSync
PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:38799/solr
START replicas=[http://127.0.0.1:36179/solr/collection1_shard2_replica_t6/]
nUpdates=100
[junit4] 2> 631578 INFO (qtp3430435-5497) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t6] webapp=/solr path=/get
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
status=0 QTime=0
[junit4] 2> 631868 INFO (qtp22976181-5423) [ ] o.a.s.u.PeerSync
PeerSync: core=collection1_shard1_replica_t1 url=http://127.0.0.1:38799/solr
DONE. We have no versions. sync failed.
[junit4] 2> 631868 INFO (qtp22976181-5423) [ ] o.a.s.c.SyncStrategy
Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 631868 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we
can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 631868 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 631868 INFO (qtp22976181-5423) [ ] o.a.s.c.ZkController
collection1_shard1_replica_t1 stopping background replication from leader
[junit4] 2> 631870 INFO (qtp22976181-5423) [ ]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/ shard1
[junit4] 2> 631878 INFO (qtp22976181-5426) [ ] o.a.s.u.PeerSync
PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:38799/solr
DONE. We have no versions. sync failed.
[junit4] 2> 631878 INFO (qtp22976181-5426) [ ] o.a.s.c.SyncStrategy
Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 631878 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we
can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 631878 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
election, clear LIR
[junit4] 2> 631878 INFO (qtp22976181-5426) [ ] o.a.s.c.ZkController
collection1_shard2_replica_t4 stopping background replication from leader
[junit4] 2> 631880 INFO (qtp22976181-5426) [ ]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/ shard2
[junit4] 2> 631981 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631981 INFO (zkCallback-957-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631981 INFO (zkCallback-957-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 631981 INFO (zkCallback-958-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 632021 INFO (qtp22976181-5423) [ ] o.a.s.c.ZkController I
am the leader, no recovery necessary
[junit4] 2> 632022 INFO (qtp22976181-5423) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG}
status=0 QTime=2098
[junit4] 2> 632030 INFO (qtp22976181-5426) [ ] o.a.s.c.ZkController I
am the leader, no recovery necessary
[junit4] 2> 632031 INFO (qtp22976181-5426) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG}
status=0 QTime=2108
[junit4] 2> 632070 INFO (qtp3430435-5431) [ ] o.a.s.c.ZkController
collection1_shard1_replica_t2 starting background replication from leader
[junit4] 2> 632070 INFO (qtp3430435-5431) [ ]
o.a.s.c.ReplicateFromLeader Will start replication from leader with poll
interval: 00:00:03
[junit4] 2> 632071 INFO (qtp3430435-5431) [ ]
o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 632072 INFO (qtp3430435-5431) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG}
status=0 QTime=2148
[junit4] 2> 632095 INFO (qtp3430435-5422) [ ] o.a.s.c.ZkController
collection1_shard2_replica_t6 starting background replication from leader
[junit4] 2> 632095 INFO (qtp3430435-5422) [ ]
o.a.s.c.ReplicateFromLeader Will start replication from leader with poll
interval: 00:00:03
[junit4] 2> 632096 INFO (qtp3430435-5422) [ ]
o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 632096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Replica core_node7 is leader but it's state is down,
skipping replication
[junit4] 2> 632097 INFO (qtp3430435-5422) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG}
status=0 QTime=2173
[junit4] 2> 632098 INFO (qtp22976181-5432) [ ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 632199 INFO (zkCallback-957-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 632199 INFO (zkCallback-958-thread-1) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 632199 INFO (zkCallback-958-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 632199 INFO (zkCallback-957-thread-2) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/collection1/state.json] for collection [collection1] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 633098 INFO (qtp22976181-5432) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2}
status=0 QTime=3495
[junit4] 2> 633098 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection:
collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 633099 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 633108 INFO
(TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[C9D4C7D991EAB5F7])
[ ] o.a.s.SolrTestCaseJ4 ###Starting leaderRecoverFromLogOnStartupTest
[junit4] 2> 633112 INFO (qtp3430435-5498) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&_version_=-1577810569051766784&distrib.from=http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:*
(-1577810569051766784)} 0 0
[junit4] 2> 633114 INFO (qtp3430435-5427) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&_version_=-1577810569052815360&distrib.from=http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/&wt=javabin&version=2}{deleteByQuery=*:*
(-1577810569052815360)} 0 1
[junit4] 2> 633114 INFO (qtp22976181-5428) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1]
webapp=/solr path=/update
params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:*
(-1577810569052815360)} 0 4
[junit4] 2> 633114 INFO (qtp22976181-5434) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4]
webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:*
(-1577810569051766784)} 0 5
[junit4] 2> 633115 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1577810569058058240,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 633115 INFO (qtp3430435-5499) [ ] o.a.s.u.TestInjection
Start waiting for replica in sync with leader
[junit4] 2> 633115 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 633116 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 633116 INFO (qtp22976181-5423) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 0
[junit4] 2> 633116 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1577810569059106816,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 633116 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 633116 INFO (qtp22976181-5423) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 633118 INFO (qtp22976181-5423) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 2
[junit4] 2> 633119 INFO (qtp3430435-5498) [ ] o.a.s.u.TestInjection
Start waiting for replica in sync with leader
[junit4] 2> 633122 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=1
[junit4] 2> 633130 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 633610 INFO
(OverseerCollectionConfigSetProcessor-98613160314339334-127.0.0.1:36179_solr-n_0000000000)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 633624 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 633635 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 634126 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 634137 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 634273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Updated masterUrl to
http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/
[junit4] 2> 634274 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 634274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 634274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 634274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 634274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 634628 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 634638 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 635096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
[junit4] 2> 635096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Updated masterUrl to
http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/
[junit4] 2> 635096 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 635097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@d8f859 commitCommandVersion:0
[junit4] 2> 635117 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@209329[collection1_shard2_replica_t6] main]
[junit4] 2> 635117 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 635118 INFO (searcherExecutor-2744-thread-1) [ ]
o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher
Searcher@209329[collection1_shard2_replica_t6]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 635130 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 635141 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 635632 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 635642 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 636133 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 636144 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 636635 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 636646 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 637137 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 637148 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 637273 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 637274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 637274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 637274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 637274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 637639 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 637650 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 638096 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 638097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 638097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 638097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 638097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 638140 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 638151 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 638642 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 638653 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 639144 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 639155 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 639646 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 639657 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 640148 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 640161 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 640273 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 640274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 640274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 640274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 640274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 640650 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 640663 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 641096 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 641097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 641097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 641097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 641097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 641152 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 641165 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 641654 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 641666 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 642156 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 642168 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 642657 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 642670 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 643160 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 643172 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 643273 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 643273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 643274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 643274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 643274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 643662 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 643675 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 644096 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 644097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 644097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 644097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 644097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 644163 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 644177 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 644665 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 644679 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 645167 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 645181 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 645669 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 645683 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 646171 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 646185 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 646281 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 646281 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 646281 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 646281 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 646281 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 646691 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 646691 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 647096 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 647097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 647097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 647097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 647097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 647194 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 647197 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 647695 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 647698 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 648197 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 648200 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 648699 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 648702 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 649201 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 649203 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 649273 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 649274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 649274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 649274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 649274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 649702 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 649705 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 650096 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 650096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 650097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 650097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 650097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 650204 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 650207 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 650706 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 650709 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 651208 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 651210 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 651709 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 651712 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 652211 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 652214 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 652273 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 652274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 652274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 652274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 652274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 652713 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 652715 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 653096 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 653097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 653097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 653097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 653097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 653215 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 653217 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 653717 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 653718 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 654218 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 654220 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 654720 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 654721 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 655222 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 655223 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 655273 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 655273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 655273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 655273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 655273 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 655723 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 655724 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 656096 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 656096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 656097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 656097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 656097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 656225 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 656226 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 656727 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 656727 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 657229 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 657229 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 657731 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 657731 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 658233 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 658233 INFO (qtp22976181-5436) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 658273 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 658274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 658274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 658274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 658274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 658735 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 658735 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 659096 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 659096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 659096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 659096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 659096 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 659239 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 659239 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 659741 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 659741 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 660242 INFO (qtp22976181-5434) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 660242 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 660744 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 660744 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 661247 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 661247 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 661273 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 661274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 661274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 661274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 661274 INFO (indexFetcher-2762-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 661750 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 661750 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 662096 INFO (qtp22976181-5432) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0
QTime=0
[junit4] 2> 662097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 662097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 662097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 662097 INFO (indexFetcher-2764-thread-1) [ ]
o.a.s.h.IndexFetcher Slave's version: 1504717417847
[junit4] 2> 662252 INFO (qtp22976181-5426) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 662252 INFO (qtp22976181-5423) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 662753 INFO (qtp22976181-5421) [ ] o.a.s.c.S.Request
[collection1_shard1_replica_t1] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 662754 INFO (qtp22976181-5428) [ ] o.a.s.c.S.Request
[collection1_shard2_replica_t4] webapp=/solr path=/replication
params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 663254 INFO (qtp3430435-5499) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{}
0 30138
[junit4] 2> 663254 INFO (qtp3430435-5498) [ ]
o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:38799/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{}
0 30134
[junit4] 2> 663254 ERROR (qtp3430435-5498) [ ] o.a.s.s.HttpSolrCall
null:java.lang.RuntimeException: java.lang.AssertionError: Core
org.apache.solr.core.SolrCore@17ef35e not in sync with leader
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:673)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:541)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:426)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at
org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:
[...truncated too long message...]
orUtil.java:188)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 796267 ERROR (coreZkRegister-2837-thread-1) [ ]
o.a.s.c.ZkContainer :org.apache.solr.common.SolrException: Error getting leader
from zk for shard shard2
[junit4] 2> at
org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1099)
[junit4] 2> at
org.apache.solr.cloud.ZkController.register(ZkController.java:971)
[junit4] 2> at
org.apache.solr.cloud.ZkController.register(ZkController.java:906)
[junit4] 2> at
org.apache.solr.core.ZkContainer.lambda$registerInZk$0(ZkContainer.java:181)
[junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Caused by: org.apache.solr.common.SolrException: CoreContainer
is closed
[junit4] 2> at
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1143)
[junit4] 2> at
org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1110)
[junit4] 2> at
org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1066)
[junit4] 2> ... 7 more
[junit4] 2>
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ]
o.a.s.c.SyncStrategy Sync replicas to
http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ]
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ]
o.a.s.c.SyncStrategy http://127.0.0.1:38799/solr/collection1_shard2_replica_t4/
has no replicas
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ] o.a.s.c.SolrCore
[collection1_shard2_replica_t4] CLOSING SolrCore
org.apache.solr.core.SolrCore@d31e83
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.collection1.shard2.replica_t4, tag=13835907
[junit4] 2> 796994 INFO (zkCallback-989-thread-2) [ ]
o.a.s.m.r.SolrJmxReporter Closing reporter
[org.apache.solr.metrics.reporters.SolrJmxReporter@1ef53ca: rootName =
solr_38799, domain = solr.core.collection1.shard2.replica_t4, service url =
null, agent id = null] for registry solr.core.collection1.shard2.replica_t4 /
com.codahale.metrics.MetricRegistry@18bf19b
[junit4] 2> 797002 INFO (zkCallback-989-thread-2) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.collection1.shard2.leader, tag=13835907
[junit4] 2> 797002 WARN (zkCallback-989-thread-2) [ ]
o.a.s.c.RecoveryStrategy Stopping recovery for
core=[collection1_shard2_replica_t4] coreNodeName=[core_node7]
[junit4] 2> 797002 INFO (zkCallback-989-thread-2) [ ]
o.a.s.c.ZkController collection1_shard2_replica_t4 stopping background
replication from leader
[junit4] 2> 797003 ERROR (zkCallback-989-thread-2) [ ]
o.a.s.c.ShardLeaderElectionContext There was a problem trying to register as
the leader:java.lang.NullPointerException
[junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:437)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
[junit4] 2> at
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
[junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
[junit4] 2> at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
[junit4] 2> at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2>
[junit4] 2> 798667 ERROR (recoveryExecutor-987-thread-1) [ ]
o.a.s.c.RecoveryStrategy SolrCore not found - cannot
recover:collection1_shard2_replica_t4
[junit4] 2> 798668 WARN (zkCallback-989-thread-2) [ ]
o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK:
[KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 798668 INFO (jetty-closer-947-thread-2) [ ]
o.e.j.s.h.ContextHandler Stopped
o.e.j.s.ServletContextHandler@16a4e4b{/solr,null,UNAVAILABLE}
[junit4] 2> 798669 ERROR
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper
server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 798669 INFO
(SUITE-TestCloudRecovery-seed#[C9D4C7D991EAB5F7]-worker) [ ]
o.a.s.c.ZkTestServer connecting to 127.0.0.1:38879 38879
[junit4] 2> 803998 INFO (Thread-1162) [ ] o.a.s.c.ZkTestServer
connecting to 127.0.0.1:38879 38879
[junit4] 2> 803998 WARN (Thread-1162) [ ] o.a.s.c.ZkTestServer Watch
limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/aliases.json
[junit4] 2> 7 /solr/clusterprops.json
[junit4] 2> 6 /solr/security.json
[junit4] 2> 6 /solr/configs/config
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 140 /solr/collections/collection1/state.json
[junit4] 2> 7 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/collections
[junit4] 2> 5 /solr/overseer/collection-queue-work
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 4 /solr/overseer/queue-work
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_C9D4C7D991EAB5F7-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70):
{id=Lucene50(blocksize=128)},
docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=108,
maxMBSortInHeap=5.814846949545646, sim=RandomSimilarity(queryNorm=false): {},
locale=zgh, timezone=America/Martinique
[junit4] 2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 9
(32-bit)/cpus=8,threads=1,free=202824608,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestMergePolicyConfig,
BasicZkTest, TestCollectionAPIs, RequestHandlersTest, SuggesterWFSTTest,
TestFieldCache, TestClassicSimilarityFactory, TestFunctionQuery,
JvmMetricsTest, BinaryUpdateRequestHandlerTest,
TestTolerantUpdateProcessorRandomCloud, TestNRTOpen,
TestConfigSetsAPIExclusivity, CollectionReloadTest,
DefaultValueUpdateProcessorTest, TestLazyCores,
OverseerCollectionConfigSetProcessorTest, ZkNodePropsTest, SearchHandlerTest,
TestExclusionRuleCollectionAccess, DistributedFacetPivotLargeTest,
TestCSVResponseWriter, SolrPluginUtilsTest, TestMultiWordSynonyms,
TestSolrCloudWithSecureImpersonation, TestPrepRecovery, TestBinaryField,
HdfsTlogReplayBufferedWhileIndexingTest, TestJoin,
DistributedFacetPivotSmallTest, FullHLLTest, CursorMarkTest, TestLegacyTerms,
IndexBasedSpellCheckerTest, VMParamsZkACLAndCredentialsProvidersTest,
SolrIndexSplitterTest, TestBadConfig, TestStressUserVersions,
TestQuerySenderNoQuery, PreAnalyzedFieldManagedSchemaCloudTest,
ConfigSetsAPITest, TestLRUStatsCache, DocumentAnalysisRequestHandlerTest,
TestFieldCacheSortRandom, TestDistributedStatsComponentCardinality,
TestSolrConfigHandlerCloud, SpellCheckCollatorWithCollapseTest,
TestGeoJSONResponseWriter, CollectionsAPIAsyncDistributedZkTest,
TestCrossCoreJoin, TestLegacyField, ExternalFileFieldSortTest,
LeaderElectionIntegrationTest, TestIntervalFaceting, UpdateParamsTest,
TestIBSimilarityFactory, CachingDirectoryFactoryTest, TestFieldTypeResource,
QueryResultKeyTest, HdfsChaosMonkeyNothingIsSafeTest, TestDistribIDF,
ChaosMonkeyNothingIsSafeTest, DistributedDebugComponentTest,
TestDistributedSearch, EchoParamsTest, TestLegacyFieldCache,
TestNestedDocsSort, SpellCheckCollatorTest, TestOmitPositions, UpdateLogTest,
CdcrReplicationDistributedZkTest, TestLockTree,
FieldMutatingUpdateProcessorTest, TestSSLRandomization, TestCloudRecovery]
[junit4] Completed [241/732 (1!)] on J0 in 174.74s, 2 tests, 1 failure <<<
FAILURES!
[...truncated 40741 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:810: The following
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:690: The following
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:678: Source checkout
is modified!!! Offending files:
* solr/licenses/commons-math3-3.6.1.jar.sha1
Total time: 78 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
