Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/651/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testBasicLeaderElection

Error Message:
IOException occured when talking to server at: 
https://127.0.0.1:55240/solr/tlog_replica_test_basic_leader_election_shard1_replica_t2

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException 
occured when talking to server at: 
https://127.0.0.1:55240/solr/tlog_replica_test_basic_leader_election_shard1_replica_t2
        at 
__randomizedtesting.SeedInfo.seed([EF0BEF7EDD81BE2:987878BF05425C18]:0)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:553)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:948)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
        at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
        at 
org.apache.solr.cloud.TestTlogReplica.testBasicLeaderElection(TestTlogReplica.java:666)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException 
occured when talking to server at: 
https://127.0.0.1:55240/solr/tlog_replica_test_basic_leader_election_shard1_replica_t2
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
        at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
        at 
org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:528)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:202)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection 
during handshake
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
        at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at 
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at 
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at 
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at 
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
        at 
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at 
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at 
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at 
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at 
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
        ... 10 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
        at sun.security.ssl.InputRecord.read(InputRecord.java:505)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
        ... 26 more




Build Log:
[...truncated 12418 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4]   2> 32397 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/init-core-data-001
   [junit4]   2> 32399 WARN  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 32399 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 32402 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 32855 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001
   [junit4]   2> 32866 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 32869 INFO  (Thread-36) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 32869 INFO  (Thread-36) [    ] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 33085 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:38436
   [junit4]   2> 33106 ERROR (Thread-36) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 33286 INFO  (zkConnectionManagerCallback-20-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33559 INFO  (jetty-launcher-17-thread-2) [    ] o.e.j.s.Server 
jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: 
daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 33559 INFO  (jetty-launcher-17-thread-1) [    ] o.e.j.s.Server 
jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: 
daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 33563 INFO  (jetty-launcher-17-thread-1) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 33563 INFO  (jetty-launcher-17-thread-1) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 33563 INFO  (jetty-launcher-17-thread-1) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 33565 INFO  (jetty-launcher-17-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@387150da{/solr,null,AVAILABLE}
   [junit4]   2> 33565 INFO  (jetty-launcher-17-thread-2) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 33565 INFO  (jetty-launcher-17-thread-2) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 33565 INFO  (jetty-launcher-17-thread-2) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 33566 INFO  (jetty-launcher-17-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@34c2c340{/solr,null,AVAILABLE}
   [junit4]   2> 33613 INFO  (jetty-launcher-17-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@1d4d2c67{SSL,[ssl, 
http/1.1]}{127.0.0.1:52284}
   [junit4]   2> 33613 INFO  (jetty-launcher-17-thread-2) [    ] o.e.j.s.Server 
Started @33717ms
   [junit4]   2> 33613 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=52284}
   [junit4]   2> 33613 ERROR (jetty-launcher-17-thread-2) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 33613 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 33614 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.4.0
   [junit4]   2> 33614 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 33614 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 33614 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-05-27T11:43:44.363Z
   [junit4]   2> 33616 INFO  (jetty-launcher-17-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@6fa10164{SSL,[ssl, 
http/1.1]}{127.0.0.1:55240}
   [junit4]   2> 33616 INFO  (jetty-launcher-17-thread-1) [    ] o.e.j.s.Server 
Started @33721ms
   [junit4]   2> 33616 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=55240}
   [junit4]   2> 33617 ERROR (jetty-launcher-17-thread-1) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 33617 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 33626 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.4.0
   [junit4]   2> 33626 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 33639 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find 
system property or JNDI)
   [junit4]   2> 33626 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 33670 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-05-27T11:43:44.419Z
   [junit4]   2> 33671 INFO  (zkConnectionManagerCallback-22-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33676 INFO  (zkConnectionManagerCallback-24-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33690 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 33730 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 33763 INFO  (jetty-launcher-17-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38436/solr
   [junit4]   2> 33766 INFO  (jetty-launcher-17-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38436/solr
   [junit4]   2> 33842 INFO  (zkConnectionManagerCallback-32-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33919 INFO  (zkConnectionManagerCallback-30-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33921 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 
0x100a03e1b630004, likely client has closed socket
   [junit4]   2> 33994 INFO  (zkConnectionManagerCallback-34-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 33997 INFO  (zkConnectionManagerCallback-36-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 34594 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34598 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:52284_solr
   [junit4]   2> 34606 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.Overseer Overseer 
(id=72233782645817350-127.0.0.1:52284_solr-n_0000000000) starting
   [junit4]   2> 34660 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 34668 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:55240_solr
   [junit4]   2> 34749 INFO  (zkCallback-33-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34749 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 34829 INFO  (zkConnectionManagerCallback-45-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 34853 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 34907 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:38436/solr ready
   [junit4]   2> 34999 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 34999 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:52284_solr
   [junit4]   2> 35001 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 35037 INFO  (zkCallback-33-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 35047 INFO  (zkCallback-44-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 35516 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_52284.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 35524 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_52284.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 35524 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_52284.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 35528 INFO  (jetty-launcher-17-thread-2) 
[n:127.0.0.1:52284_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node2/.
   [junit4]   2> 37057 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 37065 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 37065 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 37084 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/.
   [junit4]   2> 37178 INFO  (zkConnectionManagerCallback-50-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 37180 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 37182 INFO  (jetty-launcher-17-thread-1) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:38436/solr ready
   [junit4]   2> 37444 INFO  (zkConnectionManagerCallback-52-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 37507 INFO  (zkConnectionManagerCallback-57-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 37509 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 37511 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38436/solr ready
   [junit4]   2> 37594 INFO  
(SUITE-TestTlogReplica-seed#[EF0BEF7EDD81BE2]-worker) [    ] 
o.a.s.c.TestTlogReplica Using legacyCloud?: false
   [junit4]   2> 37872 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params 
val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and 
sendToOCPQueue=true
   [junit4]   2> 37891 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} 
status=0 QTime=19
   [junit4]   2> 37939 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting testKillTlogReplica
   [junit4]   2> 37946 INFO  (qtp1902644890-199) [n:127.0.0.1:52284_solr    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_tlog_replica&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 37986 INFO  (OverseerThreadFactory-115-thread-1) [    ] 
o.a.s.c.a.c.CreateCollectionCmd Create collection 
tlog_replica_test_kill_tlog_replica
   [junit4]   2> 38152 INFO  
(OverseerStateUpdate-72233782645817350-127.0.0.1:52284_solr-n_0000000000) [    
] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   
"core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:55240/solr";,
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 38164 INFO  
(OverseerStateUpdate-72233782645817350-127.0.0.1:52284_solr-n_0000000000) [    
] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"tlog_replica_test_kill_tlog_replica",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   
"core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:52284/solr";,
   [junit4]   2>   "type":"TLOG",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 38417 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr    
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 38490 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr    
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
   [junit4]   2> 38491 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr    
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
   [junit4]   2> 39495 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 39684 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 39715 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Schema name=minimal
   [junit4]   2> 39731 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 39731 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.CoreContainer 
Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t2' using 
configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
   [junit4]   2> 39735 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_52284.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2' 
(registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 39736 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 39736 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore 
[[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] ] Opening new SolrCore 
at 
[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node2/tlog_replica_test_kill_tlog_replica_shard1_replica_t2],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node2/./tlog_replica_test_kill_tlog_replica_shard1_replica_t2/data/]
   [junit4]   2> 39752 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 39760 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 39760 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.CoreContainer 
Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t1' using 
configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
   [junit4]   2> 39845 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1' 
(registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 39846 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 39846 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore 
[[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] ] Opening new SolrCore 
at 
[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/tlog_replica_test_kill_tlog_replica_shard1_replica_t1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/./tlog_replica_test_kill_tlog_replica_shard1_replica_t1/data/]
   [junit4]   2> 39919 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 39919 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 39922 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 39922 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 39924 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@25353128[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 39941 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 39941 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 39948 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 39948 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 39951 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@4e8e6e93[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
   [junit4]   2> 39956 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 39956 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 39957 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 39957 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 39958 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 39959 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 39961 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1601617454128168960
   [junit4]   2> 39962 INFO  
(searcherExecutor-121-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
Registered new searcher 
Searcher@25353128[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 39962 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1601617454129217536
   [junit4]   2> 39973 INFO  
(searcherExecutor-120-thread-1-processing-n:127.0.0.1:52284_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4) 
[n:127.0.0.1:52284_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
Registered new searcher 
Searcher@4e8e6e93[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 40024 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms 
Failed to save terms, version is not a match, retrying
   [junit4]   2> 40026 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to 
Terms{values={core_node3=0, core_node4=0}, version=1}
   [junit4]   2> 40065 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to 
Terms{values={core_node3=0}, version=0}
   [junit4]   2> 40197 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 40198 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 40199 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SyncStrategy 
Sync replicas to 
https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/
   [junit4]   2> 40221 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.PeerSync 
PeerSync: core=tlog_replica_test_kill_tlog_replica_shard1_replica_t2 
url=https://127.0.0.1:52284/solr START 
replicas=[https://127.0.0.1:55240/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t1/]
 nUpdates=100
   [junit4]   2> 40289 INFO  (qtp1638324411-192) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=12
   [junit4]   2> 40545 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.PeerSync 
PeerSync: core=tlog_replica_test_kill_tlog_replica_shard1_replica_t2 
url=https://127.0.0.1:52284/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 40546 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 40548 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 40549 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 40557 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkController 
tlog_replica_test_kill_tlog_replica_shard1_replica_t2 stopping background 
replication from leader
   [junit4]   2> 40612 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/
 shard1
   [junit4]   2> 40774 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 40797 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG}
 status=0 QTime=2382
   [junit4]   2> 40905 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 41217 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.ZkController 
tlog_replica_test_kill_tlog_replica_shard1_replica_t1 starting background 
replication from leader
   [junit4]   2> 41218 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.ReplicateFromLeader Will start replication from leader with poll 
interval: 00:00:03
   [junit4]   2> 41239 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
   [junit4]   2> 41239 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 41250 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG}
 status=0 QTime=2760
   [junit4]   2> 41347 INFO  (qtp1902644890-199) [n:127.0.0.1:52284_solr    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 41352 INFO  (zkCallback-33-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 41355 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 41994 INFO  
(OverseerCollectionConfigSetProcessor-72233782645817350-127.0.0.1:52284_solr-n_0000000000)
 [    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 42347 INFO  (qtp1902644890-199) [n:127.0.0.1:52284_solr    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_tlog_replica&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2}
 status=0 QTime=4404
   [junit4]   2> 42407 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 42596 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 42664 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to 
Terms{values={core_node3=1, core_node4=1}, version=2}
   [junit4]   2> 42679 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&wt=javabin&version=2}{add=[1
 (1601617456926818304)]} 0 2
   [junit4]   2> 42680 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/update params={wt=javabin&version=2}{add=[1 (1601617456926818304)]} 0 69
   [junit4]   2> 42696 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/update 
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
 0 0
   [junit4]   2> 42711 INFO  (qtp1902644890-196) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.DirectUpdateHandler2 start 
commit{_version_=1601617457011752960,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 42711 INFO  (qtp1902644890-196) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.SolrIndexWriter Calling setCommitData with 
IW:org.apache.solr.update.SolrIndexWriter@42cf46e3 
commitCommandVersion:1601617457011752960
   [junit4]   2> 42726 INFO  (qtp1902644890-196) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@4b0e94b9[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
   [junit4]   2> 42727 INFO  (qtp1902644890-196) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 42730 INFO  
(searcherExecutor-120-thread-1-processing-n:127.0.0.1:52284_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4) 
[n:127.0.0.1:52284_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
Registered new searcher 
Searcher@4b0e94b9[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 42730 INFO  (qtp1902644890-196) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/update 
params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=}
 0 19
   [junit4]   2> 42750 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/update 
params={_stateVer_=tlog_replica_test_kill_tlog_replica:6&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
 0 66
   [junit4]   2> 42772 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 42879 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 42982 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43093 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43206 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43341 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43447 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43565 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43683 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 43833 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 44048 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 44078 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Updated masterUrl to 
https://127.0.0.1:52284/solr/tlog_replica_test_kill_tlog_replica_shard1_replica_t2/
   [junit4]   2> 44151 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 44155 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 
QTime=0
   [junit4]   2> 44156 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 44156 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Master's version: 1527421433460
   [junit4]   2> 44156 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 44156 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Slave's version: 0
   [junit4]   2> 44156 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 44159 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&tlogFiles=false&wt=javabin&version=2&command=filelist}
 status=0 QTime=1
   [junit4]   2> 44160 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Number of files in latest index in master: 13
   [junit4]   2> 44186 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Starting download (fullCopy=false) to 
MockDirectoryWrapper(RAMDirectory@6021c46e 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@353d9ca5)
   [junit4]   2> 44207 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.pos&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=9
   [junit4]   2> 44213 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44225 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=4
   [junit4]   2> 44228 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.tip&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44231 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44234 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.doc&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44238 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.tim&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44242 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44246 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene70_0.dvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44249 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0_Lucene70_0.dvd&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44253 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=1
   [junit4]   2> 44254 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 44256 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44263 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/replication 
params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent}
 status=0 QTime=0
   [junit4]   2> 44265 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Bytes downloaded: 2136, Bytes skipped downloading: 0
   [junit4]   2> 44265 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.h.IndexFetcher Total time taken for download 
(fullCopy=false,bytesDownloaded=2136) : 0 secs (null bytes/sec) to 
MockDirectoryWrapper(RAMDirectory@6021c46e 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@353d9ca5)
   [junit4]   2> 44322 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 44358 INFO  (qtp1638324411-194) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 44360 INFO  (indexFetcher-131-thread-1) [    ] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@3f20c132[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 44363 INFO  (searcherExecutor-121-thread-1) [    ] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
Registered new searcher 
Searcher@3f20c132[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)))}
   [junit4]   2> 44474 INFO  (qtp1638324411-190) [n:127.0.0.1:55240_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 44503 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 44511 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.ChaosMonkey monkey: kill jetty! 55240
   [junit4]   2> 44620 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@6fa10164{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 44692 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1911100810
   [junit4]   2> 44693 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, 
tag=null
   [junit4]   2> 44701 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@4a2c2264: rootName = 
solr_55240, domain = solr.node, service url = null, agent id = null] for 
registry solr.node / com.codahale.metrics.MetricRegistry@5f65f242
   [junit4]   2> 44710 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, 
tag=null
   [junit4]   2> 44710 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@7699f843: rootName = 
solr_55240, domain = solr.jvm, service url = null, agent id = null] for 
registry solr.jvm / com.codahale.metrics.MetricRegistry@a1b8648
   [junit4]   2> 44718 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, 
tag=null
   [junit4]   2> 44718 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@56340555: rootName = 
solr_55240, domain = solr.jetty, service url = null, agent id = null] for 
registry solr.jetty / com.codahale.metrics.MetricRegistry@4d6b09dc
   [junit4]   2> 44721 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.ZkController Remove node as live in 
ZooKeeper:/live_nodes/127.0.0.1:55240_solr
   [junit4]   2> 44722 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, 
tag=null
   [junit4]   2> 44724 INFO  (zkCallback-49-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 44724 INFO  (zkCallback-35-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 44763 INFO  (zkCallback-44-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 44766 INFO  (zkCallback-56-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 44767 INFO  (zkCallback-33-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 44779 INFO  (coreCloseExecutor-133-thread-1) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1]  
CLOSING SolrCore org.apache.solr.core.SolrCore@5af61f9d
   [junit4]   2> 44779 INFO  (coreCloseExecutor-133-thread-1) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1, 
tag=5af61f9d
   [junit4]   2> 44779 INFO  (coreCloseExecutor-133-thread-1) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.m.r.SolrJmxReporter Closing reporter 
[org.apache.solr.metrics.reporters.SolrJmxReporter@50b917af: rootName = 
solr_55240, domain = 
solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1, service url = 
null, agent id = null] for registry 
solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1 / 
com.codahale.metrics.MetricRegistry@52e6124b
   [junit4]   2> 44796 INFO  (coreCloseExecutor-133-thread-1) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.m.SolrMetricManager Closing metric reporters for 
registry=solr.collection.tlog_replica_test_kill_tlog_replica.shard1.leader, 
tag=5af61f9d
   [junit4]   2> 44816 INFO  (zkCallback-33-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 44825 INFO  (zkCallback-35-thread-2) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [1])
   [junit4]   2> 44892 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@387150da{/solr,null,UNAVAILABLE}
   [junit4]   2> 44901 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 44903 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.ChaosMonkey monkey: stop jetty! 55240
   [junit4]   2> 44936 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.ZkShardTerms 
Successful update of terms at 
/collections/tlog_replica_test_kill_tlog_replica/terms/shard1 to 
Terms{values={core_node3=1, core_node4=2}, version=3}
   [junit4]   2> 44936 INFO  (qtp1902644890-193) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/update params={wt=javabin&version=2}{add=[2 (1601617459333300224)]} 0 11
   [junit4]   2> 44940 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.DirectUpdateHandler2 start 
commit{_version_=1601617459349028864,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 44941 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.SolrIndexWriter Calling setCommitData with 
IW:org.apache.solr.update.SolrIndexWriter@42cf46e3 
commitCommandVersion:1601617459349028864
   [junit4]   2> 44970 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@524f3e7d[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
   [junit4]   2> 44972 INFO  
(searcherExecutor-120-thread-1-processing-n:127.0.0.1:52284_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4) 
[n:127.0.0.1:52284_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
Registered new searcher 
Searcher@524f3e7d[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.4.0):C1)
 Uninverting(_1(7.4.0):C1)))}
   [junit4]   2> 44974 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 44974 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] 
o.a.s.u.p.LogUpdateProcessorFactory 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/update 
params={_stateVer_=tlog_replica_test_kill_tlog_replica:6&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=}
 0 34
   [junit4]   2> 45004 INFO  (qtp1902644890-200) [n:127.0.0.1:52284_solr 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.S.Request 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t2]  webapp=/solr 
path=/select params={q=*:*&wt=javabin&version=2} hits=2 status=0 QTime=0
   [junit4]   2> 45006 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.ChaosMonkey monkey: starting jetty! 55240
   [junit4]   2> 45015 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: 
daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
   [junit4]   2> 45016 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 45016 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 45016 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 45016 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7ee24f67{/solr,null,AVAILABLE}
   [junit4]   2> 45017 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@38290d98{SSL,[ssl, 
http/1.1]}{127.0.0.1:55240}
   [junit4]   2> 45017 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.e.j.s.Server Started @45127ms
   [junit4]   2> 45017 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=55240}
   [junit4]   2> 45018 ERROR 
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 45018 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 45018 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
7.4.0
   [junit4]   2> 45018 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 45018 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 45019 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-05-27T11:43:55.768Z
   [junit4]   2> 45023 INFO  (zkConnectionManagerCallback-59-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 45025 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 45050 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38436/solr
   [junit4]   2> 45122 INFO  (zkConnectionManagerCallback-63-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 45133 INFO  (zkConnectionManagerCallback-65-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 45172 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 45177 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 45185 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.TransientSolrCoreCacheDefault Allocating 
transient cache for 2147483647 transient cores
   [junit4]   2> 45185 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:55240_solr
   [junit4]   2> 45187 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 45191 INFO  (zkCallback-56-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 45191 INFO  (zkCallback-44-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 45209 INFO  (zkCallback-64-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 45288 INFO  (zkCallback-35-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for 
collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... 
(live nodes size: [2])
   [junit4]   2> 47083 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 47094 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 47094 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 47099 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.CorePropertiesLocator Found 1 core 
definitions underneath 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/.
   [junit4]   2> 47099 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.CorePropertiesLocator Cores are: 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1]
   [junit4]   2> 47293 INFO  (zkConnectionManagerCallback-72-thread-1) [    ] 
o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 47422 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (2)
   [junit4]   2> 47424 INFO  
(TEST-TestTlogReplica.testKillTlogReplica-seed#[EF0BEF7EDD81BE2]) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster 
at 127.0.0.1:38436/solr ready
   [junit4]   2> 47517 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 47531 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr    ] o.a.s.s.IndexSchema 
[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Schema name=minimal
   [junit4]   2> 47560 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with 
uniqueid field id
   [junit4]   2> 47618 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.4.0
   [junit4]   2> 47634 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.s.IndexSchema [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
Schema name=minimal
   [junit4]   2> 47637 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 47637 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.CoreContainer Creating SolrCore 
'tlog_replica_test_kill_tlog_replica_shard1_replica_t1' using configuration 
from collection tlog_replica_test_kill_tlog_replica, trusted=true
   [junit4]   2> 47637 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr_55240.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1' 
(registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@776afdcf
   [junit4]   2> 47638 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
   [junit4]   2> 47638 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrCore [[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] ] 
Opening new SolrCore at 
[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/tlog_replica_test_kill_tlog_replica_shard1_replica_t1],
 
dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_EF0BEF7EDD81BE2-001/tempDir-001/node1/./tlog_replica_test_kill_tlog_replica_shard1_replica_t1/data/]
   [junit4]   2> 47715 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 47716 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 47717 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 47718 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 47719 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@62046eda[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
   [junit4]   2> 47721 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 47722 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 47722 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 47724 INFO  
(searcherExecutor-141-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
Registered new searcher 
Searcher@62046eda[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 47724 INFO  
(coreLoadExecutor-140-thread-1-processing-n:127.0.0.1:55240_solr) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1601617462268264448
   [junit4]   2> 47744 INFO  
(coreZkRegister-135-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.ZkController Core needs to 
recover:tlog_replica_test_kill_tlog_replica_shard1_replica_t1
   [junit4]   2> 47746 INFO  
(updateExecutor-60-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 47754 INFO  
(recoveryExecutor-61-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 47757 INFO  
(recoveryExecutor-61-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1 
c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3) 
[n:127.0.0.1:55240_solr c:tlog_replica_test_kill_tlog_replica s:shard1 
r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] 
o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 47757 INFO  
(recoveryExecutor-61-thread-1-processing-n:127.0.0.1:55240_solr 
x:tlog_replica_test_kill_tlog_rep

[...truncated too long message...]

it4] OK      6.81s J0 | TestTlogReplica.testDeleteById
   [junit4] OK      8.56s J0 | TestTlogReplica.testOnlyLeaderIndexes
   [junit4] OK      11.2s J0 | TestTlogReplica.testRemoveLeader
   [junit4] OK      9.02s J0 | 
TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates
   [junit4] OK      9.91s J0 | TestTlogReplica.testBasicLeaderElection
   [junit4] OK      4.53s J0 | TestTlogReplica.testRealTimeGet
   [junit4] Completed [2/5] on J0 in 125.71s, 13 tests
   [junit4] 
   [junit4] Duplicate suite name used with XML reports: 
org.apache.solr.cloud.TestTlogReplica. This may confuse tools that process XML 
reports. Set 'ignoreDuplicateSuites' to true to skip this message.
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4] OK      12.4s J0 | TestTlogReplica.testKillTlogReplica
   [junit4] OK      7.64s J0 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:15E04FBB1B28E3D2]}
   [junit4] OK      7.18s J0 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:A1B6F3472FEA28FE]}
   [junit4] OK      13.1s J0 | TestTlogReplica.testRecovery
   [junit4] OK      11.2s J0 | TestTlogReplica.testKillLeader
   [junit4] OK      8.45s J0 | TestTlogReplica.testAddRemoveTlogReplica
   [junit4] OK      6.45s J0 | TestTlogReplica.testAddDocs
   [junit4] OK      5.44s J0 | TestTlogReplica.testDeleteById
   [junit4] OK      9.50s J0 | TestTlogReplica.testOnlyLeaderIndexes
   [junit4] OK      11.1s J0 | TestTlogReplica.testRemoveLeader
   [junit4] OK      8.89s J0 | 
TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates
   [junit4] OK      9.58s J0 | TestTlogReplica.testBasicLeaderElection
   [junit4] OK      4.52s J0 | TestTlogReplica.testRealTimeGet
   [junit4] Completed [3/5] on J0 in 119.03s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4] OK      12.1s J1 | TestTlogReplica.testKillTlogReplica
   [junit4] OK      7.29s J1 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:15E04FBB1B28E3D2]}
   [junit4] OK      7.22s J1 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:A1B6F3472FEA28FE]}
   [junit4] OK      12.8s J1 | TestTlogReplica.testRecovery
   [junit4] OK      11.2s J1 | TestTlogReplica.testKillLeader
   [junit4] OK      8.45s J1 | TestTlogReplica.testAddRemoveTlogReplica
   [junit4] OK      6.47s J1 | TestTlogReplica.testAddDocs
   [junit4] OK      7.48s J1 | TestTlogReplica.testDeleteById
   [junit4] OK      10.8s J1 | TestTlogReplica.testOnlyLeaderIndexes
   [junit4] OK      10.4s J1 | TestTlogReplica.testRemoveLeader
   [junit4] OK      8.86s J1 | 
TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates
   [junit4] OK      9.07s J1 | TestTlogReplica.testBasicLeaderElection
   [junit4] OK      4.36s J1 | TestTlogReplica.testRealTimeGet
   [junit4] Completed [4/5] on J1 in 121.90s, 13 tests
   [junit4] 
   [junit4] Suite: org.apache.solr.cloud.TestTlogReplica
   [junit4] OK      12.4s J0 | TestTlogReplica.testKillTlogReplica
   [junit4] OK      6.05s J0 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:15E04FBB1B28E3D2]}
   [junit4] OK      7.31s J0 | TestTlogReplica.testCreateDelete 
{seed=[EF0BEF7EDD81BE2:A1B6F3472FEA28FE]}
   [junit4] OK      11.8s J0 | TestTlogReplica.testRecovery
   [junit4] OK      11.4s J0 | TestTlogReplica.testKillLeader
   [junit4] OK      8.84s J0 | TestTlogReplica.testAddRemoveTlogReplica
   [junit4] OK      6.38s J0 | TestTlogReplica.testAddDocs
   [junit4] OK      6.50s J0 | TestTlogReplica.testDeleteById
   [junit4] OK      9.74s J0 | TestTlogReplica.testOnlyLeaderIndexes
   [junit4] OK      9.86s J0 | TestTlogReplica.testRemoveLeader
   [junit4] OK      9.21s J0 | 
TestTlogReplica.testOutOfOrderDBQWithInPlaceUpdates
   [junit4] OK      9.86s J0 | TestTlogReplica.testBasicLeaderElection
   [junit4] OK      4.39s J0 | TestTlogReplica.testRealTimeGet
   [junit4] Completed [5/5] on J0 in 116.58s, 13 tests
   [junit4] 
   [junit4] JVM J0:     0.67 ..   363.16 =   362.49s
   [junit4] JVM J1:     0.67 ..   248.63 =   247.96s
   [junit4] Execution time total: 6 minutes 3 seconds
   [junit4] Tests summary: 5 suites, 65 tests
   [junit4] Could not remove temporary path: 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1
 (java.nio.file.DirectoryNotEmptyException: Remaining files: 
[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp])
   [junit4] Could not remove temporary path: 
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0
 (java.nio.file.DirectoryNotEmptyException: Remaining files: 
[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp])
     [echo] 5 slowest tests:
[junit4:tophints] 624.59s | org.apache.solr.cloud.BasicDistributedZkTest
[junit4:tophints] 123.45s | org.apache.solr.cloud.TestTlogReplica
[junit4:tophints] 116.72s | org.apache.solr.update.SoftAutoCommitTest
[junit4:tophints] 115.68s | org.apache.solr.cloud.ZkFailoverTest
[junit4:tophints] 102.65s | 
org.apache.solr.cloud.autoscaling.SearchRateTriggerTest

-check-totals:

test-nocompile:

BUILD SUCCESSFUL
Total time: 6 minutes 5 seconds
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.TestTlogReplica
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
        at hudson.FilePath$34.hasMatch(FilePath.java:2678)
        at hudson.FilePath$34.invoke(FilePath.java:2557)
        at hudson.FilePath$34.invoke(FilePath.java:2547)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2918)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to Solaris VBOX
                at 
hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at 
hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1036)
                at hudson.FilePath.act(FilePath.java:1025)
                at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
                at 
hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
                at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
                at 
hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
                at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
                at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
                at hudson.model.Build$BuildExecution.post2(Build.java:186)
                at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
                at hudson.model.Run.execute(Run.java:1819)
                at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                at 
hudson.model.ResourceController.execute(ResourceController.java:97)
                at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2920)
        at hudson.remoting.UserRequest.perform(UserRequest.java:212)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no 
matches found within 10000
        at hudson.FilePath.act(FilePath.java:1038)
        at hudson.FilePath.act(FilePath.java:1025)
        at hudson.FilePath.validateAntFileMask(FilePath.java:2547)
        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
        at 
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
        at 
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
        at hudson.model.Build$BuildExecution.post2(Build.java:186)
        at 
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
        at hudson.model.Run.execute(Run.java:1819)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern 
"**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all 
warnings.
<Git Blamer> GIT_COMMIT=3c6fb974aefc50f411cba90b2b267804fbb6e1cb, 
workspace=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris
[WARNINGS] Computing warning deltas based on reference build #650
Recording test results
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to