Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/618/
Java: 64bit/jdk1.8.0_112 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:
Mismatch in counts between replicas

Stack Trace:
java.lang.AssertionError: Mismatch in counts between replicas
        at 
__randomizedtesting.SeedInfo.seed([979A93FEF1590B37:1FCEAC245FA566CF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.RecoveryZkTest.assertShardConsistency(RecoveryZkTest.java:143)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:126)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
        at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11582 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\init-core-data-001
   [junit4]   2> 1401741 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1401747 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001
   [junit4]   2> 1401747 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1401748 INFO  (Thread-3119) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1401748 INFO  (Thread-3119) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1401855 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:59156
   [junit4]   2> 1401892 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1401895 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@71127320{/solr,null,AVAILABLE}
   [junit4]   2> 1401895 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@55fef8dd{SSL,[ssl, 
http/1.1]}{127.0.0.1:59160}
   [junit4]   2> 1401896 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.e.j.s.Server Started @1406632ms
   [junit4]   2> 1401896 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=59160}
   [junit4]   2> 1401896 ERROR (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1401896 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1401896 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1401896 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1401897 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1401897 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-12-12T13:54:45.403Z
   [junit4]   2> 1401897 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@201b04dd{/solr,null,AVAILABLE}
   [junit4]   2> 1401900 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.AbstractConnector Started ServerConnector@6724aee4{SSL,[ssl, 
http/1.1]}{127.0.0.1:59163}
   [junit4]   2> 1401900 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.Server Started @1406635ms
   [junit4]   2> 1401900 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=59163}
   [junit4]   2> 1401900 ERROR (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1401900 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.4.0
   [junit4]   2> 1401901 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1401901 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1401901 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2016-12-12T13:54:45.407Z
   [junit4]   2> 1401918 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1401918 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1401947 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1401950 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59156/solr
   [junit4]   2> 1401951 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1401951 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59156/solr
   [junit4]   2> 1401971 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x158f350718d0003, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1402012 INFO  (jetty-launcher-2042-thread-1) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:59160_solr
   [junit4]   2> 1402014 INFO  (jetty-launcher-2042-thread-2) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:59163_solr
   [junit4]   2> 1402016 INFO  (jetty-launcher-2042-thread-1) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.Overseer Overseer 
(id=97094918816464902-127.0.0.1:59160_solr-n_0000000000) starting
   [junit4]   2> 1402018 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1402020 INFO  
(zkCallback-2052-thread-1-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1402032 INFO  (jetty-launcher-2042-thread-1) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:59160_solr
   [junit4]   2> 1402036 INFO  
(OverseerStateUpdate-97094918816464902-127.0.0.1:59160_solr-n_0000000000) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1402037 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1402137 INFO  (jetty-launcher-2042-thread-1) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node1\.
   [junit4]   2> 1402169 INFO  (jetty-launcher-2042-thread-2) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node2\.
   [junit4]   2> 1402181 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1402183 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59156/solr ready
   [junit4]   2> 1402230 INFO  
(TEST-RecoveryZkTest.test-seed#[979A93FEF1590B37]) [    ] o.a.s.SolrTestCaseJ4 
###Starting test
   [junit4]   2> 1402289 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1402293 INFO  
(OverseerThreadFactory-6053-thread-1-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
recoverytest
   [junit4]   2> 1402420 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1402459 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1402570 INFO  
(zkCallback-2052-thread-3-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1402570 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1403440 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1403448 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica1] Schema name=minimal
   [junit4]   2> 1403457 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1403466 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica1' using configuration from 
collection recoverytest
   [junit4]   2> 1403466 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.SolrCore [[recoverytest_shard1_replica1] ] Opening new SolrCore at 
[C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node1\recoverytest_shard1_replica1],
 
dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node1\.\recoverytest_shard1_replica1\data\]
   [junit4]   2> 1403477 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 1403488 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica2] Schema name=minimal
   [junit4]   2> 1403491 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1403491 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1403491 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1403493 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica2' using configuration from 
collection recoverytest
   [junit4]   2> 1403493 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1403493 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1403493 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.c.SolrCore [[recoverytest_shard1_replica2] ] Opening new SolrCore at 
[C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node2\recoverytest_shard1_replica2],
 
dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001\tempDir-001\node2\.\recoverytest_shard1_replica2\data\]
   [junit4]   2> 1403496 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@68e69203[recoverytest_shard1_replica1] main]
   [junit4]   2> 1403498 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1403498 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1403498 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1403498 INFO  
(searcherExecutor-6058-thread-1-processing-n:127.0.0.1:59160_solr 
x:recoverytest_shard1_replica1 s:shard1 c:recoverytest r:core_node1) 
[n:127.0.0.1:59160_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_replica1] 
Registered new searcher Searcher@68e69203[recoverytest_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1403500 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1553518702893203456
   [junit4]   2> 1403509 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9999ms
   [junit4]   2> 1403520 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1403520 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1403541 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1403541 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1403542 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@1a57645a[recoverytest_shard1_replica2] main]
   [junit4]   2> 1403543 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1403544 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1403544 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1403544 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1553518702939340800
   [junit4]   2> 1403545 INFO  
(searcherExecutor-6061-thread-1-processing-n:127.0.0.1:59163_solr 
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest r:core_node2) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_replica2] 
Registered new searcher Searcher@1a57645a[recoverytest_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1403615 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1403615 INFO  
(zkCallback-2052-thread-1-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404010 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1404010 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1404010 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.SyncStrategy Sync replicas to 
https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/
   [junit4]   2> 1404010 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica1 
url=https://127.0.0.1:59160/solr START 
replicas=[https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/] 
nUpdates=100
   [junit4]   2> 1404011 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.PeerSync PeerSync: core=recoverytest_shard1_replica1 
url=https://127.0.0.1:59160/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 1404025 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.c.S.Request [recoverytest_shard1_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 1404026 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the 
next candidate
   [junit4]   2> 1404026 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1404032 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/ shard1
   [junit4]   2> 1404138 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404138 INFO  
(zkCallback-2052-thread-3-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404183 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1404188 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=1768
   [junit4]   2> 1404290 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404290 INFO  
(zkCallback-2052-thread-1-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404559 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2098
   [junit4]   2> 1404563 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1404659 INFO  
(zkCallback-2052-thread-3-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1404659 INFO  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1405563 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=3274
   [junit4]   2> 1405566 INFO  
(TEST-RecoveryZkTest.test-seed#[979A93FEF1590B37]) [    ] 
o.a.s.c.RecoveryZkTest Indexing 700 documents
   [junit4]   2> 1405601 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-0
 (1553518705079484416)]} 0 3
   [junit4]   2> 1405602 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 
(1553518705079484416)]} 0 16
   [junit4]   2> 1405606 INFO  (qtp577026899-12400) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-0
 (1553518705086824448)]} 0 0
   [junit4]   2> 1405608 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 
(1553518705086824448)]} 0 15
   [junit4]   2> 1405610 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-1
 (1553518705102553088)]} 0 0
   [junit4]   2> 1405610 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 
(1553518705102553088)]} 0 3
   [junit4]   2> 1405612 INFO  (qtp577026899-12403) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-1
 (1553518705104650240)]} 0 0
   [junit4]   2> 1405612 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 
(1553518705104650240)]} 0 3
   [junit4]   2> 1405615 INFO  (qtp577026899-12416) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-1
 (-1553518705108844544)]} 0 0
   [junit4]   2> 1405616 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-1 
(-1553518705108844544)]} 0 2
   [junit4]   2> 1405618 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-1
 (-1553518705109893120)]} 0 0
   [junit4]   2> 1405618 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-1 
(-1553518705109893120)]} 0 3
   [junit4]   2> 1405620 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-2
 (1553518705114087424)]} 0 0
   [junit4]   2> 1405620 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 
(1553518705114087424)]} 0 3
   [junit4]   2> 1405621 INFO  (qtp577026899-12405) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-2
 (1553518705116184576)]} 0 0
   [junit4]   2> 1405622 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 
(1553518705116184576)]} 0 1
   [junit4]   2> 1405625 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-3
 (1553518705118281728)]} 0 0
   [junit4]   2> 1405626 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 
(1553518705118281728)]} 0 3
   [junit4]   2> 1405628 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-3
 (1553518705120378880)]} 0 0
   [junit4]   2> 1405628 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 
(1553518705120378880)]} 0 4
   [junit4]   2> 1405629 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-3
 (-1553518705124573184)]} 0 0
   [junit4]   2> 1405629 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-3 
(-1553518705124573184)]} 0 1
   [junit4]   2> 1405632 INFO  (qtp577026899-12403) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-3
 (-1553518705127718912)]} 0 0
   [junit4]   2> 1405633 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-3 
(-1553518705127718912)]} 0 1
   [junit4]   2> 1405634 INFO  (qtp577026899-12416) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-4
 (1553518705128767488)]} 0 0
   [junit4]   2> 1405635 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 
(1553518705128767488)]} 0 3
   [junit4]   2> 1405636 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-4
 (1553518705130864640)]} 0 0
   [junit4]   2> 1405637 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 
(1553518705130864640)]} 0 2
   [junit4]   2> 1405639 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-5
 (1553518705135058944)]} 0 0
   [junit4]   2> 1405640 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-5 
(1553518705135058944)]} 0 1
   [junit4]   2> 1405641 INFO  (qtp577026899-12405) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-5
 (1553518705136107520)]} 0 0
   [junit4]   2> 1405642 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 
(1553518705136107520)]} 0 2
   [junit4]   2> 1405644 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-6
 (1553518705139253248)]} 0 0
   [junit4]   2> 1405645 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-6 
(1553518705139253248)]} 0 2
   [junit4]   2> 1405645 INFO  (qtp577026899-12400) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-6
 (1553518705141350400)]} 0 0
   [junit4]   2> 1405646 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6 
(1553518705141350400)]} 0 2
   [junit4]   2> 1405649 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-7
 (1553518705143447552)]} 0 0
   [junit4]   2> 1405650 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-7 
(1553518705143447552)]} 0 3
   [junit4]   2> 1405651 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-7
 (1553518705146593280)]} 0 0
   [junit4]   2> 1405652 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-7 
(1553518705146593280)]} 0 2
   [junit4]   2> 1405654 INFO  (qtp577026899-12403) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-8
 (1553518705149739008)]} 0 0
   [junit4]   2> 1405655 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-8 
(1553518705149739008)]} 0 3
   [junit4]   2> 1405656 INFO  (qtp577026899-12416) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-8
 (1553518705151836160)]} 0 0
   [junit4]   2> 1405657 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-8 
(1553518705151836160)]} 0 2
   [junit4]   2> 1405660 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-7
 (-1553518705154981888)]} 0 0
   [junit4]   2> 1405660 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-7 
(-1553518705154981888)]} 0 3
   [junit4]   2> 1405662 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-7
 (-1553518705158127616)]} 0 0
   [junit4]   2> 1405663 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-7 
(-1553518705158127616)]} 0 3
   [junit4]   2> 1405664 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-9
 (1553518705161273344)]} 0 0
   [junit4]   2> 1405665 INFO  (qtp249880025-12394) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-9 
(1553518705161273344)]} 0 2
   [junit4]   2> 1405667 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-9
 (1553518705163370496)]} 0 0
   [junit4]   2> 1405668 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-9 
(1553518705163370496)]} 0 2
   [junit4]   2> 1405668 INFO  (qtp577026899-12400) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-10
 (1553518705165467648)]} 0 0
   [junit4]   2> 1405668 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-10 
(1553518705165467648)]} 0 1
   [junit4]   2> 1405678 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-8
 (-1553518705170710529)]} 0 0
   [junit4]   2> 1405678 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-10
 (1553518705170710528)]} 0 0
   [junit4]   2> 1405678 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-8 
(-1553518705170710529)]} 0 6
   [junit4]   2> 1405678 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-10 
(1553518705170710528)]} 0 6
   [junit4]   2> 1405681 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-11
 (1553518705178050560)]} 0 0
   [junit4]   2> 1405682 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-11 
(1553518705178050560)]} 0 2
   [junit4]   2> 1405682 INFO  (qtp577026899-12416) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-8
 (-1553518705178050561)]} 0 0
   [junit4]   2> 1405684 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-8 
(-1553518705178050561)]} 0 3
   [junit4]   2> 1405689 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-12
 (1553518705184342016)]} 0 0
   [junit4]   2> 1405689 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-12 
(1553518705184342016)]} 0 3
   [junit4]   2> 1405689 INFO  (qtp577026899-12405) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-11
 (1553518705184342017)]} 0 0
   [junit4]   2> 1405690 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-11 
(1553518705184342017)]} 0 3
   [junit4]   2> 1405692 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-10
 (-1553518705190633473)]} 0 0
   [junit4]   2> 1405692 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-10 
(-1553518705190633473)]} 0 1
   [junit4]   2> 1405692 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-12
 (1553518705190633472)]} 0 0
   [junit4]   2> 1405692 INFO  (qtp249880025-12394) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-12 
(1553518705190633472)]} 0 1
   [junit4]   2> 1405697 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-10
 (-1553518705194827776)]} 0 0
   [junit4]   2> 1405697 INFO  (qtp577026899-12400) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-13
 (1553518705194827777)]} 0 0
   [junit4]   2> 1405697 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-10 
(-1553518705194827776)]} 0 2
   [junit4]   2> 1405697 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-13 
(1553518705194827777)]} 0 2
   [junit4]   2> 1405702 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-11
 (-1553518705201119232)]} 0 0
   [junit4]   2> 1405702 INFO  (qtp577026899-12403) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-13
 (1553518705201119233)]} 0 0
   [junit4]   2> 1405703 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-11 
(-1553518705201119232)]} 0 2
   [junit4]   2> 1405703 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-13 
(1553518705201119233)]} 0 2
   [junit4]   2> 1405705 INFO  (qtp577026899-12416) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-11
 (-1553518705204264960)]} 0 0
   [junit4]   2> 1405705 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-14
 (1553518705204264961)]} 0 0
   [junit4]   2> 1405705 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-11 
(-1553518705204264960)]} 0 1
   [junit4]   2> 1405705 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-14 
(1553518705204264961)]} 0 1
   [junit4]   2> 1405709 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-15
 (1553518705208459264)]} 0 0
   [junit4]   2> 1405710 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-15 
(1553518705208459264)]} 0 1
   [junit4]   2> 1405710 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-14
 (1553518705208459265)]} 0 0
   [junit4]   2> 1405710 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-14 
(1553518705208459265)]} 0 1
   [junit4]   2> 1405715 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-15
 (1553518705212653569)]} 0 1
   [junit4]   2> 1405717 INFO  (qtp249880025-12394) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-15 
(1553518705212653569)]} 0 4
   [junit4]   2> 1405717 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-16
 (1553518705212653568)]} 0 2
   [junit4]   2> 1405717 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-16 
(1553518705212653568)]} 0 5
   [junit4]   2> 1405724 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-17
 (1553518705221042177)]} 0 0
   [junit4]   2> 1405724 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-17 
(1553518705221042177)]} 0 3
   [junit4]   2> 1405725 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-16
 (1553518705221042176)]} 0 0
   [junit4]   2> 1405725 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-16 
(1553518705221042176)]} 0 4
   [junit4]   2> 1405733 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-18
 (1553518705233625089)]} 0 0
   [junit4]   2> 1405734 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-18 
(1553518705233625089)]} 0 3
   [junit4]   2> 1405734 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-17
 (1553518705233625088)]} 0 0
   [junit4]   2> 1405734 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-17 
(1553518705233625088)]} 0 3
   [junit4]   2> 1405737 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-14
 (-1553518705237819393)]} 0 0
   [junit4]   2> 1405737 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-14 
(-1553518705237819393)]} 0 1
   [junit4]   2> 1405737 INFO  (qtp577026899-12403) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-18
 (1553518705237819392)]} 0 0
   [junit4]   2> 1405737 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-18 
(1553518705237819392)]} 0 2
   [junit4]   2> 1405743 INFO  (qtp577026899-12417) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-19
 (1553518705242013697)]} 0 0
   [junit4]   2> 1405743 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-14
 (-1553518705242013696)]} 0 0
   [junit4]   2> 1405743 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-14 
(-1553518705242013696)]} 0 3
   [junit4]   2> 1405743 INFO  (qtp249880025-12394) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-19 
(1553518705242013697)]} 0 3
   [junit4]   2> 1405748 INFO  (qtp577026899-12405) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-19
 (1553518705248305153)]} 0 0
   [junit4]   2> 1405749 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-19 
(1553518705248305153)]} 0 2
   [junit4]   2> 1405749 INFO  (qtp577026899-12415) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-20
 (1553518705248305152)]} 0 0
   [junit4]   2> 1405749 INFO  (qtp249880025-12395) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-20 
(1553518705248305152)]} 0 3
   [junit4]   2> 1405751 INFO  (qtp577026899-12398) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-20
 (1553518705252499456)]} 0 0
   [junit4]   2> 1405752 INFO  (qtp249880025-12413) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-20 
(1553518705252499456)]} 0 1
   [junit4]   2> 1405752 INFO  (qtp577026899-12400) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-17
 (-1553518705253548032)]} 0 0
   [junit4]   2> 1405752 INFO  (qtp249880025-12414) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-17 
(-1553518705253548032)]} 0 1
   [junit4]   2> 1405755 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-17
 (-1553518705256693760)]} 0 0
   [junit4]   2> 1405756 INFO  (qtp249880025-12393) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-17 
(-1553518705256693760)]} 0 2
   [junit4]   2> 1405760 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-21
 (1553518705258790912)]} 0 0
   [junit4]   2> 1405760 INFO  (qtp249880025-12388) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-21 
(1553518705258790912)]} 0 3
   [junit4]   2> 1405762 INFO  (qtp577026899-12401) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-21
 (1553518705262985216)]} 0 0
   [junit4]   2> 1405763 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-21 
(1553518705262985216)]} 0 3
   [junit4]   2> 1405764 INFO  (qtp577026899-12399) [n:127.0.0.1:59163_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:59160/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-21
 (-1553518705265082368)]} 0 0
   [junit4]   2> 1405765 INFO  (qtp249880025-12391) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-21 
(-1553518705265082368)]} 0 2
   [junit4]   2> 1405772 WARN  
(updateExecutor-2046-thread-1-processing-https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2
 x:recoverytest_shard1_replica1 r:core_node1 n:127.0.0.1:59160_solr s:shard1 
c:recoverytest) [n:127.0.0.1:59160_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica1] o.a.s.c.s.i.ConcurrentUpdateSolrClient Failed 
to parse error response from 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2 due to: 
java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in 
not in 'javabin' format
   [junit4]   2> 1405773 ERROR 
(updateExecutor-2046-thread-1-processing-https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2
 x:recoverytest_shard1_replica1 r:core_node1 n:127.0.0.1:59160_solr s:shard1 
c:recoverytest) [n:127.0.0.1:59160_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica1] o.a.s.u.StreamingSolrClients error
   [junit4]   2> 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:59163/solr/recoverytest_shard1_replica2: 
Service Unavailable
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> request: 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=https%3A%2F%2F127.0.0.1%3A59160%2Fsolr%2Frecoverytest_shard1_replica1%2F&wt=javabin&version=2
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405773 WARN  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
https://127.0.0.1:59163/solr
   [junit4]   2> 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:59163/solr/recoverytest_shard1_replica2: 
Service Unavailable
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> request: 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=https%3A%2F%2F127.0.0.1%3A59160%2Fsolr%2Frecoverytest_shard1_replica1%2F&wt=javabin&version=2
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405773 ERROR (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/
   [junit4]   2> 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:59163/solr/recoverytest_shard1_replica2: 
Service Unavailable
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> request: 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/update?update.distrib=FROMLEADER&distrib.from=https%3A%2F%2F127.0.0.1%3A59160%2Fsolr%2Frecoverytest_shard1_replica1%2F&wt=javabin&version=2
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405773 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.ZkController Put replica core=recoverytest_shard1_replica2 
coreNodeName=core_node2 on 127.0.0.1:59163_solr into leader-initiated recovery.
   [junit4]   2> 1405773 INFO  (qtp249880025-12392) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-21 
(-1553518705271373824)]} 0 5
   [junit4]   2> 1405775 ERROR 
(updateExecutor-2046-thread-2-processing-https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2
 x:recoverytest_shard1_replica1 r:core_node1 n:127.0.0.1:59160_solr s:shard1 
c:recoverytest) [n:127.0.0.1:59160_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica1] o.a.s.u.StreamingSolrClients error
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:59163 
failed to respond
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405776 WARN  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
https://127.0.0.1:59163/solr
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:59163 
failed to respond
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405777 ERROR (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/
   [junit4]   2> org.apache.http.NoHttpResponseException: 127.0.0.1:59163 
failed to respond
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1405777 INFO  (qtp249880025-12412) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-22 
(1553518705275568128)]} 0 5
   [junit4]   2> 1405777 INFO  
(TEST-RecoveryZkTest.test-seed#[979A93FEF1590B37]) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@6724aee4{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1405782 INFO  
(TEST-RecoveryZkTest.test-seed#[979A93FEF1590B37]) [    ] o.a.s.c.CoreContainer 
Shutting down CoreContainer instance=496394507
   [junit4]   2> 1405790 INFO  
(updateExecutor-2046-thread-1-processing-x:recoverytest_shard1_replica1 
r:core_node1 https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2// 
n:127.0.0.1:59160_solr s:shard1 c:recoverytest) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.LeaderInitiatedRecoveryThread Put replica 
core=recoverytest_shard1_replica2 coreNodeName=core_node2 on 
127.0.0.1:59163_solr into leader-initiated recovery.
   [junit4]   2> 1405790 WARN  
(updateExecutor-2046-thread-1-processing-x:recoverytest_shard1_replica1 
r:core_node1 https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2// 
n:127.0.0.1:59160_solr s:shard1 c:recoverytest) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.LeaderInitiatedRecoveryThread Leader is publishing 
core=recoverytest_shard1_replica2 coreNodeName =core_node2 state=down on behalf 
of un-reachable replica 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/
   [junit4]   2> 1405792 INFO  
(updateExecutor-2046-thread-1-processing-x:recoverytest_shard1_replica1 
r:core_node1 https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2// 
n:127.0.0.1:59160_solr s:shard1 c:recoverytest) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.LeaderInitiatedRecoveryThread 
LeaderInitiatedRecoveryThread-recoverytest_shard1_replica2 started running to 
send REQUESTRECOVERY command to 
https://127.0.0.1:59163/solr/recoverytest_shard1_replica2/; will try for a max 
of 600 secs
   [junit4]   2> 1405793 INFO  
(updateExecutor-2046-thread-1-processing-x:recoverytest_shard1_replica1 
r:core_node1 https:////127.0.0.1:59163//solr//recoverytest_shard1_replica2// 
n:127.0.0.1:59160_solr s:shard1 c:recoverytest) [n:127.0.0.1:59160_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica1] 
o.a.s.c.LeaderInitiatedRecoveryThread Asking core=recoverytest_shard1_replica2 
coreNodeName=core_node2 on https://127.0.0.1:59163/solr to recover
   [junit4]   2> 1405804 INFO  (coreCloseExecutor-6068-thread-1) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_replica2] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@3fcdd585
   [junit4]   2> 1405805 INFO  (coreCloseExecutor-6068-thread-1) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.u.DirectUpdateHandler2 Committing on 
IndexWriter close.
   [junit4]   2> 1405805 INFO  (coreCloseExecutor-6068-thread-1) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.u.SolrIndexWriter Calling setCommitData 
with IW:org.apache.solr.update.SolrIndexWriter@25481ae3
   [junit4]   2> 1405819 WARN  
(zkCallback-2053-thread-1-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr   

[...truncated too long message...]

9 INFO  (jetty-launcher-2042-thread-1) [    ] o.e.j.s.AbstractConnector Stopped 
ServerConnector@48cba9bd{SSL,[ssl, http/1.1]}{127.0.0.1:59163}
   [junit4]   2> 1414671 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=147865016
   [junit4]   2> 1414671 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.AbstractConnector Stopped ServerConnector@55fef8dd{SSL,[ssl, 
http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1414671 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=513653463
   [junit4]   2> 1414676 INFO  (coreCloseExecutor-6082-thread-1) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_replica2] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@37c79a73
   [junit4]   2> 1414676 INFO  (coreCloseExecutor-6083-thread-1) 
[n:127.0.0.1:59160_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_replica1] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@3bca3655
   [junit4]   2> 1414684 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.a.s.c.Overseer Overseer 
(id=97094918816464902-127.0.0.1:59160_solr-n_0000000000) closing
   [junit4]   2> 1414685 INFO  
(OverseerStateUpdate-97094918816464902-127.0.0.1:59160_solr-n_0000000000) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:59160_solr
   [junit4]   2> 1414687 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x158f350718d0006, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1414687 WARN  
(zkCallback-2052-thread-2-processing-n:127.0.0.1:59160_solr) 
[n:127.0.0.1:59160_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1414689 ERROR 
(recoveryExecutor-2062-thread-1-processing-n:127.0.0.1:59163_solr 
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest r:core_node2) 
[n:127.0.0.1:59163_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica2] o.a.s.c.RecoveryStrategy SolrCore not found - 
cannot recover:recoverytest_shard1_replica2
   [junit4]   2> 1414691 INFO  (jetty-launcher-2042-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@71127320{/solr,null,UNAVAILABLE}
   [junit4]   2> 1414693 INFO  
(zkCallback-2064-thread-3-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1414693 INFO  
(zkCallback-2064-thread-2-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:59163_solr
   [junit4]   2> 1414693 WARN  
(zkCallback-2064-thread-3-processing-n:127.0.0.1:59163_solr) 
[n:127.0.0.1:59163_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1416194 INFO  (jetty-launcher-2042-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@59f158b7{/solr,null,UNAVAILABLE}
   [junit4]   2> 1416196 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:59156 59156
   [junit4]   2> 1416197 INFO  (Thread-3119) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:59156 59156
   [junit4]   2> 1416198 WARN  (Thread-3119) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/aliases.json
   [junit4]   2>        3       /solr/security.json
   [junit4]   2>        3       /solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/collections/recoverytest/state.json
   [junit4]   2>        4       /solr/clusterstate.json
   [junit4]   2>        4       /solr/clusterprops.json
   [junit4]   2>        2       
/solr/collections/recoverytest/leader_elect/shard1/election/97094918816464902-core_node1-n_0000000000
   [junit4]   2>        2       
/solr/overseer_elect/election/97094918816464902-127.0.0.1:59160_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        24      /solr/overseer/queue
   [junit4]   2>        11      /solr/overseer/collection-queue-work
   [junit4]   2>        4       /solr/collections
   [junit4]   2>        3       /solr/live_nodes
   [junit4]   2>        3       /solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 1416199 INFO  
(SUITE-RecoveryZkTest-seed#[979A93FEF1590B37]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.RecoveryZkTest_979A93FEF1590B37-001
   [junit4]   2> Dec 12, 2016 1:54:59 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), 
sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=und, 
timezone=America/Argentina/San_Juan
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_112 
(64-bit)/cpus=3,threads=1,free=226495024,total=440401920
   [junit4]   2> NOTE: All tests run in this JVM: [PreAnalyzedFieldTest, 
BasicDistributedZk2Test, HdfsSyncSliceTest, TestRuleBasedAuthorizationPlugin, 
ClassificationUpdateProcessorFactoryTest, TestAuthorizationFramework, 
DeleteInactiveReplicaTest, FieldMutatingUpdateProcessorTest, 
DistributedTermsComponentTest, RegexBoostProcessorTest, TestSolrXml, 
TestRequestForwarding, TestQuerySenderListener, ReplicationFactorTest, 
TestMergePolicyConfig, TestCloudManagedSchema, TestSolrConfigHandler, 
TestWriterPerf, TermVectorComponentDistributedTest, TestRebalanceLeaders, 
TestMinMaxOnMultiValuedField, TestQuerySenderNoQuery, 
HdfsWriteToMultipleCollectionsTest, TestReload, TestSolrCoreSnapshots, 
DistanceUnitsTest, TestMiniSolrCloudCluster, EchoParamsTest, RuleEngineTest, 
TestFieldTypeResource, TestQueryUtils, JSONWriterTest, CollectionReloadTest, 
EnumFieldTest, SliceStateTest, StatsReloadRaceTest, CdcrReplicationHandlerTest, 
TestLeaderElectionZkExpiry, SolrIndexConfigTest, TestManagedSchema, 
TestRangeQuery, TestFiltering, TestQueryWrapperFilter, BasicZkTest, 
WordBreakSolrSpellCheckerTest, TolerantUpdateProcessorTest, 
TestSolrDeletionPolicy2, TestSearchPerf, AutoCommitTest, TestPushWriter, 
TestUniqueKeyFieldResource, BooleanFieldTest, HdfsChaosMonkeySafeLeaderTest, 
CurrencyFieldOpenExchangeTest, MinimalSchemaTest, TestFreeTextSuggestions, 
ForceLeaderTest, NoCacheHeaderTest, TestReqParamsAPI, 
DistributedQueryComponentOptimizationTest, TestCustomStream, 
TestBlendedInfixSuggestions, ConnectionReuseTest, 
CoreMergeIndexesAdminHandlerTest, TestPivotHelperCode, SpellCheckCollatorTest, 
TestBlobHandler, OverseerModifyCollectionTest, TestMaxScoreQueryParser, 
TestSystemIdResolver, JsonLoaderTest, TestFastWriter, TestSolrCoreProperties, 
DistanceFunctionTest, WrapperMergePolicyFactoryTest, JavabinLoaderTest, 
DateRangeFieldTest, SharedFSAutoReplicaFailoverTest, HLLUtilTest, 
DistribJoinFromCollectionTest, DistributedVersionInfoTest, TestQueryTypes, 
BlockJoinFacetDistribTest, TestCloudPseudoReturnFields, ConvertedLegacyTest, 
TestConfig, RequestHandlersTest, TestDistributedGrouping, TestBadConfig, 
ZkSolrClientTest, BlockCacheTest, TestConfigSetsAPIZkFailure, 
TestClusterProperties, TestIndexSearcher, TestFieldCache, 
ParsingFieldUpdateProcessorsTest, ResourceLoaderTest, SolrInfoMBeanTest, 
TestMacroExpander, SolrCloudExampleTest, SuggesterWFSTTest, 
TestCloudSchemaless, TestSchemaResource, BitVectorTest, 
TestSolrQueryParserDefaultOperatorResource, TestStressUserVersions, 
TestJsonFacetRefinement, CacheHeaderTest, CircularListTest, 
TestChildDocTransformer, TestInitParams, ScriptEngineTest, 
TestAddFieldRealTimeGet, CachingDirectoryFactoryTest, 
DocExpirationUpdateProcessorFactoryTest, TestFieldCacheWithThreads, 
HighlighterConfigTest, TestBackupRepositoryFactory, CursorMarkTest, 
HdfsRecoveryZkTest, TestDefaultSearchFieldResource, 
TestManagedSynonymFilterFactory, PeerSyncReplicationTest, TestRandomFlRTGCloud, 
SearchHandlerTest, DeleteShardTest, TestTestInjection, SimpleMLTQParserTest, 
TestSmileRequest, AnalysisErrorHandlingTest, SolrCoreCheckLockOnStartupTest, 
TestRawTransformer, TestSchemaVersionResource, TestCoreDiscovery, 
TimerUtilsTest, SimpleCollectionCreateDeleteTest, TestJsonRequest, 
TestLMJelinekMercerSimilarityFactory, TestAuthenticationFramework, 
NotRequiredUniqueKeyTest, BasicFunctionalityTest, 
TestPHPSerializedResponseWriter, CollectionTooManyReplicasTest, 
TestFieldCacheSortRandom, TestLMDirichletSimilarityFactory, MBeansHandlerTest, 
TestRecovery, TestSizeLimitedDistributedMap, TestSuggestSpellingConverter, 
URLClassifyProcessorTest, UpdateParamsTest, CSVRequestHandlerTest, TestUtils, 
ShardRoutingCustomTest, TestStressCloudBlindAtomicUpdates, TestRecoveryHdfs, 
DirectoryFactoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, 
StatsComponentTest, DistributedFacetPivotSmallAdvancedTest, 
HdfsTlogReplayBufferedWhileIndexingTest, TestSubQueryTransformerCrossCore, 
ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, 
CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, 
LeaderElectionIntegrationTest, ShardRoutingTest, RecoveryZkTest]
   [junit4] Completed [276/660 (1!)] on J1 in 15.00s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 63682 lines...]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to