[JENKINS] Lucene-Solr-6.6-Windows (64bit/jdk1.8.0_131) - Build # 16 - Still Unstable!

Policeman Jenkins Server Sat, 03 Jun 2017 09:17:46 -0700

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.6-Windows/16/
Java: 64bit/jdk1.8.0_131 -XX:+UseCompressedOops -XX:+UseG1GC


1 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryZkTest.test

Error Message:
Mismatch in counts between replicas

Stack Trace:
java.lang.AssertionError: Mismatch in counts between replicas
        at 
__randomizedtesting.SeedInfo.seed([59FA3534EB8F1295:D1AE0AEE45737F6D]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.RecoveryZkTest.assertShardConsistency(RecoveryZkTest.java:143)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:126)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12042 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
   [junit4]   2> Creating dataDir: 
C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\init-core-data-001
   [junit4]   2> 1302501 WARN  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
   [junit4]   2> 1302501 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1302506 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1302507 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001
   [junit4]   2> 1302508 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1302508 INFO  (Thread-2888) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1302508 INFO  (Thread-2888) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1302513 ERROR (Thread-2888) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1302609 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:63148
   [junit4]   2> 1302616 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c6e9ede5b0000, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1302619 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1302619 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1302620 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@43461e9f{/solr,null,AVAILABLE}
   [junit4]   2> 1302620 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@2ffb2bca{/solr,null,AVAILABLE}
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@47268a21{HTTP/1.1,[http/1.1]}{127.0.0.1:63153}
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@7a754b2d{HTTP/1.1,[http/1.1]}{127.0.0.1:63152}
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.e.j.s.Server Started @1306357ms
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.e.j.s.Server Started @1306357ms
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=63153}
   [junit4]   2> 1302628 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=63152}
   [junit4]   2> 1302628 ERROR (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.6.0
   [junit4]   2> 1302628 ERROR (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 
6.6.0
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-03T15:42:04.495Z
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1302629 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-06-03T15:42:04.495Z
   [junit4]   2> 1302633 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1302634 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1302642 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c6e9ede5b0002, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1302643 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c6e9ede5b0001, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1302646 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1302646 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1302647 INFO  (jetty-launcher-2098-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63148/solr
   [junit4]   2> 1302647 INFO  (jetty-launcher-2098-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63148/solr
   [junit4]   2> 1302650 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c6e9ede5b0003, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1302651 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] 
o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from 
client sessionid 0x15c6e9ede5b0004, likely client has closed socket
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
   [junit4]   2>        at 
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1302672 INFO  (jetty-launcher-2098-thread-2) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1302672 INFO  (jetty-launcher-2098-thread-1) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1302673 INFO  (jetty-launcher-2098-thread-2) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:63153_solr
   [junit4]   2> 1302673 INFO  (jetty-launcher-2098-thread-1) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:63152_solr
   [junit4]   2> 1302674 INFO  (jetty-launcher-2098-thread-2) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.Overseer Overseer 
(id=98074920509702149-127.0.0.1:63153_solr-n_0000000000) starting
   [junit4]   2> 1302674 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1302674 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1302682 INFO  (jetty-launcher-2098-thread-2) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:63153_solr
   [junit4]   2> 1302690 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1302690 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1302781 INFO  (jetty-launcher-2098-thread-2) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node1\.
   [junit4]   2> 1302783 INFO  (jetty-launcher-2098-thread-1) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node2\.
   [junit4]   2> 1302819 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1302820 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63148/solr ready
   [junit4]   2> 1302849 INFO  
(TEST-RecoveryZkTest.test-seed#[59FA3534EB8F1295]) [    ] o.a.s.SolrTestCaseJ4 
###Starting test
   [junit4]   2> 1302855 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1302860 INFO  
(OverseerThreadFactory-5547-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
recoverytest
   [junit4]   2> 1302967 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1302967 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1302968 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1302970 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1303074 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1303074 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1303998 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.6.0
   [junit4]   2> 1303998 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.6.0
   [junit4]   2> 1304006 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica2] Schema name=minimal
   [junit4]   2> 1304006 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica1] Schema name=minimal
   [junit4]   2> 1304015 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1304015 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica2' using configuration from 
collection recoverytest, trusted=true
   [junit4]   2> 1304015 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1304015 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica1' using configuration from 
collection recoverytest, trusted=true
   [junit4]   2> 1304015 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1304015 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1304015 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
[[recoverytest_shard1_replica2] ] Opening new SolrCore at 
[C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node2\recoverytest_shard1_replica2],
 
dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node2\.\recoverytest_shard1_replica2\data\]
   [junit4]   2> 1304015 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
[[recoverytest_shard1_replica1] ] Opening new SolrCore at 
[C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node1\recoverytest_shard1_replica1],
 
dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001\tempDir-001\node1\.\recoverytest_shard1_replica1\data\]
   [junit4]   2> 1304130 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1304130 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1304131 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1304131 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1304132 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1304132 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1304133 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@270dad72[recoverytest_shard1_replica2] main]
   [junit4]   2> 1304133 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1304133 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1304134 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@416da6fc[recoverytest_shard1_replica1] main]
   [junit4]   2> 1304135 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1304135 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1304135 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1304135 INFO  
(searcherExecutor-5552-thread-1-processing-n:127.0.0.1:63152_solr 
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
[recoverytest_shard1_replica2] Registered new searcher 
Searcher@270dad72[recoverytest_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1304136 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1569198729857073152
   [junit4]   2> 1304136 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1304136 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1304136 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1304136 INFO  
(searcherExecutor-5553-thread-1-processing-n:127.0.0.1:63153_solr 
x:recoverytest_shard1_replica1 s:shard1 c:recoverytest) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
[recoverytest_shard1_replica1] Registered new searcher 
Searcher@416da6fc[recoverytest_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1304136 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1569198729857073152
   [junit4]   2> 1304141 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1304141 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1304143 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/
   [junit4]   2> 1304143 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.PeerSync 
PeerSync: core=recoverytest_shard1_replica1 url=http://127.0.0.1:63153/solr 
START replicas=[http://127.0.0.1:63152/solr/recoverytest_shard1_replica2/] 
nUpdates=100
   [junit4]   2> 1304143 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.PeerSync 
PeerSync: core=recoverytest_shard1_replica1 url=http://127.0.0.1:63153/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 1304147 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.c.S.Request [recoverytest_shard1_replica2]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=0
   [junit4]   2> 1304147 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1304147 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1304147 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1304151 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/ shard1
   [junit4]   2> 1304253 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1304253 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1304303 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 1304307 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=1339
   [junit4]   2> 1304410 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1304410 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1305146 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2178
   [junit4]   2> 1305152 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1305247 INFO  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1305247 INFO  
(zkCallback-2110-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1306152 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=3296
   [junit4]   2> 1306153 INFO  
(TEST-RecoveryZkTest.test-seed#[59FA3534EB8F1295]) [    ] 
o.a.s.c.RecoveryZkTest Indexing 1350 documents
   [junit4]   2> 1306166 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-0
 (1569198731976245248)]} 0 2
   [junit4]   2> 1306166 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-0
 (1569198731976245249)]} 0 1
   [junit4]   2> 1306166 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 
(1569198731976245248)]} 0 9
   [junit4]   2> 1306166 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 
(1569198731976245249)]} 0 9
   [junit4]   2> 1306170 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-1
 (1569198731987779585)]} 0 0
   [junit4]   2> 1306170 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 
(1569198731987779585)]} 0 2
   [junit4]   2> 1306170 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-1
 (1569198731987779584)]} 0 1
   [junit4]   2> 1306171 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 
(1569198731987779584)]} 0 2
   [junit4]   2> 1306174 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-2
 (1569198731991973888)]} 0 0
   [junit4]   2> 1306175 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 
(1569198731991973888)]} 0 2
   [junit4]   2> 1306175 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-2
 (1569198731991973889)]} 0 0
   [junit4]   2> 1306176 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 
(1569198731991973889)]} 0 2
   [junit4]   2> 1306178 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-3
 (1569198731996168192)]} 0 0
   [junit4]   2> 1306179 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 
(1569198731996168192)]} 0 2
   [junit4]   2> 1306180 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-3
 (1569198731998265344)]} 0 0
   [junit4]   2> 1306180 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 
(1569198731998265344)]} 0 2
   [junit4]   2> 1306181 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-1
 (-1569198732000362496)]} 0 0
   [junit4]   2> 1306181 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-1 
(-1569198732000362496)]} 0 1
   [junit4]   2> 1306183 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-1
 (-1569198732001411072)]} 0 0
   [junit4]   2> 1306183 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-1 
(-1569198732001411072)]} 0 2
   [junit4]   2> 1306185 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-4
 (1569198732003508224)]} 0 0
   [junit4]   2> 1306185 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 
(1569198732003508224)]} 0 2
   [junit4]   2> 1306186 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-4
 (1569198732005605376)]} 0 0
   [junit4]   2> 1306186 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 
(1569198732005605376)]} 0 1
   [junit4]   2> 1306188 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-5
 (1569198732006653952)]} 0 0
   [junit4]   2> 1306188 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 
(1569198732006653952)]} 0 2
   [junit4]   2> 1306190 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-5
 (1569198732008751104)]} 0 0
   [junit4]   2> 1306190 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-5 
(1569198732008751104)]} 0 2
   [junit4]   2> 1306191 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-6
 (1569198732010848256)]} 0 0
   [junit4]   2> 1306191 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6 
(1569198732010848256)]} 0 1
   [junit4]   2> 1306193 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-6
 (1569198732011896832)]} 0 0
   [junit4]   2> 1306193 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-6 
(1569198732011896832)]} 0 2
   [junit4]   2> 1306195 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-2
 (-1569198732013993984)]} 0 0
   [junit4]   2> 1306195 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-2 
(-1569198732013993984)]} 0 2
   [junit4]   2> 1306196 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-2
 (-1569198732016091136)]} 0 0
   [junit4]   2> 1306196 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-2 
(-1569198732016091136)]} 0 1
   [junit4]   2> 1306198 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-7
 (1569198732017139712)]} 0 0
   [junit4]   2> 1306198 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-7 
(1569198732017139712)]} 0 2
   [junit4]   2> 1306200 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-7
 (1569198732019236864)]} 0 0
   [junit4]   2> 1306201 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-7 
(1569198732019236864)]} 0 2
   [junit4]   2> 1306202 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-8
 (1569198732021334016)]} 0 0
   [junit4]   2> 1306202 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-8 
(1569198732021334016)]} 0 2
   [junit4]   2> 1306204 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.su.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  webapp=/solr 
path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-8
 (1569198732023431168)]} 0 0
   [junit4]   2> 1306204 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-8 
(1569198732023431168)]} 0 2
   [junit4]   2> 1306205 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-9
 (1569198732025528320)]} 0 0
   [junit4]   2> 1306206 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-9 
(1569198732025528320)]} 0 1
   [junit4]   2> 1306208 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-4
 (-1569198732028674049)]} 0 0
   [junit4]   2> 1306208 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-9
 (1569198732028674048)]} 0 0
   [junit4]   2> 1306208 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-4 
(-1569198732028674049)]} 0 1
   [junit4]   2> 1306208 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-9 
(1569198732028674048)]} 0 1
   [junit4]   2> 1306212 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-10
 (1569198732031819776)]} 0 0
   [junit4]   2> 1306212 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-4
 (-1569198732031819777)]} 0 0
   [junit4]   2> 1306212 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-10 
(1569198732031819776)]} 0 2
   [junit4]   2> 1306212 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-4 
(-1569198732031819777)]} 0 2
   [junit4]   2> 1306215 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-5
 (-1569198732036014081)]} 0 0
   [junit4]   2> 1306215 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-10
 (1569198732036014080)]} 0 0
   [junit4]   2> 1306215 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-10 
(1569198732036014080)]} 0 1
   [junit4]   2> 1306215 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-5 
(-1569198732036014081)]} 0 1
   [junit4]   2> 1306219 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-5
 (-1569198732039159809)]} 0 0
   [junit4]   2> 1306219 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-11
 (1569198732039159808)]} 0 0
   [junit4]   2> 1306219 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-5 
(-1569198732039159809)]} 0 2
   [junit4]   2> 1306219 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-11 
(1569198732039159808)]} 0 2
   [junit4]   2> 1306222 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-12
 (1569198732043354112)]} 0 0
   [junit4]   2> 1306222 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-11
 (1569198732043354113)]} 0 0
   [junit4]   2> 1306222 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-12 
(1569198732043354112)]} 0 1
   [junit4]   2> 1306222 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-11 
(1569198732043354113)]} 0 1
   [junit4]   2> 1306225 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-12
 (1569198732046499840)]} 0 0
   [junit4]   2> 1306225 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-13
 (1569198732046499841)]} 0 0
   [junit4]   2> 1306226 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-12 
(1569198732046499840)]} 0 1
   [junit4]   2> 1306226 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-13 
(1569198732046499841)]} 0 1
   [junit4]   2> 1306228 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-13
 (1569198732049645568)]} 0 0
   [junit4]   2> 1306228 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-6
 (-1569198732049645569)]} 0 0
   [junit4]   2> 1306228 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-13 
(1569198732049645568)]} 0 1
   [junit4]   2> 1306228 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-6 
(-1569198732049645569)]} 0 1
   [junit4]   2> 1306235 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-6
 (-1569198732056985601)]} 0 0
   [junit4]   2> 1306235 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-14
 (1569198732056985600)]} 0 0
   [junit4]   2> 1306236 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-6 
(-1569198732056985601)]} 0 2
   [junit4]   2> 1306236 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-14 
(1569198732056985600)]} 0 2
   [junit4]   2> 1306238 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-15
 (1569198732060131329)]} 0 0
   [junit4]   2> 1306238 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-14
 (1569198732060131328)]} 0 0
   [junit4]   2> 1306238 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-15 
(1569198732060131329)]} 0 1
   [junit4]   2> 1306238 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-14 
(1569198732060131328)]} 0 1
   [junit4]   2> 1306242 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-10
 (-1569198732063277057)]} 0 0
   [junit4]   2> 1306242 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-15
 (1569198732063277056)]} 0 0
   [junit4]   2> 1306242 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-10 
(-1569198732063277057)]} 0 2
   [junit4]   2> 1306242 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-15 
(1569198732063277056)]} 0 2
   [junit4]   2> 1306245 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-16
 (1569198732067471360)]} 0 0
   [junit4]   2> 1306245 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-10
 (-1569198732067471361)]} 0 0
   [junit4]   2> 1306245 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-16 
(1569198732067471360)]} 0 1
   [junit4]   2> 1306245 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-10 
(-1569198732067471361)]} 0 1
   [junit4]   2> 1306248 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-16
 (-1569198732070617089)]} 0 0
   [junit4]   2> 1306248 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-16
 (1569198732070617088)]} 0 0
   [junit4]   2> 1306248 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-16 
(-1569198732070617089)]} 0 1
   [junit4]   2> 1306248 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-16 
(1569198732070617088)]} 0 1
   [junit4]   2> 1306252 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-16
 (-1569198732073762817)]} 0 0
   [junit4]   2> 1306252 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-17
 (1569198732073762816)]} 0 0
   [junit4]   2> 1306252 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-16 
(-1569198732073762817)]} 0 2
   [junit4]   2> 1306252 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-17 
(1569198732073762816)]} 0 2
   [junit4]   2> 1306256 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-17
 (1569198732079005696)]} 0 0
   [junit4]   2> 1306256 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-18
 (1569198732079005697)]} 0 0
   [junit4]   2> 1306256 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-17 
(1569198732079005696)]} 0 1
   [junit4]   2> 1306256 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-18 
(1569198732079005697)]} 0 1
   [junit4]   2> 1306260 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-18
 (-1569198732083200001)]} 0 0
   [junit4]   2> 1306260 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-18
 (1569198732083200000)]} 0 0
   [junit4]   2> 1306260 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-18 
(-1569198732083200001)]} 0 1
   [junit4]   2> 1306260 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-18 
(1569198732083200000)]} 0 1
   [junit4]   2> 1306262 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-18
 (-1569198732085297153)]} 0 0
   [junit4]   2> 1306262 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-19
 (1569198732085297152)]} 0 0
   [junit4]   2> 1306262 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-19 
(1569198732085297152)]} 0 1
   [junit4]   2> 1306262 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-18 
(-1569198732085297153)]} 0 1
   [junit4]   2> 1306265 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-19
 (-1569198732088442880)]} 0 0
   [junit4]   2> 1306265 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-19
 (1569198732088442881)]} 0 0
   [junit4]   2> 1306265 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-19 
(1569198732088442881)]} 0 1
   [junit4]   2> 1306265 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-19 
(-1569198732088442880)]} 0 1
   [junit4]   2> 1306269 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-19
 (-1569198732091588608)]} 0 0
   [junit4]   2> 1306269 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-20
 (1569198732091588609)]} 0 0
   [junit4]   2> 1306269 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-19 
(-1569198732091588608)]} 0 2
   [junit4]   2> 1306269 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-20 
(1569198732091588609)]} 0 2
   [junit4]   2> 1306273 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-20
 (1569198732096831489)]} 0 0
   [junit4]   2> 1306273 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-21
 (1569198732096831488)]} 0 0
   [junit4]   2> 1306273 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-21 
(1569198732096831488)]} 0 1
   [junit4]   2> 1306273 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-20 
(1569198732096831489)]} 0 1
   [junit4]   2> 1306277 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-21
 (-1569198732099977216)]} 0 0
   [junit4]   2> 1306277 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-21
 (1569198732099977217)]} 0 0
   [junit4]   2> 1306277 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-21 
(-1569198732099977216)]} 0 2
   [junit4]   2> 1306277 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-21 
(1569198732099977217)]} 0 2
   [junit4]   2> 1306280 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-21
 (-1569198732104171521)]} 0 0
   [junit4]   2> 1306280 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-22
 (1569198732104171520)]} 0 0
   [junit4]   2> 1306280 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-22 
(1569198732104171520)]} 0 1
   [junit4]   2> 1306280 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-21 
(-1569198732104171521)]} 0 1
   [junit4]   2> 1306283 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-22
 (1569198732107317249)]} 0 0
   [junit4]   2> 1306283 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-23
 (1569198732107317248)]} 0 0
   [junit4]   2> 1306283 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-23 
(1569198732107317248)]} 0 1
   [junit4]   2> 1306283 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-22 
(1569198732107317249)]} 0 1
   [junit4]   2> 1306286 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-24
 (1569198732110462976)]} 0 0
   [junit4]   2> 1306286 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-23
 (1569198732110462977)]} 0 0
   [junit4]   2> 1306286 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-24 
(1569198732110462976)]} 0 1
   [junit4]   2> 1306286 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-23 
(1569198732110462977)]} 0 1
   [junit4]   2> 1306289 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-24
 (1569198732113608704)]} 0 0
   [junit4]   2> 1306289 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-24
 (-1569198732113608705)]} 0 0
   [junit4]   2> 1306289 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-24 
(1569198732113608704)]} 0 1
   [junit4]   2> 1306289 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-24 
(-1569198732113608705)]} 0 1
   [junit4]   2> 1306292 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-24
 (-1569198732116754433)]} 0 0
   [junit4]   2> 1306292 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-25
 (1569198732116754432)]} 0 0
   [junit4]   2> 1306292 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-25 
(1569198732116754432)]} 0 1
   [junit4]   2> 1306292 INFO  (qtp368088523-12203) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-24 
(-1569198732116754433)]} 0 1
   [junit4]   2> 1306295 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-25
 (1569198732119900160)]} 0 0
   [junit4]   2> 1306295 INFO  (qtp507204493-12194) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-26
 (1569198732119900161)]} 0 0
   [junit4]   2> 1306295 INFO  (qtp368088523-12201) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-25 
(1569198732119900160)]} 0 1
   [junit4]   2> 1306295 INFO  (qtp368088523-12207) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-26 
(1569198732119900161)]} 0 1
   [junit4]   2> 1306299 INFO  (qtp507204493-12200) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-26
 (1569198732123045888)]} 0 0
   [junit4]   2> 1306299 INFO  (qtp507204493-12197) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-27
 (1569198732123045889)]} 0 0
   [junit4]   2> 1306299 INFO  (qtp368088523-12214) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-26 
(1569198732123045888)]} 0 2
   [junit4]   2> 1306299 INFO  (qtp368088523-12205) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-27 
(1569198732123045889)]} 0 2
   [junit4]   2> 1306303 INFO  (qtp507204493-12195) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-27
 (-1569198732127240193)]} 0 0
   [junit4]   2> 1306303 INFO  (qtp507204493-12198) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-27
 (1569198732127240192)]} 0 0
   [junit4]   2> 1306303 INFO  (qtp368088523-12213) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-27 
(-1569198732127240193)]} 0 2
   [junit4]   2> 1306303 INFO  (qtp368088523-12210) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-27 
(1569198732127240192)]} 0 2
   [junit4]   2> 1306307 INFO  (qtp507204493-12196) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-28
 (1569198732131434496)]} 0 0
   [junit4]   2> 1306307 INFO  (qtp507204493-12199) [n:127.0.0.1:63152_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:63153/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-27
 (-1569198732131434497)]} 0 0
   [junit4]   2> 1306307 INFO  (qtp368088523-12206) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-28 
(1569198732131434496)]} 0 2
   [junit4]   2> 1306307 INFO  (qtp368088523-12215) [n:127.0.0.1:63153_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-27 
(-1569198732131434497)]} 0 2
   [junit4]   2> 1306310 INFO  (q

[...truncated too long message...]

read-2) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1310264 INFO  (jetty-closer-2099-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 1310264 INFO  (jetty-closer-2099-thread-2) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 1310269 INFO  (coreCloseExecutor-5576-thread-1) 
[n:127.0.0.1:63152_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_replica2] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@6b85c828
   [junit4]   2> 1310274 INFO  (coreCloseExecutor-5576-thread-1) 
[n:127.0.0.1:63152_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.recoverytest.shard1.replica2
   [junit4]   2> 1310276 INFO  (coreCloseExecutor-5577-thread-1) 
[n:127.0.0.1:63153_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_replica1] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@7cb00798
   [junit4]   2> 1310278 INFO  (coreCloseExecutor-5577-thread-1) 
[n:127.0.0.1:63153_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric 
reporters for: solr.core.recoverytest.shard1.replica1
   [junit4]   2> 1310279 INFO  (jetty-closer-2099-thread-1) [    ] 
o.a.s.c.Overseer Overseer 
(id=98074920509702149-127.0.0.1:63153_solr-n_0000000000) closing
   [junit4]   2> 1310279 INFO  
(OverseerStateUpdate-98074920509702149-127.0.0.1:63153_solr-n_0000000000) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 
127.0.0.1:63153_solr
   [junit4]   2> 1310281 WARN  
(zkCallback-2109-thread-1-processing-n:127.0.0.1:63153_solr) 
[n:127.0.0.1:63153_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1310281 INFO  (jetty-closer-2099-thread-1) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2ffb2bca{/solr,null,UNAVAILABLE}
   [junit4]   2> 1310281 INFO  
(zkCallback-2121-thread-1-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:63152_solr
   [junit4]   2> 1310285 INFO  
(zkCallback-2121-thread-2-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (2) -> (1)
   [junit4]   2> 1316804 ERROR 
(recoveryExecutor-2119-thread-1-processing-n:127.0.0.1:63152_solr 
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest r:core_node1) 
[n:127.0.0.1:63152_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.c.RecoveryStrategy SolrCore not found - 
cannot recover:recoverytest_shard1_replica2
   [junit4]   2> 1316805 WARN  
(zkCallback-2121-thread-3-processing-n:127.0.0.1:63152_solr) 
[n:127.0.0.1:63152_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, 
but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 1316806 INFO  (jetty-closer-2099-thread-2) [    ] 
o.e.j.s.h.ContextHandler Stopped 
o.e.j.s.ServletContextHandler@2c31d56d{/solr,null,UNAVAILABLE}
   [junit4]   2> 1316807 ERROR 
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper 
server won't take any action on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1316807 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:63148 63148
   [junit4]   2> 1316808 INFO  (Thread-2888) [    ] o.a.s.c.ZkTestServer 
connecting to 127.0.0.1:63148 63148
   [junit4]   2> 1316809 WARN  (Thread-2888) [    ] o.a.s.c.ZkTestServer Watch 
limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/aliases.json
   [junit4]   2>        4       /solr/clusterprops.json
   [junit4]   2>        3       /solr/security.json
   [junit4]   2>        3       /solr/configs/conf
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/collections/recoverytest/state.json
   [junit4]   2>        4       /solr/clusterstate.json
   [junit4]   2>        2       
/solr/overseer_elect/election/98074920509702149-127.0.0.1:63153_solr-n_0000000000
   [junit4]   2>        2       
/solr/collections/recoverytest/leader_elect/shard1/election/98074920509702149-core_node2-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2>        4       /solr/collections
   [junit4]   2>        3       /solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 1316809 INFO  
(SUITE-RecoveryZkTest-seed#[59FA3534EB8F1295]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
C:\Users\jenkins\workspace\Lucene-Solr-6.6-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_59FA3534EB8F1295-001
   [junit4]   2> Jun 03, 2017 3:42:18 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene62, 
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=sq-AL, 
timezone=America/Costa_Rica
   [junit4]   2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_131 
(64-bit)/cpus=3,threads=1,free=128560168,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: 
[SegmentsInfoRequestHandlerTest, MultiTermTest, CurrencyFieldXmlFileTest, 
TestHighFrequencyDictionaryFactory, TestFastOutputStream, 
TestSchemaSimilarityResource, WrapperMergePolicyFactoryTest, 
HdfsRestartWhileUpdatingTest, SolrCoreMetricManagerTest, 
ParsingFieldUpdateProcessorsTest, BlockDirectoryTest, 
PKIAuthenticationIntegrationTest, DeleteLastCustomShardedReplicaTest, 
TestExactStatsCache, TestCloudManagedSchema, JSONWriterTest, 
OpenExchangeRatesOrgProviderTest, TestInPlaceUpdatesStandalone, 
TestGraphMLResponseWriter, TermsComponentTest, TestRestoreCore, 
TestSolrCloudWithKerberosAlt, DeleteStatusTest, RemoteQueryErrorTest, 
PrimitiveFieldTypeTest, TestPayloadScoreQParserPlugin, 
TestPKIAuthenticationPlugin, TestAddFieldRealTimeGet, TestReplicaProperties, 
RankQueryTest, TestDistributedStatsComponentCardinality, 
TestComplexPhraseLeadingWildcard, TestSchemalessBufferedUpdates, 
CloudExitableDirectoryReaderTest, CdcrRequestHandlerTest, OverseerStatusTest, 
TestDistribIDF, DirectoryFactoryTest, TestDistributedSearch, 
TestSerializedLuceneMatchVersion, HdfsCollectionsAPIDistributedZkTest, 
TestFieldTypeCollectionResource, DirectSolrSpellCheckerTest, 
V2ApiIntegrationTest, ManagedSchemaRoundRobinCloudTest, 
TestFieldCacheVsDocValues, TestComponentsName, TestBinaryField, 
ClassificationUpdateProcessorTest, VersionInfoTest, ResponseLogComponentTest, 
TestTestInjection, RulesTest, ZkNodePropsTest, TestUpdate, 
PeerSyncWithIndexFingerprintCachingTest, SpatialFilterTest, 
TestHashPartitioner, CollectionStateFormat2Test, TestSQLHandlerNonCloud, 
StressHdfsTest, TestImpersonationWithHadoopAuth, XmlUpdateRequestHandlerTest, 
SolrCoreTest, DistributedDebugComponentTest, BitVectorTest, 
TestPerFieldSimilarity, OutputWriterTest, SpatialRPTFieldTypeTest, 
TestConfigSetsAPIExclusivity, LeaderInitiatedRecoveryOnCommitTest, 
OverseerTest, ConcurrentDeleteAndCreateCollectionTest, TestStressReorder, 
HttpPartitionTest, CdcrReplicationDistributedZkTest, 
TestManagedStopFilterFactory, TestIBSimilarityFactory, PluginInfoTest, 
InfixSuggestersTest, MoreLikeThisHandlerTest, TestReloadAndDeleteDocs, 
TestFilteredDocIdSet, TestNumericTerms64, TriLevelCompositeIdRoutingTest, 
AnalysisAfterCoreReloadTest, DeleteReplicaTest, TestPartialUpdateDeduplication, 
DocumentAnalysisRequestHandlerTest, PeerSyncTest, TestAnalyzedSuggestions, 
IndexBasedSpellCheckerTest, TestExtendedDismaxParser, 
StandardRequestHandlerTest, ActionThrottleTest, TestLegacyFieldCache, 
TestPseudoReturnFields, TestDistribDocBasedVersion, DistributedQueueTest, 
TestSolrFieldCacheMBean, TestXmlQParser, TestDocumentBuilder, TestConfigSets, 
BadComponentTest, TestDynamicFieldCollectionResource, 
ConfigureRecoveryStrategyTest, MetricUtilsTest, PolyFieldTest, 
TestFieldCacheReopen, PingRequestHandlerTest, TestHdfsUpdateLog, 
RequestLoggingTest, OverseerModifyCollectionTest, TestRandomDVFaceting, 
TestManagedResource, TestDocTermOrds, TestEmbeddedSolrServerConstructors, 
TestCryptoKeys, TestRawTransformer, ZkControllerTest, TestCSVResponseWriter, 
IndexSchemaRuntimeFieldTest, TestStressVersions, TestSegmentSorting, TestSolrJ, 
TestDFISimilarityFactory, TestRTGBase, TestFieldCollectionResource, 
TestCustomStream, TestReRankQParserPlugin, TestCharFilters, 
CoreAdminRequestStatusTest, ChaosMonkeyNothingIsSafeTest, TestManagedSchemaAPI, 
DirectUpdateHandlerTest, TestConfigsApi, TestCoreContainer, 
ConjunctionSolrSpellCheckerTest, TestClassNameShortening, 
TestReplicationHandlerBackup, PathHierarchyTokenizerFactoryTest, 
TestFileDictionaryLookup, SaslZkACLProviderTest, BasicDistributedZkTest, 
TestMinMaxOnMultiValuedField, TestSolrIndexConfig, NumericFieldsTest, 
HighlighterTest, TestLeaderInitiatedRecoveryThread, TestLRUCache, 
FileUtilsTest, TestNamedUpdateProcessors, TestXIncludeConfig, 
TestImplicitCoreProperties, HdfsThreadLeakTest, AnalyticsQueryTest, 
MigrateRouteKeyTest, MoveReplicaTest, TestLazyCores, 
CollectionsAPIAsyncDistributedZkTest, HdfsRecoverLeaseTest, 
SchemaVersionSpecificBehaviorTest, ZkStateReaderTest, SolrPluginUtilsTest, 
TestJsonRequest, TestFieldCacheSort, RequiredFieldsTest, ShardRoutingTest, 
BadCopyFieldTest, TestExclusionRuleCollectionAccess, UnloadDistributedZkTest, 
SyncSliceTest, OpenCloseCoreStressTest, BasicZkTest, RecoveryZkTest]
   [junit4] Completed [321/711 (1!)] on J0 in 14.82s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 44724 lines...]

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[JENKINS] Lucene-Solr-6.6-Windows (64bit/jdk1.8.0_131) - Build # 16 - Still Unstable!

Reply via email to