Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/992/

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.test

Error Message:
Mismatch in counts between replicas

Stack Trace:
java.lang.AssertionError: Mismatch in counts between replicas
        at 
__randomizedtesting.SeedInfo.seed([2BFDE402CBB3335E:A3A9DBD8654F5EA6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at 
org.apache.solr.cloud.RecoveryZkTest.assertShardConsistency(RecoveryZkTest.java:143)
        at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:126)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 12739 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/init-core-data-001
   [junit4]   2> 1685776 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=3 numCloses=3
   [junit4]   2> 1685777 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using PointFields
   [junit4]   2> 1685779 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1685780 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/tempDir-001
   [junit4]   2> 1685780 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1685780 INFO  (Thread-2890) [    ] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1685780 INFO  (Thread-2890) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1685786 ERROR (Thread-2890) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1685880 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.ZkTestServer start zk server on port:45627
   [junit4]   2> 1685890 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1685890 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1685892 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@1629e524{/solr,null,AVAILABLE}
   [junit4]   2> 1685892 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@446800e8{/solr,null,AVAILABLE}
   [junit4]   2> 1685892 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@4e6a8406{HTTP/1.1,[http/1.1]}{127.0.0.1:35192}
   [junit4]   2> 1685893 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.e.j.s.Server Started @1689303ms
   [junit4]   2> 1685893 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=35192}
   [junit4]   2> 1685893 ERROR (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1685893 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@5d23c0de{HTTP/1.1,[http/1.1]}{127.0.0.1:58794}
   [junit4]   2> 1685893 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.e.j.s.Server Started @1689303ms
   [junit4]   2> 1685893 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, 
hostPort=58794}
   [junit4]   2> 1685893 ERROR (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1685897 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-03T00:38:50.177Z
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
6.7.0
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1685898 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2017-07-03T00:38:50.177Z
   [junit4]   2> 1685901 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1685901 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1685915 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1685915 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1685919 INFO  (jetty-launcher-2235-thread-2) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45627/solr
   [junit4]   2> 1685920 INFO  (jetty-launcher-2235-thread-1) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45627/solr
   [junit4]   2> 1685975 INFO  (jetty-launcher-2235-thread-2) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1685975 INFO  (jetty-launcher-2235-thread-1) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1685977 INFO  (jetty-launcher-2235-thread-2) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.OverseerElectionContext I am going to be 
the leader 127.0.0.1:58794_solr
   [junit4]   2> 1685977 INFO  (jetty-launcher-2235-thread-1) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:35192_solr
   [junit4]   2> 1685982 INFO  (jetty-launcher-2235-thread-2) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.Overseer Overseer 
(id=98241238142943237-127.0.0.1:58794_solr-n_0000000000) starting
   [junit4]   2> 1685994 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1686003 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1686012 INFO  (jetty-launcher-2235-thread-2) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:58794_solr
   [junit4]   2> 1686013 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1686014 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (1) -> (2)
   [junit4]   2> 1686103 INFO  (jetty-launcher-2235-thread-1) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/tempDir-001/node1/.
   [junit4]   2> 1686111 INFO  (jetty-launcher-2235-thread-2) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core 
definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/tempDir-001/node2/.
   [junit4]   2> 1686151 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1686153 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45627/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1686278 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried 
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1686314 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1686328 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 1686362 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
 to ./temp/Jetty_localhost_58734_hdfs____3k385b/webapp
   [junit4]   2> 1686925 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:58734
   [junit4]   2> 1687116 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1687118 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 1687129 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_57499_datanode____s992zd/webapp
   [junit4]   2> 1687610 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:57499
   [junit4]   2> 1687969 WARN  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1687971 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
jetty-6.1.26
   [junit4]   2> 1687987 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Extract 
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
 to ./temp/Jetty_localhost_42469_datanode____2574m8/webapp
   [junit4]   2> 1688169 INFO  (IPC Server handler 2 on 57072) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-0408aa7c-d5a7-4f49-8e3d-88c9a3e48464 node 
DatanodeRegistration(127.0.0.1:57552, 
datanodeUuid=3d9da445-d6e7-416e-a21a-bbf6eb237059, infoPort=34283, 
infoSecurePort=0, ipcPort=35494, 
storageInfo=lv=-56;cid=testClusterID;nsid=1261024213;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 1688169 INFO  (IPC Server handler 2 on 57072) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-d8019b74-9c83-426b-9f8c-42b2ab288188 node 
DatanodeRegistration(127.0.0.1:57552, 
datanodeUuid=3d9da445-d6e7-416e-a21a-bbf6eb237059, infoPort=34283, 
infoSecurePort=0, ipcPort=35494, 
storageInfo=lv=-56;cid=testClusterID;nsid=1261024213;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1688467 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] o.m.log 
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42469
   [junit4]   2> 1688776 INFO  (IPC Server handler 8 on 57072) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-19e3f59a-2551-4a00-b983-7d5636dde648 node 
DatanodeRegistration(127.0.0.1:53090, 
datanodeUuid=ed099e42-c80d-4f7e-bc64-e322ddd596be, infoPort=50166, 
infoSecurePort=0, ipcPort=51882, 
storageInfo=lv=-56;cid=testClusterID;nsid=1261024213;c=0), blocks: 0, 
hasStaleStorage: true, processing time: 1 msecs
   [junit4]   2> 1688776 INFO  (IPC Server handler 8 on 57072) [    ] 
BlockStateChange BLOCK* processReport: from storage 
DS-921e0cb7-2467-421a-bafa-b758de84f342 node 
DatanodeRegistration(127.0.0.1:53090, 
datanodeUuid=ed099e42-c80d-4f7e-bc64-e322ddd596be, infoPort=50166, 
infoSecurePort=0, ipcPort=51882, 
storageInfo=lv=-56;cid=testClusterID;nsid=1261024213;c=0), blocks: 0, 
hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 1688977 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1688979 INFO  (qtp2016258528-12976) [n:127.0.0.1:35192_solr    
] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1688982 INFO  
(OverseerThreadFactory-5169-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.CreateCollectionCmd Create collection 
recoverytest
   [junit4]   2> 1689124 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1689124 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1689135 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr    
] o.a.s.h.a.CoreAdminOperation core create command 
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1689135 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr    
] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 
2147483647 transient cores
   [junit4]   2> 1689243 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1689243 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1690150 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 1690154 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 6.7.0
   [junit4]   2> 1690158 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica1] Schema name=minimal
   [junit4]   2> 1690162 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1690162 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica1' using configuration from 
collection recoverytest, trusted=true
   [junit4]   2> 1690163 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
[recoverytest_shard1_replica2] Schema name=minimal
   [junit4]   2> 1690166 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema 
Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 1690167 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.CoreContainer 
Creating SolrCore 'recoverytest_shard1_replica2' using configuration from 
collection recoverytest, trusted=true
   [junit4]   2> 1690177 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:57072/data
   [junit4]   2> 1690178 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1690178 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1690178 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
[[recoverytest_shard1_replica2] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/tempDir-001/node2/recoverytest_shard1_replica2],
 dataDir=[hdfs://localhost:57072/data/recoverytest/core_node1/data/]
   [junit4]   2> 1690179 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77238d65
   [junit4]   2> 1690180 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node1/data/snapshot_metadata
   [junit4]   2> 1690183 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:57072/data
   [junit4]   2> 1690183 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1690183 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
solr.RecoveryStrategy.Builder
   [junit4]   2> 1690183 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
[[recoverytest_shard1_replica1] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001/tempDir-001/node1/recoverytest_shard1_replica1],
 dataDir=[hdfs://localhost:57072/data/recoverytest/core_node2/data/]
   [junit4]   2> 1690184 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX 
Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77238d65
   [junit4]   2> 1690184 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node2/data/snapshot_metadata
   [junit4]   2> 1690192 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1690192 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 1690192 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1690193 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1690193 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 1690193 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1690225 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1690229 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1690237 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node1/data
   [junit4]   2> 1690256 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node2/data
   [junit4]   2> 1690301 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node1/data/index
   [junit4]   2> 1690314 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory creating directory factory for path 
hdfs://localhost:57072/data/recoverytest/core_node2/data/index
   [junit4]   2> 1690315 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1690315 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 1690315 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1690322 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct 
memory allocation set to [true]
   [junit4]   2> 1690322 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of 
[16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 1690322 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 1690374 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1690420 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1690571 INFO  (IPC Server handler 9 on 57072) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53090 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-0408aa7c-d5a7-4f49-8e3d-88c9a3e48464:NORMAL:127.0.0.1:57552|RBW],
 
ReplicaUC[[DISK]DS-19e3f59a-2551-4a00-b983-7d5636dde648:NORMAL:127.0.0.1:53090|FINALIZED]]}
 size 0
   [junit4]   2> 1690571 INFO  (IPC Server handler 0 on 57072) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57552 is 
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-0408aa7c-d5a7-4f49-8e3d-88c9a3e48464:NORMAL:127.0.0.1:57552|RBW],
 
ReplicaUC[[DISK]DS-19e3f59a-2551-4a00-b983-7d5636dde648:NORMAL:127.0.0.1:53090|FINALIZED]]}
 size 0
   [junit4]   2> 1690605 INFO  (IPC Server handler 6 on 57072) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57552 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-d8019b74-9c83-426b-9f8c-42b2ab288188:NORMAL:127.0.0.1:57552|RBW],
 
ReplicaUC[[DISK]DS-921e0cb7-2467-421a-bafa-b758de84f342:NORMAL:127.0.0.1:53090|RBW]]}
 size 0
   [junit4]   2> 1690610 INFO  (IPC Server handler 7 on 57072) [    ] 
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53090 is 
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, 
primaryNodeIndex=-1, 
replicas=[ReplicaUC[[DISK]DS-d8019b74-9c83-426b-9f8c-42b2ab288188:NORMAL:127.0.0.1:57552|RBW],
 
ReplicaUC[[DISK]DS-921e0cb7-2467-421a-bafa-b758de84f342:NORMAL:127.0.0.1:53090|RBW]]}
 size 0
   [junit4]   2> 1690770 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1690770 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1690770 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.HdfsUpdateLog 
Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1690783 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1690783 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1690783 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.HdfsUpdateLog 
Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 1690786 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1690786 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1690797 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1690797 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1690848 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@4f0f6ba3[recoverytest_shard1_replica2] main]
   [junit4]   2> 1690850 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1690851 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1690851 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1690867 INFO  
(searcherExecutor-5175-thread-1-processing-n:127.0.0.1:58794_solr 
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SolrCore 
[recoverytest_shard1_replica2] Registered new searcher 
Searcher@4f0f6ba3[recoverytest_shard1_replica2] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1690874 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@22e53b64[recoverytest_shard1_replica1] main]
   [junit4]   2> 1690876 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf
   [junit4]   2> 1690876 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 1690876 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] 
o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1690883 INFO  
(searcherExecutor-5174-thread-1-processing-n:127.0.0.1:35192_solr 
x:recoverytest_shard1_replica1 s:shard1 c:recoverytest) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.c.SolrCore 
[recoverytest_shard1_replica1] Registered new searcher 
Searcher@22e53b64[recoverytest_shard1_replica1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1690884 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1571859815635877888
   [junit4]   2> 1690921 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for 
shard shard1: total=2 found=1 timeoutin=9997ms
   [junit4]   2> 1690928 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1571859815682015232
   [junit4]   2> 1691022 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691023 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691422 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1691422 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1691422 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/
   [junit4]   2> 1691423 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.PeerSync 
PeerSync: core=recoverytest_shard1_replica2 url=http://127.0.0.1:58794/solr 
START replicas=[http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/] 
nUpdates=100
   [junit4]   2> 1691429 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.u.PeerSync 
PeerSync: core=recoverytest_shard1_replica2 url=http://127.0.0.1:58794/solr 
DONE.  We have no versions.  sync failed.
   [junit4]   2> 1691433 INFO  (qtp2016258528-12982) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.c.S.Request [recoverytest_shard1_replica1]  webapp=/solr path=/get 
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
 status=0 QTime=1
   [junit4]   2> 1691434 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.SyncStrategy 
Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1691434 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we 
can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1691434 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1691441 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/ shard1
   [junit4]   2> 1691545 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691545 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691593 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.c.ZkController I 
am the leader, no recovery necessary
   [junit4]   2> 1691596 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica2] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2471
   [junit4]   2> 1691700 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691700 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1691942 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1  x:recoverytest_shard1_replica1] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
 status=0 QTime=2807
   [junit4]   2> 1691946 INFO  (qtp2016258528-12976) [n:127.0.0.1:35192_solr    
] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1691951 INFO  
(OverseerCollectionConfigSetProcessor-98241238142943237-127.0.0.1:58794_solr-n_0000000000)
 [n:127.0.0.1:58794_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1692048 INFO  
(zkCallback-2247-thread-1-processing-n:127.0.0.1:35192_solr) 
[n:127.0.0.1:35192_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1692048 INFO  
(zkCallback-2246-thread-1-processing-n:127.0.0.1:58794_solr) 
[n:127.0.0.1:58794_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: 
[WatchedEvent state:SyncConnected type:NodeDataChanged 
path:/collections/recoverytest/state.json] for collection [recoverytest] has 
occurred - updating... (live nodes size: [2])
   [junit4]   2> 1692947 INFO  (qtp2016258528-12976) [n:127.0.0.1:35192_solr    
] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2}
 status=0 QTime=3967
   [junit4]   2> 1692950 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.c.RecoveryZkTest Indexing 1200 documents
   [junit4]   2> 1693095 INFO  (qtp2016258528-12984) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[2-0
 (1571859817821110272)]} 0 90
   [junit4]   2> 1693095 INFO  (qtp1884874493-12983) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0 
(1571859817821110272)]} 0 127
   [junit4]   2> 1693096 INFO  (qtp2016258528-12972) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[1-0
 (1571859817813770240)]} 0 92
   [junit4]   2> 1693096 INFO  (qtp1884874493-12971) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0 
(1571859817813770240)]} 0 135
   [junit4]   2> 1693105 INFO  (qtp2016258528-12982) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[2-1
 (1571859817956376576)]} 0 6
   [junit4]   2> 1693106 INFO  (qtp1884874493-12981) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1 
(1571859817956376576)]} 0 8
   [junit4]   2> 1693107 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[1-1
 (1571859817958473728)]} 0 5
   [junit4]   2> 1693107 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1 
(1571859817958473728)]} 0 8
   [junit4]   2> 1693117 INFO  (qtp2016258528-12976) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[2-2
 (1571859817967910912)]} 0 6
   [junit4]   2> 1693118 INFO  (qtp1884874493-12973) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2 
(1571859817967910912)]} 0 9
   [junit4]   2> 1693118 INFO  (qtp2016258528-12970) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[1-2
 (1571859817968959488)]} 0 5
   [junit4]   2> 1693119 INFO  (qtp1884874493-12985) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2 
(1571859817968959488)]} 0 10
   [junit4]   2> 1693149 INFO  (qtp2016258528-12984) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[1-3
 (1571859817981542400)]} 0 26
   [junit4]   2> 1693149 INFO  (qtp2016258528-12972) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[2-3
 (1571859817981542401)]} 0 23
   [junit4]   2> 1693149 INFO  (qtp1884874493-12983) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3 
(1571859817981542400)]} 0 28
   [junit4]   2> 1693149 INFO  (qtp1884874493-12971) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3 
(1571859817981542401)]} 0 28
   [junit4]   2> 1693165 INFO  (qtp2016258528-12982) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[2-4
 (1571859818012999680)]} 0 11
   [junit4]   2> 1693165 INFO  (qtp1884874493-12981) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4 
(1571859818012999680)]} 0 13
   [junit4]   2> 1693170 ERROR 
(updateExecutor-2240-thread-2-processing-http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1
 x:recoverytest_shard1_replica2 r:core_node1 n:127.0.0.1:58794_solr s:shard1 
c:recoverytest) [n:127.0.0.1:58794_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.u.StreamingSolrClients error
   [junit4]   2> java.net.SocketException: Connection reset
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:210)
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]   2>        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693171 WARN  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
http://127.0.0.1:35192/solr
   [junit4]   2> java.net.SocketException: Connection reset
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:210)
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]   2>        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693171 ERROR (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/
   [junit4]   2> java.net.SocketException: Connection reset
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:210)
   [junit4]   2>        at 
java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]   2>        at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]   2>        at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]   2>        at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]   2>        at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]   2>        at 
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693172 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.c.ZkController Put replica core=recoverytest_shard1_replica1 
coreNodeName=core_node2 on 127.0.0.1:35192_solr into leader-initiated recovery.
   [junit4]   2> 1693172 INFO  (qtp1884874493-12977) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4 
(1571859818014048256)]} 0 20
   [junit4]   2> 1693172 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.e.j.s.AbstractConnector Stopped 
ServerConnector@4e6a8406{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 1693188 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=741939438
   [junit4]   2> 1693191 ERROR 
(updateExecutor-2240-thread-1-processing-http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1
 x:recoverytest_shard1_replica2 r:core_node1 n:127.0.0.1:58794_solr s:shard1 
c:recoverytest) [n:127.0.0.1:58794_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.u.StreamingSolrClients error
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693191 INFO  (qtp2016258528-12980) [n:127.0.0.1:35192_solr 
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]  
webapp=/solr path=/update 
params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:58794/solr/recoverytest_shard1_replica2/&wt=javabin&version=2}{add=[1-4
 (1571859818014048256)]} 0 36
   [junit4]   2> 1693192 WARN  (qtp1884874493-12973) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
http://127.0.0.1:35192/solr
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693192 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 1693192 ERROR (qtp1884874493-12973) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693192 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jvm
   [junit4]   2> 1693193 INFO  
(TEST-HdfsRecoveryZkTest.test-seed#[2BFDE402CBB3335E]) [    ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.jetty
   [junit4]   2> 1693196 INFO  (qtp1884874493-12973) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-4 
(-1571859818053894144)]} 0 6
   [junit4]   2> 1693200 ERROR 
(updateExecutor-2240-thread-1-processing-http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1
 x:recoverytest_shard1_replica2 r:core_node1 n:127.0.0.1:58794_solr s:shard1 
c:recoverytest) [n:127.0.0.1:58794_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.u.StreamingSolrClients error
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693201 WARN  (qtp1884874493-12983) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
http://127.0.0.1:35192/solr
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693201 ERROR (qtp1884874493-12983) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693202 INFO  (qtp1884874493-12983) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5 
(1571859818064379904)]} 0 1
   [junit4]   2> 1693204 INFO  
(updateExecutor-2240-thread-2-processing-x:recoverytest_shard1_replica2 
r:core_node1 http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1// 
n:127.0.0.1:58794_solr s:shard1 c:recoverytest) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.c.LeaderInitiatedRecoveryThread Put replica 
core=recoverytest_shard1_replica1 coreNodeName=core_node2 on 
127.0.0.1:35192_solr into leader-initiated recovery.
   [junit4]   2> 1693205 WARN  
(updateExecutor-2240-thread-2-processing-x:recoverytest_shard1_replica2 
r:core_node1 http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1// 
n:127.0.0.1:58794_solr s:shard1 c:recoverytest) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.c.LeaderInitiatedRecoveryThread Leader is publishing 
core=recoverytest_shard1_replica1 coreNodeName =core_node2 state=down on behalf 
of un-reachable replica 
http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/
   [junit4]   2> 1693205 ERROR 
(updateExecutor-2240-thread-1-processing-http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1
 x:recoverytest_shard1_replica2 r:core_node1 n:127.0.0.1:58794_solr s:shard1 
c:recoverytest) [n:127.0.0.1:58794_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.u.StreamingSolrClients error
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693206 WARN  (qtp1884874493-12971) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
http://127.0.0.1:35192/solr
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693206 ERROR (qtp1884874493-12971) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
replica http://127.0.0.1:35192/solr/recoverytest_shard1_replica1/
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693206 INFO  (qtp1884874493-12971) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]  
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6 
(1571859818068574208)]} 0 2
   [junit4]   2> 1693207 INFO  (coreCloseExecutor-5184-thread-1) 
[n:127.0.0.1:35192_solr c:recoverytest s:shard1 r:core_node2 
x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_replica1] 
 CLOSING SolrCore org.apache.solr.core.SolrCore@7ada98d8
   [junit4]   2> 1693207 ERROR 
(updateExecutor-2240-thread-3-processing-http:////127.0.0.1:35192//solr//recoverytest_shard1_replica1
 x:recoverytest_shard1_replica2 r:core_node1 n:127.0.0.1:58794_solr s:shard1 
c:recoverytest) [n:127.0.0.1:58794_solr c:recoverytest s:shard1 r:core_node1 
x:recoverytest_shard1_replica2] o.a.s.u.StreamingSolrClients error
   [junit4]   2> java.net.ConnectException: Connection refused (Connection 
refused)
   [junit4]   2>        at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
   [junit4]   2>        at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
   [junit4]   2>        at 
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]   2>        at java.net.Socket.connect(Socket.java:589)
   [junit4]   2>        at 
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
   [junit4]   2>        at 
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
   [junit4]   2>        at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
   [junit4]   2>        at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
   [junit4]   2>        at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2>        at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
   [junit4]   2>        at 
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
   [junit4]   2>        at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
   [junit4]   2>        at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2>        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 1693208 WARN  (qtp1884874493-12985) [n:127.0.0.1:58794_solr 
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] 
o.a.s.u.p.DistributedUpdateProcessor Error sending update to 
http://127.0.0.1:35192/solr
   [junit4]   2> java.ne

[...truncated too long message...]

r(MBeans.java:81)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2>        at 
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:203)
   [junit4]   2>        at 
org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.teardownClass(HdfsRecoveryZkTest.java:51)
   [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>        at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2>        at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2>        ... 54 more
   [junit4]   2> 1712952 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.c.ZkTestServer connecting to 127.0.0.1:45627 45627
   [junit4]   2> 1712953 INFO  
(SUITE-HdfsRecoveryZkTest-seed#[2BFDE402CBB3335E]-worker) [    ] 
o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_2BFDE402CBB3335E-001
   [junit4]   2> Jul 03, 2017 12:39:17 AM 
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 34 leaked 
thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
{rnd_b=FST50, _version_=PostingsFormat(name=Asserting), 
a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))),
 a_i=FST50, id=FST50}, docValues:{}, maxPointsInLeafNode=497, 
maxMBSortInHeap=7.052933139630767, 
sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=is-IS, 
timezone=Etc/GMT-8
   [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 
1.8.0_131 (64-bit)/cpus=4,threads=2,free=107934352,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [TestLRUCache, 
ZkControllerTest, TestPartialUpdateDeduplication, CleanupOldIndexTest, 
ClusterStateUpdateTest, TestConfig, TestMergePolicyConfig, JsonValidatorTest, 
SoftAutoCommitTest, TestConfigReload, LoggingHandlerTest, 
DeleteInactiveReplicaTest, TestSolrCoreSnapshots, TestBlendedInfixSuggestions, 
TestQueryWrapperFilter, AnalyticsMergeStrategyTest, 
TriLevelCompositeIdRoutingTest, TestSolrQueryParserDefaultOperatorResource, 
ZkCLITest, TestRandomRequestDistribution, TestSha256AuthenticationProvider, 
HdfsWriteToMultipleCollectionsTest, UUIDFieldTest, HighlighterTest, 
HdfsCollectionsAPIDistributedZkTest, TestCursorMarkWithoutUniqueKey, 
TestHashQParserPlugin, TestFieldSortValues, TestStandardQParsers, 
TestCollectionAPIs, LeaderFailoverAfterPartitionTest, 
DistributedIntervalFacetingTest, SmileWriterTest, TestCloudPivotFacet, 
TestChildDocTransformer, UniqFieldsUpdateProcessorFactoryTest, 
TestSortByMinMaxFunction, TestConfigOverlay, TestXmlQParser, 
RestartWhileUpdatingTest, SaslZkACLProviderTest, TestCoreDiscovery, 
AnalysisAfterCoreReloadTest, EchoParamsTest, TestFieldTypeResource, 
SpellCheckComponentTest, TestConfigSets, TestAtomicUpdateErrorCases, 
TestCopyFieldCollectionResource, SpatialFilterTest, AddBlockUpdateTest, 
TestDeleteCollectionOnDownNodes, TestWriterPerf, TestSolrDynamicMBean, 
TestSolrCloudWithDelegationTokens, CacheHeaderTest, RuleEngineTest, 
OutputWriterTest, DocValuesMultiTest, ConnectionManagerTest, 
TestStressVersions, DistributedFacetPivotLargeTest, OverseerTaskQueueTest, 
MoveReplicaTest, SimpleMLTQParserTest, HdfsThreadLeakTest, 
TestManagedSchemaThreadSafety, BasicAuthIntegrationTest, JSONWriterTest, 
TestMiniSolrCloudClusterSSL, TestLuceneMatchVersion, 
TolerantUpdateProcessorTest, FieldAnalysisRequestHandlerTest, 
ConjunctionSolrSpellCheckerTest, TestSchemaSimilarityResource, 
SuggesterTSTTest, MetricsHandlerTest, SolrIndexConfigTest, 
TestBinaryResponseWriter, SparseHLLTest, DistributedFacetPivotSmallTest, 
RAMDirectoryFactoryTest, TestHashPartitioner, ForceLeaderTest, 
LeaderInitiatedRecoveryOnShardRestartTest, CollectionStateFormat2Test, 
TestOmitPositions, SystemInfoHandlerTest, VersionInfoTest, 
TestQuerySenderListener, ClassificationUpdateProcessorTest, 
HighlighterMaxOffsetTest, SharedFSAutoReplicaFailoverUtilsTest, 
TestIntervalFaceting, TestDistributedStatsComponentCardinality, 
DocumentBuilderTest, TestQuerySenderNoQuery, TestShardHandlerFactory, 
OpenCloseCoreStressTest, SynonymTokenizerTest, TestHalfAndHalfDocValues, 
TestSchemalessBufferedUpdates, TestStressCloudBlindAtomicUpdates, 
RequiredFieldsTest, BigEndianAscendingWordSerializerTest, 
TestEmbeddedSolrServerConstructors, TestBinaryField, TestValueSourceCache, 
SolrCmdDistributorTest, ExternalFileFieldSortTest, TestIBSimilarityFactory, 
TestSubQueryTransformerDistrib, SegmentsInfoRequestHandlerTest, 
CoreAdminRequestStatusTest, TestReqParamsAPI, SyncSliceTest, 
LeaderElectionIntegrationTest, LeaderElectionTest, ZkSolrClientTest, 
ShardRoutingCustomTest, TestDistributedSearch, NoCacheHeaderTest, 
SchemaVersionSpecificBehaviorTest, TestWordDelimiterFilterFactory, 
DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, 
LukeRequestHandlerTest, DirectSolrSpellCheckerTest, 
DocumentAnalysisRequestHandlerTest, FileBasedSpellCheckerTest, 
SolrPluginUtilsTest, ReturnFieldsTest, JsonLoaderTest, TestLFUCache, 
CdcrRequestHandlerTest, CdcrVersionReplicationTest, 
ConcurrentDeleteAndCreateCollectionTest, CreateCollectionCleanupTest, 
CustomCollectionTest, DeleteStatusTest, DistribJoinFromCollectionTest, 
DistributedQueueTest, DocValuesNotIndexedTest, HttpPartitionTest, 
OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, 
RollingRestartTest, SharedFSAutoReplicaFailoverTest, SolrCloudExampleTest, 
TestCloudPseudoReturnFields, TestCloudRecovery, TestConfigSetsAPI, 
TestRequestForwarding, TestRequestStatusCollectionAPI, TestSSLRandomization, 
TestSizeLimitedDistributedMap, TestSolrCloudWithKerberosAlt, 
TestTolerantUpdateProcessorCloud, TlogReplayBufferedWhileIndexingTest, 
VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, 
HdfsBasicDistributedZkTest, HdfsChaosMonkeyNothingIsSafeTest, 
HdfsChaosMonkeySafeLeaderTest, HdfsNNFailoverTest, HdfsRecoveryZkTest]
   [junit4] Completed [544/713 (1!)] on J1 in 36.94s, 1 test, 1 failure <<< 
FAILURES!

[...truncated 44070 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to