Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/631/
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
Error Message:
1 thread leaked from SUITE scope at
org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest: 1) Thread[id=4949,
name=OverseerHdfsCoreFailoverThread-97183716480974859-127.0.0.1:45024_solr-n_0000000002,
state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at
java.lang.Thread.sleep(Native Method) at
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
at java.lang.Thread.run(Thread.java:745)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE
scope at org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest:
1) Thread[id=4949,
name=OverseerHdfsCoreFailoverThread-97183716480974859-127.0.0.1:45024_solr-n_0000000002,
state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([B7D3BF2CEA67C129]:0)
Build Log:
[...truncated 11279 lines...]
[junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest
[junit4] 2> Creating dataDir:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/init-core-data-001
[junit4] 2> 641542 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via:
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 641544 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/tempDir-001
[junit4] 2> 641544 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 641545 INFO (Thread-964) [ ] o.a.s.c.ZkTestServer client
port:0.0.0.0/0.0.0.0:0
[junit4] 2> 641545 INFO (Thread-964) [ ] o.a.s.c.ZkTestServer Starting
server
[junit4] 2> 641645 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.ZkTestServer start zk server on port:54324
[junit4] 2> 641658 INFO (jetty-launcher-692-thread-1) [ ]
o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 641658 INFO (jetty-launcher-692-thread-2) [ ]
o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 641660 INFO (jetty-launcher-692-thread-2) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@381db14f{/solr,null,AVAILABLE}
[junit4] 2> 641665 INFO (jetty-launcher-692-thread-2) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@337cad39{SSL,[ssl,
http/1.1]}{127.0.0.1:45024}
[junit4] 2> 641665 INFO (jetty-launcher-692-thread-2) [ ]
o.e.j.s.Server Started @646346ms
[junit4] 2> 641665 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
hostPort=45024}
[junit4] 2> 641667 INFO (jetty-launcher-692-thread-1) [ ]
o.e.j.s.h.ContextHandler Started
o.e.j.s.ServletContextHandler@5803a847{/solr,null,AVAILABLE}
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.e.j.s.AbstractConnector Started ServerConnector@64296fcd{SSL,[ssl,
http/1.1]}{127.0.0.1:56347}
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.e.j.s.Server Started @646355ms
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
hostPort=56347}
[junit4] 2> 641674 ERROR (jetty-launcher-692-thread-1) [ ]
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
6.4.0
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 641674 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2016-12-28T06:17:10.220Z
[junit4] 2> 641675 ERROR (jetty-launcher-692-thread-2) [ ]
o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be
missing or incomplete.
[junit4] 2> 641675 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version
6.4.0
[junit4] 2> 641675 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 641675 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 641675 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
2016-12-28T06:17:10.221Z
[junit4] 2> 641691 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 641700 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 641711 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params:
socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 641718 INFO (jetty-launcher-692-thread-1) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54324/solr
[junit4] 2> 641721 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params:
socketTimeout=340000&connTimeout=45000&retry=true
[junit4] 2> 641726 INFO (jetty-launcher-692-thread-2) [ ]
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54324/solr
[junit4] 2> 641730 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ]
o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from
client sessionid 0x159441342cd0004, likely client has closed socket
[junit4] 2> at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 641781 INFO (jetty-launcher-692-thread-2)
[n:127.0.0.1:45024_solr ] o.a.s.c.OverseerElectionContext I am going to be
the leader 127.0.0.1:45024_solr
[junit4] 2> 641782 INFO (jetty-launcher-692-thread-2)
[n:127.0.0.1:45024_solr ] o.a.s.c.Overseer Overseer
(id=97183716480974854-127.0.0.1:45024_solr-n_0000000000) starting
[junit4] 2> 641784 INFO (jetty-launcher-692-thread-1)
[n:127.0.0.1:56347_solr ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:56347_solr
[junit4] 2> 641787 INFO
(zkCallback-703-thread-1-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (1)
[junit4] 2> 641787 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (0) -> (1)
[junit4] 2> 641797 INFO (jetty-launcher-692-thread-2)
[n:127.0.0.1:45024_solr ] o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/127.0.0.1:45024_solr
[junit4] 2> 641798 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (1) -> (2)
[junit4] 2> 641805 INFO
(zkCallback-703-thread-1-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from
ZooKeeper... (1) -> (2)
[junit4] 2> 641839 INFO (jetty-launcher-692-thread-1)
[n:127.0.0.1:56347_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/tempDir-001/node1/.
[junit4] 2> 641921 INFO (jetty-launcher-692-thread-2)
[n:127.0.0.1:45024_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
definitions underneath
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/tempDir-001/node2/.
[junit4] 2> 642257 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 642258 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54324/solr ready
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 642307 WARN
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.h.m.i.MetricsConfig Cannot locate configuration: tried
hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 642314 WARN
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 642316 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
jetty-6.1.26
[junit4] 2> 642328 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Extract
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs
to ./temp/Jetty_localhost_32949_hdfs____1eenhl/webapp
[junit4] 2> 642728 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:32949
[junit4] 2> 642866 WARN
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 642876 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
jetty-6.1.26
[junit4] 2> 642893 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Extract
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
to ./temp/Jetty_localhost_54880_datanode____.7qtkrs/webapp
[junit4] 2> 643311 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54880
[junit4] 2> 643453 WARN
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 643461 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
jetty-6.1.26
[junit4] 2> 643482 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Extract
jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode
to ./temp/Jetty_localhost_35803_datanode____.8mb9q8/webapp
[junit4] 2> 643558 INFO (IPC Server handler 3 on 46631) [ ]
BlockStateChange BLOCK* processReport: from storage
DS-7191c964-ddf5-48e6-bf7c-5543920d9f36 node
DatanodeRegistration(127.0.0.1:58399,
datanodeUuid=f9915712-d78f-4689-a0c8-20521d0231dc, infoPort=57667,
infoSecurePort=0, ipcPort=36601,
storageInfo=lv=-56;cid=testClusterID;nsid=1909039308;c=0), blocks: 0,
hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 643558 INFO (IPC Server handler 3 on 46631) [ ]
BlockStateChange BLOCK* processReport: from storage
DS-1b7e9668-aea3-4c9b-939a-e0a248489b8a node
DatanodeRegistration(127.0.0.1:58399,
datanodeUuid=f9915712-d78f-4689-a0c8-20521d0231dc, infoPort=57667,
infoSecurePort=0, ipcPort=36601,
storageInfo=lv=-56;cid=testClusterID;nsid=1909039308;c=0), blocks: 0,
hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 643989 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ] o.m.log
Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35803
[junit4] 2> 644192 INFO (IPC Server handler 7 on 46631) [ ]
BlockStateChange BLOCK* processReport: from storage
DS-b81d6228-6f50-4490-8009-4933d077b8f3 node
DatanodeRegistration(127.0.0.1:60883,
datanodeUuid=28c7f2fb-2370-4e70-898a-19abc6561c15, infoPort=46074,
infoSecurePort=0, ipcPort=38957,
storageInfo=lv=-56;cid=testClusterID;nsid=1909039308;c=0), blocks: 0,
hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 644192 INFO (IPC Server handler 7 on 46631) [ ]
BlockStateChange BLOCK* processReport: from storage
DS-9af62e21-05f1-421d-9185-4c869df6f991 node
DatanodeRegistration(127.0.0.1:60883,
datanodeUuid=28c7f2fb-2370-4e70-898a-19abc6561c15, infoPort=46074,
infoSecurePort=0, ipcPort=38957,
storageInfo=lv=-56;cid=testClusterID;nsid=1909039308;c=0), blocks: 0,
hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 644325 INFO
(TEST-HdfsRecoveryZkTest.test-seed#[B7D3BF2CEA67C129]) [ ]
o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 644393 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params
replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2
and sendToOCPQueue=true
[junit4] 2> 644395 INFO
(OverseerThreadFactory-1806-thread-1-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.CreateCollectionCmd Create collection
recoverytest
[junit4] 2> 644526 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
[junit4] 2> 644581 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2
[junit4] 2> 644686 INFO
(zkCallback-703-thread-3-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 644686 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 645551 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 6.4.0
[junit4] 2> 645567 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema
[recoverytest_shard1_replica2] Schema name=minimal
[junit4] 2> 645570 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 645571 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.CoreContainer
Creating SolrCore 'recoverytest_shard1_replica2' using configuration from
collection recoverytest
[junit4] 2> 645594 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SolrConfig
Using Lucene MatchVersion: 6.4.0
[junit4] 2> 645617 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema
[recoverytest_shard1_replica1] Schema name=minimal
[junit4] 2> 645620 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.s.IndexSchema
Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 645624 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:46631/data
[junit4] 2> 645625 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 645625 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.SolrCore
[[recoverytest_shard1_replica2] ] Opening new SolrCore at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/tempDir-001/node2/recoverytest_shard1_replica2],
dataDir=[hdfs://localhost:46631/data/recoverytest/core_node1/data/]
[junit4] 2> 645628 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.CoreContainer
Creating SolrCore 'recoverytest_shard1_replica1' using configuration from
collection recoverytest
[junit4] 2> 645631 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:46631/data
[junit4] 2> 645631 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
[junit4] 2> 645631 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SolrCore
[[recoverytest_shard1_replica1] ] Opening new SolrCore at
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_B7D3BF2CEA67C129-001/tempDir-001/node1/recoverytest_shard1_replica1],
dataDir=[hdfs://localhost:46631/data/recoverytest/core_node2/data/]
[junit4] 2> 645632 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node1/data/snapshot_metadata
[junit4] 2> 645636 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node2/data/snapshot_metadata
[junit4] 2> 645683 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 645683 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[16777216] will allocate [1] slabs and use ~[16777216] bytes
[junit4] 2> 645683 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
[junit4] 2> 645691 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 645691 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[16777216] will allocate [1] slabs and use ~[16777216] bytes
[junit4] 2> 648142 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 648143 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 648150 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node2/data
[junit4] 2> 648156 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node1/data
[junit4] 2> 648226 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node2/data/index
[junit4] 2> 648228 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory creating directory factory for path
hdfs://localhost:46631/data/recoverytest/core_node1/data/index
[junit4] 2> 648234 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 648234 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[16777216] will allocate [1] slabs and use ~[16777216] bytes
[junit4] 2> 648246 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 648260 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct
memory allocation set to [true]
[junit4] 2> 648260 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of
[16777216] will allocate [1] slabs and use ~[16777216] bytes
[junit4] 2> 648271 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.s.b.BlockDirectory Block cache on write is disabled
[junit4] 2> 648530 INFO (IPC Server handler 5 on 46631) [ ]
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60883 is
added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-1b7e9668-aea3-4c9b-939a-e0a248489b8a:NORMAL:127.0.0.1:58399|RBW],
ReplicaUC[[DISK]DS-b81d6228-6f50-4490-8009-4933d077b8f3:NORMAL:127.0.0.1:60883|FINALIZED]]}
size 0
[junit4] 2> 648532 INFO (IPC Server handler 9 on 46631) [ ]
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58399 is
added to blk_1073741825_1001 size 71
[junit4] 2> 648576 INFO (IPC Server handler 2 on 46631) [ ]
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:58399 is
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-9af62e21-05f1-421d-9185-4c869df6f991:NORMAL:127.0.0.1:60883|RBW],
ReplicaUC[[DISK]DS-7191c964-ddf5-48e6-bf7c-5543920d9f36:NORMAL:127.0.0.1:58399|FINALIZED]]}
size 0
[junit4] 2> 648579 INFO (IPC Server handler 6 on 46631) [ ]
BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60883 is
added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-9af62e21-05f1-421d-9185-4c869df6f991:NORMAL:127.0.0.1:60883|RBW],
ReplicaUC[[DISK]DS-7191c964-ddf5-48e6-bf7c-5543920d9f36:NORMAL:127.0.0.1:58399|FINALIZED]]}
size 0
[junit4] 2> 648644 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 648644 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 648644 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.HdfsUpdateLog
Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 648651 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.UpdateHandler
Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
[junit4] 2> 648651 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog
Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 648651 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.HdfsUpdateLog
Initializing HdfsUpdateLog: tlogDfsReplication=3
[junit4] 2> 648669 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 648669 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 648670 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker
Hard AutoCommit: disabled
[junit4] 2> 648670 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.CommitTracker
Soft AutoCommit: disabled
[junit4] 2> 648807 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@35adf6e8[recoverytest_shard1_replica1] main]
[junit4] 2> 648811 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@61341108[recoverytest_shard1_replica2] main]
[junit4] 2> 648812 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf
[junit4] 2> 648812 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 648815 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 648815 INFO
(searcherExecutor-1812-thread-1-processing-n:127.0.0.1:56347_solr
x:recoverytest_shard1_replica1 s:shard1 c:recoverytest) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SolrCore
[recoverytest_shard1_replica1] Registered new searcher
Searcher@35adf6e8[recoverytest_shard1_replica1]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 648815 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1554939471316647936
[junit4] 2> 648825 INFO
(zkCallback-703-thread-1-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 648828 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 648829 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for
shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 648831 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase:
/configs/conf
[junit4] 2> 648832 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 648832 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2]
o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 648833 INFO
(searcherExecutor-1811-thread-1-processing-n:127.0.0.1:45024_solr
x:recoverytest_shard1_replica2 s:shard1 c:recoverytest) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.SolrCore
[recoverytest_shard1_replica2] Registered new searcher
Searcher@61341108[recoverytest_shard1_replica2]
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 648833 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.u.UpdateLog
Could not find max version in index or recent updates, using new clock
1554939471335522304
[junit4] 2> 649434 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 649435 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 649435 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy
Sync replicas to https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/
[junit4] 2> 649435 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.PeerSync
PeerSync: core=recoverytest_shard1_replica1 url=https://127.0.0.1:56347/solr
START replicas=[https://127.0.0.1:45024/solr/recoverytest_shard1_replica2/]
nUpdates=100
[junit4] 2> 649435 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.u.PeerSync
PeerSync: core=recoverytest_shard1_replica1 url=https://127.0.0.1:56347/solr
DONE. We have no versions. sync failed.
[junit4] 2> 649446 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.c.S.Request [recoverytest_shard1_replica2] webapp=/solr path=/get
params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2}
status=0 QTime=2
[junit4] 2> 649447 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SyncStrategy
Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 649447 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we
can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 649453 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1]
o.a.s.c.ShardLeaderElectionContext I am the new leader:
https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/ shard1
[junit4] 2> 649557 INFO
(zkCallback-703-thread-1-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 649558 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 649606 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.ZkController I
am the leader, no recovery necessary
[junit4] 2> 649609 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica1&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
status=0 QTime=5027
[junit4] 2> 649711 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 649723 INFO
(zkCallback-703-thread-2-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 649850 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=recoverytest_shard1_replica2&action=CREATE&numShards=1&collection=recoverytest&shard=shard1&wt=javabin&version=2}
status=0 QTime=5323
[junit4] 2> 649854 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr ]
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most
30 seconds. Check all shard replicas
[junit4] 2> 649967 INFO
(zkCallback-702-thread-1-processing-n:127.0.0.1:56347_solr)
[n:127.0.0.1:56347_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 649968 INFO
(zkCallback-703-thread-2-processing-n:127.0.0.1:45024_solr)
[n:127.0.0.1:45024_solr ] o.a.s.c.c.ZkStateReader A cluster state change:
[WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/recoverytest/state.json] for collection [recoverytest] has
occurred - updating... (live nodes size: [2])
[junit4] 2> 650492 INFO
(OverseerCollectionConfigSetProcessor-97183716480974854-127.0.0.1:45024_solr-n_0000000000)
[n:127.0.0.1:45024_solr ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may
have disconnected from ZooKeeper
[junit4] 2> 650855 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&name=recoverytest&action=CREATE&numShards=1&wt=javabin&version=2}
status=0 QTime=6461
[junit4] 2> 650870 INFO
(TEST-HdfsRecoveryZkTest.test-seed#[B7D3BF2CEA67C129]) [ ]
o.a.s.c.RecoveryZkTest Indexing 1200 documents
[junit4] 2> 651119 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-0
(1554939473500831745)]} 0 127
[junit4] 2> 651120 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-0
(1554939473500831745)]} 0 221
[junit4] 2> 651120 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-0
(1554939473500831744)]} 0 137
[junit4] 2> 651122 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-0
(1554939473500831744)]} 0 224
[junit4] 2> 651133 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-1
(1554939473735712768)]} 0 4
[junit4] 2> 651134 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-1
(1554939473740955648)]} 0 2
[junit4] 2> 651135 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-1
(1554939473740955648)]} 0 7
[junit4] 2> 651135 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-1
(1554939473735712768)]} 0 13
[junit4] 2> 651145 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-2
(1554939473751441408)]} 0 5
[junit4] 2> 651146 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-2
(1554939473751441408)]} 0 9
[junit4] 2> 651154 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-2
(1554939473756684288)]} 0 8
[junit4] 2> 651154 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-2
(1554939473756684288)]} 0 11
[junit4] 2> 651158 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-3
(1554939473762975744)]} 0 3
[junit4] 2> 651160 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-3
(1554939473762975744)]} 0 11
[junit4] 2> 651166 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-3
(1554939473774510080)]} 0 4
[junit4] 2> 651167 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-3
(1554939473774510080)]} 0 7
[junit4] 2> 651169 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-4
(1554939473777655808)]} 0 4
[junit4] 2> 651169 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-4
(1554939473777655808)]} 0 7
[junit4] 2> 651178 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-5
(1554939473789190144)]} 0 2
[junit4] 2> 651178 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-5
(1554939473789190144)]} 0 4
[junit4] 2> 651180 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-4
(1554939473790238720)]} 0 3
[junit4] 2> 651180 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-4
(1554939473790238720)]} 0 6
[junit4] 2> 651185 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-6
(1554939473797578752)]} 0 2
[junit4] 2> 651186 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-6
(1554939473797578752)]} 0 4
[junit4] 2> 651194 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-5
(1554939473798627328)]} 0 9
[junit4] 2> 651194 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-5
(1554939473798627328)]} 0 12
[junit4] 2> 651199 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-7
(1554939473812258816)]} 0 1
[junit4] 2> 651201 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-7
(1554939473812258816)]} 0 5
[junit4] 2> 651202 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-6
(1554939473814355968)]} 0 2
[junit4] 2> 651203 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-6
(1554939473814355968)]} 0 5
[junit4] 2> 651210 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-7
(1554939473822744576)]} 0 3
[junit4] 2> 651211 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-7
(1554939473822744576)]} 0 6
[junit4] 2> 651214 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-8
(1554939473826938880)]} 0 2
[junit4] 2> 651214 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-8
(1554939473826938880)]} 0 5
[junit4] 2> 651228 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-4
(-1554939473835327488)]} 0 7
[junit4] 2> 651228 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-4
(-1554939473835327488)]} 0 11
[junit4] 2> 651230 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-8
(1554939473836376064)]} 0 3
[junit4] 2> 651230 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-8
(1554939473836376064)]} 0 12
[junit4] 2> 651237 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-4
(-1554939473851056128)]} 0 3
[junit4] 2> 651238 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-4
(-1554939473851056128)]} 0 5
[junit4] 2> 651239 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-9
(1554939473852104704)]} 0 4
[junit4] 2> 651240 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-9
(1554939473852104704)]} 0 6
[junit4] 2> 651244 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-9
(1554939473859444736)]} 0 2
[junit4] 2> 651245 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-9
(1554939473859444736)]} 0 4
[junit4] 2> 651247 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-5
(-1554939473862590464)]} 0 2
[junit4] 2> 651247 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-5
(-1554939473862590464)]} 0 4
[junit4] 2> 651253 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-5
(-1554939473869930496)]} 0 1
[junit4] 2> 651253 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-5
(-1554939473869930496)]} 0 3
[junit4] 2> 651258 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-10
(1554939473869930497)]} 0 2
[junit4] 2> 651258 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-10
(1554939473869930497)]} 0 8
[junit4] 2> 651261 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-10
(1554939473875173376)]} 0 2
[junit4] 2> 651261 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-10
(1554939473875173376)]} 0 5
[junit4] 2> 651269 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-6
(-1554939473884610560)]} 0 3
[junit4] 2> 651272 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-6
(-1554939473884610560)]} 0 8
[junit4] 2> 651276 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-6
(-1554939473894047744)]} 0 1
[junit4] 2> 651277 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-6
(-1554939473894047744)]} 0 4
[junit4] 2> 651282 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-11
(1554939473900339200)]} 0 1
[junit4] 2> 651284 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-11
(1554939473900339200)]} 0 5
[junit4] 2> 651287 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-11
(1554939473900339201)]} 0 3
[junit4] 2> 651287 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-11
(1554939473900339201)]} 0 8
[junit4] 2> 651292 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-12
(1554939473908727808)]} 0 1
[junit4] 2> 651292 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-12
(1554939473908727808)]} 0 5
[junit4] 2> 651300 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-12
(1554939473910824960)]} 0 5
[junit4] 2> 651300 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-12
(1554939473910824960)]} 0 11
[junit4] 2> 651301 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-7
(-1554939473916067840)]} 0 5
[junit4] 2> 651301 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-7
(-1554939473916067840)]} 0 7
[junit4] 2> 651306 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-7
(-1554939473924456448)]} 0 1
[junit4] 2> 651306 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-7
(-1554939473924456448)]} 0 3
[junit4] 2> 651311 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-13
(1554939473927602176)]} 0 3
[junit4] 2> 651311 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-13
(1554939473927602176)]} 0 6
[junit4] 2> 651314 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-13
(1554939473933893632)]} 0 1
[junit4] 2> 651315 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-13
(1554939473933893632)]} 0 3
[junit4] 2> 651317 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-11
(-1554939473935990784)]} 0 1
[junit4] 2> 651317 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-11
(-1554939473935990784)]} 0 4
[junit4] 2> 651319 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-11
(-1554939473939136512)]} 0 1
[junit4] 2> 651320 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-11
(-1554939473939136512)]} 0 3
[junit4] 2> 651322 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-14
(1554939473942282240)]} 0 1
[junit4] 2> 651322 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-14
(1554939473942282240)]} 0 3
[junit4] 2> 651324 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-14
(1554939473945427968)]} 0 1
[junit4] 2> 651325 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-14
(1554939473945427968)]} 0 3
[junit4] 2> 651327 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-15
(1554939473947525120)]} 0 1
[junit4] 2> 651327 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-15
(1554939473947525120)]} 0 3
[junit4] 2> 651329 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-15
(1554939473950670848)]} 0 1
[junit4] 2> 651330 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-15
(1554939473950670848)]} 0 3
[junit4] 2> 651333 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-12
(-1554939473952768000)]} 0 1
[junit4] 2> 651334 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-12
(-1554939473952768000)]} 0 4
[junit4] 2> 651334 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-12
(-1554939473955913728)]} 0 1
[junit4] 2> 651335 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-12
(-1554939473955913728)]} 0 3
[junit4] 2> 651339 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-16
(1554939473960108032)]} 0 1
[junit4] 2> 651339 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-16
(1554939473960108032)]} 0 3
[junit4] 2> 651340 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-16
(1554939473961156608)]} 0 1
[junit4] 2> 651340 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-16
(1554939473961156608)]} 0 3
[junit4] 2> 651345 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-17
(1554939473966399488)]} 0 1
[junit4] 2> 651345 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-17
(1554939473966399488)]} 0 3
[junit4] 2> 651345 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-17
(1554939473965350912)]} 0 1
[junit4] 2> 651346 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-17
(1554939473965350912)]} 0 4
[junit4] 2> 651350 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-13
(-1554939473971642368)]} 0 1
[junit4] 2> 651350 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-13
(-1554939473971642368)]} 0 3
[junit4] 2> 651351 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-13
(-1554939473972690944)]} 0 1
[junit4] 2> 651351 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-13
(-1554939473972690944)]} 0 3
[junit4] 2> 651358 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-18
(1554939473976885248)]} 0 1
[junit4] 2> 651358 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-18
(1554939473976885248)]} 0 5
[junit4] 2> 651358 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-18
(1554939473977933824)]} 0 1
[junit4] 2> 651359 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-18
(1554939473977933824)]} 0 6
[junit4] 2> 651363 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-19
(1554939473985273856)]} 0 1
[junit4] 2> 651363 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-19
(1554939473985273856)]} 0 3
[junit4] 2> 651364 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-19
(1554939473986322432)]} 0 1
[junit4] 2> 651365 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-19
(1554939473986322432)]} 0 3
[junit4] 2> 651368 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-14
(-1554939473990516736)]} 0 1
[junit4] 2> 651369 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-14
(-1554939473990516736)]} 0 3
[junit4] 2> 651370 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-14
(-1554939473992613888)]} 0 1
[junit4] 2> 651370 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-14
(-1554939473992613888)]} 0 3
[junit4] 2> 651374 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-20
(1554939473996808192)]} 0 1
[junit4] 2> 651375 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-20
(1554939473996808192)]} 0 3
[junit4] 2> 651375 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-20
(1554939473997856768)]} 0 1
[junit4] 2> 651376 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-20
(1554939473997856768)]} 0 3
[junit4] 2> 651380 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-21
(1554939474003099648)]} 0 1
[junit4] 2> 651380 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-21
(1554939474003099648)]} 0 3
[junit4] 2> 651380 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-21
(1554939474003099649)]} 0 1
[junit4] 2> 651381 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-21
(1554939474003099649)]} 0 3
[junit4] 2> 651385 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-15
(-1554939474008342528)]} 0 1
[junit4] 2> 651385 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-15
(-1554939474008342528)]} 0 3
[junit4] 2> 651386 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-15
(-1554939474009391104)]} 0 1
[junit4] 2> 651386 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-15
(-1554939474009391104)]} 0 3
[junit4] 2> 651390 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-22
(1554939474013585408)]} 0 1
[junit4] 2> 651391 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-22
(1554939474013585408)]} 0 3
[junit4] 2> 651391 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-22
(1554939474014633984)]} 0 1
[junit4] 2> 651392 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-22
(1554939474014633984)]} 0 3
[junit4] 2> 651395 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-16
(-1554939474019876864)]} 0 1
[junit4] 2> 651396 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-16
(-1554939474019876864)]} 0 3
[junit4] 2> 651396 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-16
(-1554939474020925440)]} 0 1
[junit4] 2> 651397 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-16
(-1554939474020925440)]} 0 3
[junit4] 2> 651401 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-23
(1554939474025119744)]} 0 1
[junit4] 2> 651402 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-23
(1554939474025119744)]} 0 3
[junit4] 2> 651402 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-23
(1554939474026168320)]} 0 1
[junit4] 2> 651402 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-23
(1554939474026168320)]} 0 3
[junit4] 2> 651407 INFO (qtp860614578-4105) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-17
(-1554939474031411200)]} 0 1
[junit4] 2> 651407 INFO (qtp1343162015-4100) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-17
(-1554939474031411200)]} 0 3
[junit4] 2> 651408 INFO (qtp860614578-4102) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-17
(-1554939474031411201)]} 0 1
[junit4] 2> 651408 INFO (qtp1343162015-4111) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-17
(-1554939474031411201)]} 0 3
[junit4] 2> 651412 INFO (qtp860614578-4107) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[2-24
(1554939474036654080)]} 0 1
[junit4] 2> 651413 INFO (qtp1343162015-4116) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[2-24
(1554939474036654080)]} 0 4
[junit4] 2> 651414 INFO (qtp860614578-4106) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{add=[1-24
(1554939474037702656)]} 0 2
[junit4] 2> 651414 INFO (qtp1343162015-4114) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{add=[1-24
(1554939474037702656)]} 0 4
[junit4] 2> 651418 INFO (qtp860614578-4104) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[2-18
(-1554939474042945536)]} 0 1
[junit4] 2> 651419 INFO (qtp1343162015-4115) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[2-18
(-1554939474042945536)]} 0 3
[junit4] 2> 651419 INFO (qtp860614578-4101) [n:127.0.0.1:45024_solr
c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2]
webapp=/solr path=/update
params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:56347/solr/recoverytest_shard1_replica1/&wt=javabin&version=2}{delete=[1-18
(-1554939474043994112)]} 0 1
[junit4] 2> 651419 INFO (qtp1343162015-4112) [n:127.0.0.1:56347_solr
c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1]
o.a.s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1]
webapp=/solr path=/update params={wt=javabin&version=2}{delete=[1-18
(-1554939474043
[...truncated too long message...]
tMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
[junit4] 2> at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
[junit4] 2> at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4] 2> at
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
[junit4] 2> at
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
[junit4] 2> at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
[junit4] 2> at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
[junit4] 2> at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
[junit4] 2> at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
[junit4] 2> at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
[junit4] 2> at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
[junit4] 2> at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
[junit4] 2> at
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
[junit4] 2> at
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
[junit4] 2> at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
[junit4] 2> at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
[junit4] 2> at
org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
[junit4] 2> at
org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest.teardownClass(HdfsRecoveryZkTest.java:51)
[junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4] 2> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] 2> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2> at java.lang.reflect.Method.invoke(Method.java:498)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
[junit4] 2> at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
[junit4] 2> at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
[junit4] 2> at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
[junit4] 2> at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
[junit4] 2> at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
[junit4] 2> at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Caused by: java.lang.NullPointerException
[junit4] 2> at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
[junit4] 2> at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
[junit4] 2> at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
[junit4] 2> ... 54 more
[junit4] 2> 675892 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.c.ZkTestServer connecting to 127.0.0.1:54324 54324
[junit4] 2> 675892 INFO
(SUITE-HdfsRecoveryZkTest-seed#[B7D3BF2CEA67C129]-worker) [ ]
o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> Dec 28, 2016 6:17:44 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 35 leaked
thread(s).
[junit4] 2> Dec 28, 2016 6:18:04 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at
org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest:
[junit4] 2> 1) Thread[id=4949,
name=OverseerHdfsCoreFailoverThread-97183716480974859-127.0.0.1:45024_solr-n_0000000002,
state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> Dec 28, 2016 6:18:04 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=4949,
name=OverseerHdfsCoreFailoverThread-97183716480974859-127.0.0.1:45024_solr-n_0000000002,
state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] 2> Dec 28, 2016 6:18:04 AM
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are:
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=22527, maxDocsPerChunk=4, blockSize=3),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=22527, blockSize=3)),
sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=es-SV,
timezone=America/Kralendijk
[junit4] 2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation
1.8.0_102 (64-bit)/cpus=4,threads=5,free=128517432,total=492830720
[junit4] 2> NOTE: All tests run in this JVM:
[PathHierarchyTokenizerFactoryTest, TestCustomSort, BlockJoinFacetRandomTest,
TestCustomDocTransformer, TestOverriddenPrefixQueryForCustomFieldType,
TestBlobHandler, CoreMergeIndexesAdminHandlerTest,
TestLeaderElectionWithEmptyReplica, AnalyticsQueryTest, TestCloudInspectUtil,
TestSubQueryTransformerDistrib, OverseerRolesTest, TestRandomFlRTGCloud,
MBeansHandlerTest, TestFastLRUCache, AtomicUpdatesTest,
SignatureUpdateProcessorFactoryTest, PolyFieldTest, TestNumericTerms64,
DateRangeFieldTest, TestConfigSetImmutable, TestSortByMinMaxFunction,
BitVectorTest, TestOmitPositions, SimpleCollectionCreateDeleteTest,
TestUseDocValuesAsStored, TestPseudoReturnFields, CloudMLTQParserTest,
AnalysisAfterCoreReloadTest, ReplicaListTransformerTest, NumericFieldsTest,
RuleEngineTest, ActionThrottleTest, TestSQLHandler, TestRemoteStreaming,
TestDistribIDF, CloudExitableDirectoryReaderTest, TestRawResponseWriter,
RegexBoostProcessorTest, TestSha256AuthenticationProvider, TestStressLucene,
TestSerializedLuceneMatchVersion, TestCrossCoreJoin, TestFieldCache,
TestSolrCloudSnapshots, UUIDFieldTest, HLLSerializationTest,
LeaderFailureAfterFreshStartTest, TestPerFieldSimilarityWithDefaultOverride,
TestImplicitCoreProperties, ExternalFileFieldSortTest, TestIntervalFaceting,
UniqFieldsUpdateProcessorFactoryTest, TestDistributedSearch, SolrXmlInZkTest,
TestXmlQParser, HdfsThreadLeakTest, ZkStateReaderTest, TestNRTOpen,
ConcurrentDeleteAndCreateCollectionTest, TestSolrConfigHandler, TestCSVLoader,
QueryEqualityTest, TestLocalFSCloudBackupRestore, ReplaceNodeTest,
DocumentAnalysisRequestHandlerTest, HdfsRecoveryZkTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=HdfsRecoveryZkTest
-Dtests.seed=B7D3BF2CEA67C129 -Dtests.multiplier=2 -Dtests.slow=true
-Dtests.locale=es-SV -Dtests.timezone=America/Kralendijk -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | HdfsRecoveryZkTest (suite) <<<
[junit4] > Throwable #1:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE
scope at org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest:
[junit4] > 1) Thread[id=4949,
name=OverseerHdfsCoreFailoverThread-97183716480974859-127.0.0.1:45024_solr-n_0000000002,
state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at
__randomizedtesting.SeedInfo.seed([B7D3BF2CEA67C129]:0)
[junit4] Completed [186/672 (1!)] on J0 in 54.90s, 1 test, 1 error <<<
FAILURES!
[...truncated 64023 lines...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
