Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1659/

3 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue
 {#2}

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0)




Build Log:
[...truncated 14339 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/init-core-data-001
   [junit4]   2> 1911582 WARN  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=42 numCloses=42
   [junit4]   2> 1911582 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 1911584 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 1911584 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [    ] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 1911584 INFO  
(SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [    ] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1911589 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1911589 INFO  (Thread-55391) [    ] o.a.s.c.ZkTestServer 
client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1911589 INFO  (Thread-55391) [    ] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 1911591 ERROR (Thread-55391) [    ] o.a.z.s.ZooKeeperServer 
ZKShutdownHandler is not registered, so ZooKeeper server won't take any action 
on ERROR or SHUTDOWN server state changes
   [junit4]   2> 1911689 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkTestServer start zk server on port:42889
   [junit4]   2> 1911697 INFO  (zkConnectionManagerCallback-4819-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1911701 INFO  (zkConnectionManagerCallback-4821-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1911705 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1911707 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1911708 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1911709 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1911710 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1911711 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1911712 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
 to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1911713 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
 to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1911714 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
 to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1911715 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
 to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1911716 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractZkTestCase put 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
 to /configs/conf1/synonyms.txt
   [junit4]   2> 1911718 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly 
asked otherwise
   [junit4]   2> 1911831 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1911832 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1911832 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1911832 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1911832 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7858dc0d{/,null,AVAILABLE}
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@10e2e8ff{HTTP/1.1,[http/1.1]}{127.0.0.1:37722}
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server Started @1911872ms
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/control/data,
 replicaType=NRT, hostContext=/, hostPort=37722, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores}
   [junit4]   2> 1911833 ERROR 
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1911833 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1911834 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-10-06T09:56:18.739Z
   [junit4]   2> 1911835 INFO  (zkConnectionManagerCallback-4823-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1911836 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1911836 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/solr.xml
   [junit4]   2> 1911839 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 1911839 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1911841 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1911973 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr
   [junit4]   2> 1911974 INFO  (zkConnectionManagerCallback-4827-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1911976 INFO  (zkConnectionManagerCallback-4829-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1912061 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.c.OverseerElectionContext I am going to be the 
leader 127.0.0.1:37722_
   [junit4]   2> 1912062 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer Overseer 
(id=72457920339378180-127.0.0.1:37722_-n_0000000000) starting
   [junit4]   2> 1912069 INFO  (zkConnectionManagerCallback-4836-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1912070 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 
127.0.0.1:42889/solr ready
   [junit4]   2> 1912071 INFO  
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer Starting to work on the main queue : 
127.0.0.1:37722_
   [junit4]   2> 1912072 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:37722_
   [junit4]   2> 1912076 INFO  
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.c.ZkStateReader Updated live nodes from 
ZooKeeper... (0) -> (1)
   [junit4]   2> 1912085 INFO  (zkCallback-4835-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1912093 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.h.a.MetricsHistoryHandler No .system collection, 
keeping metrics history in memory.
   [junit4]   2> 1912108 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.node' (registry 'solr.node') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1912117 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jvm' (registry 'solr.jvm') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1912117 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.jetty' (registry 'solr.jetty') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1912118 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) 
[n:127.0.0.1:37722_    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions 
underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores
   [junit4]   2> 1912138 INFO  (zkConnectionManagerCallback-4842-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1912138 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1912139 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready
   [junit4]   2> 1912140 INFO  (qtp1443383752-70377) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37722_&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1912142 INFO  
(OverseerThreadFactory-7558-thread-1-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
control_collection
   [junit4]   2> 1912145 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "name":"control_collection",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "createNodeSet":"127.0.0.1:37722_",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 1912249 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_    
x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core 
create command 
qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1912250 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_    
x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault 
Allocating transient cache for 4 transient cores
   [junit4]   2> 1912253 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:37722";,
   [junit4]   2>   "node_name":"127.0.0.1:37722_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1913274 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1913286 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 1913355 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1913371 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' 
using configuration from collection control_collection, trusted=true
   [junit4]   2> 1913371 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.control_collection.shard1.replica_n1' (registry 
'solr.core.control_collection.shard1.replica_n1') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1913372 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore 
at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores/control_collection_shard1_replica_n1],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 1913374 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0]
   [junit4]   2> 1913375 WARN  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1913413 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateHandler Using UpdateLog implementation: 
org.apache.solr.update.UpdateLog
   [junit4]   2> 1913413 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH 
numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1913414 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1913414 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1913414 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, 
floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, 
segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924
   [junit4]   2> 1913415 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@6d8f0158[control_collection_shard1_replica_n1] main]
   [junit4]   2> 1913416 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1913416 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1913416 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1913417 INFO  
(searcherExecutor-7563-thread-1-processing-n:127.0.0.1:37722_ 
x:control_collection_shard1_replica_n1 c:control_collection s:shard1) 
[n:127.0.0.1:37722_ c:control_collection s:shard1  
x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore 
[control_collection_shard1_replica_n1] Registered new searcher 
Searcher@6d8f0158[control_collection_shard1_replica_n1] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1913418 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.u.UpdateLog Could not find max version in index or recent updates, using 
new clock 1613569489973018624
   [junit4]   2> 1913423 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ZkShardTerms Successful update of terms at 
/collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, 
version=0}
   [junit4]   2> 1913426 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1913426 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1913426 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync replicas to 
http://127.0.0.1:37722/control_collection_shard1_replica_n1/
   [junit4]   2> 1913426 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1913426 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.SyncStrategy 
http://127.0.0.1:37722/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 1913427 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1913430 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:37722/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 1913430 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:37722";,
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "state":"active"} current state version: 0
   [junit4]   2> 1913532 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1913534 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"control_collection_shard1_replica_n1",
   [junit4]   2>   "core_node_name":"core_node2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:37722";,
   [junit4]   2>   "node_name":"127.0.0.1:37722_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1913543 INFO  (qtp1443383752-70372) [n:127.0.0.1:37722_ 
c:control_collection s:shard1  x:control_collection_shard1_replica_n1] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1294
   [junit4]   2> 1913547 INFO  (qtp1443383752-70377) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1913635 INFO  (zkCallback-4828-thread-1) [    ] 
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent 
state:SyncConnected type:NodeDataChanged 
path:/collections/control_collection/state.json] for collection 
[control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1914147 INFO  
(OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000)
 [n:127.0.0.1:37722_    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1914547 INFO  (qtp1443383752-70377) [n:127.0.0.1:37722_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37722_&wt=javabin&version=2}
 status=0 QTime=2407
   [junit4]   2> 1914552 INFO  (zkConnectionManagerCallback-4847-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1914556 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1914557 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready
   [junit4]   2> 1914557 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection 
loss:false
   [junit4]   2> 1914558 INFO  (qtp1443383752-70379) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params 
collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1914561 INFO  
(OverseerThreadFactory-7558-thread-2-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_    ] o.a.s.c.a.c.CreateCollectionCmd Create collection 
collection1
   [junit4]   2> 1914565 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "createNodeSet":"",
   [junit4]   2>   "stateFormat":"1",
   [junit4]   2>   "nrtReplicas":"1",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 1914771 WARN  
(OverseerThreadFactory-7558-thread-2-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_    ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to 
create a collection (collection1) without cores.
   [junit4]   2> 1914773 INFO  (qtp1443383752-70379) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 
30 seconds. Check all shard replicas
   [junit4]   2> 1914774 INFO  (qtp1443383752-70379) [n:127.0.0.1:37722_    ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections 
params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2}
 status=0 QTime=215
   [junit4]   2> 1914884 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001
 of type NRT
   [junit4]   2> 1914885 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1914885 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1914886 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1914886 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1914886 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@22b7039a{/,null,AVAILABLE}
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@18b2b7b7{HTTP/1.1,[http/1.1]}{127.0.0.1:41583}
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server Started @1914931ms
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty1,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=41583, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores}
   [junit4]   2> 1914892 ERROR 
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1914892 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-10-06T09:56:21.798Z
   [junit4]   2> 1914904 INFO  (zkConnectionManagerCallback-4849-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1914905 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1914905 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/solr.xml
   [junit4]   2> 1914908 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 1914908 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1914910 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1915151 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr
   [junit4]   2> 1915152 INFO  (zkConnectionManagerCallback-4853-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1915155 INFO  (zkConnectionManagerCallback-4855-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1915162 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1915165 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1915165 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:41583_
   [junit4]   2> 1915166 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:41583_"} current state version: 1
   [junit4]   2> 1915166 INFO  (zkCallback-4835-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1915166 INFO  (zkCallback-4846-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1915166 INFO  (zkCallback-4828-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1915168 INFO  (zkCallback-4854-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1915186 INFO  (zkConnectionManagerCallback-4862-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1915187 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1915188 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready
   [junit4]   2> 1915188 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 1915206 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1915216 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1915216 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1915217 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores
   [junit4]   2> 1915245 INFO  (qtp1443383752-70373) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params 
node=127.0.0.1:41583_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1915248 INFO  
(OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000)
 [n:127.0.0.1:37722_    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1915249 INFO  
(OverseerThreadFactory-7558-thread-3-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_ c:collection1 s:shard2  ] o.a.s.c.a.c.AddReplicaCmd Node 
Identified 127.0.0.1:41583_ for creating new replica of shard shard2 for 
collection collection1
   [junit4]   2> 1915251 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_    
x:collection1_shard2_replica_n61] o.a.s.h.a.CoreAdminOperation core create 
command 
qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1915253 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 2, message 
= {
   [junit4]   2>   "core":"collection1_shard2_replica_n61",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41583";,
   [junit4]   2>   "node_name":"127.0.0.1:41583_",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 1
   [junit4]   2> 1916266 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1916314 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema 
[collection1_shard2_replica_n61] Schema name=test
   [junit4]   2> 1916419 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1916480 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1_shard2_replica_n61' using configuration from 
collection collection1, trusted=true
   [junit4]   2> 1916481 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1.shard2.replica_n61' (registry 
'solr.core.collection1.shard2.replica_n61') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1916481 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore 
[[collection1_shard2_replica_n61] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores/collection1_shard2_replica_n61],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores/collection1_shard2_replica_n61/data/]
   [junit4]   2> 1916483 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0]
   [junit4]   2> 1916497 WARN  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1916553 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1916554 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1916555 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1916555 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1916555 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, 
floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, 
segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924
   [junit4]   2> 1916556 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@74dde355[collection1_shard2_replica_n61] main]
   [junit4]   2> 1916557 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1916557 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1916558 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1916558 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1613569493265547264
   [junit4]   2> 1916560 INFO  
(searcherExecutor-7577-thread-1-processing-n:127.0.0.1:41583_ 
x:collection1_shard2_replica_n61 c:collection1 s:shard2) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SolrCore 
[collection1_shard2_replica_n61] Registered new searcher 
Searcher@74dde355[collection1_shard2_replica_n61] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1916575 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkShardTerms 
Successful update of terms at /collections/collection1/terms/shard2 to 
Terms{values={core_node62=0}, version=0}
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:41583/collection1_shard2_replica_n61/
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy 
http://127.0.0.1:41583/collection1_shard2_replica_n61/ has no replicas
   [junit4]   2> 1916578 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1916582 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:41583/collection1_shard2_replica_n61/ shard2
   [junit4]   2> 1916584 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:41583";,
   [junit4]   2>   "core":"collection1_shard2_replica_n61",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1916733 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 1916734 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1_shard2_replica_n61",
   [junit4]   2>   "core_node_name":"core_node62",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41583";,
   [junit4]   2>   "node_name":"127.0.0.1:41583_",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard2",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1916748 INFO  (qtp2023914897-70432) [n:127.0.0.1:41583_ 
c:collection1 s:shard2  x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1497
   [junit4]   2> 1916750 INFO  (qtp1443383752-70373) [n:127.0.0.1:37722_ 
c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/collections 
params={node=127.0.0.1:41583_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2}
 status=0 QTime=1504
   [junit4]   2> 1916886 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001
 of type NRT
   [junit4]   2> 1916887 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1916888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1916888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1916888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 1916888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@7c456d2e{/,null,AVAILABLE}
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@3e21c36{HTTP/1.1,[http/1.1]}{127.0.0.1:46212}
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server Started @1916928ms
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty2,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=46212, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores}
   [junit4]   2> 1916889 ERROR 
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1916889 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-10-06T09:56:23.795Z
   [junit4]   2> 1916896 INFO  (zkConnectionManagerCallback-4865-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1916897 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1916897 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/solr.xml
   [junit4]   2> 1916901 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 1916901 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1916902 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1917065 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr
   [junit4]   2> 1917067 INFO  (zkConnectionManagerCallback-4869-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1917069 INFO  (zkConnectionManagerCallback-4871-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1917074 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1917078 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1917078 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:46212_
   [junit4]   2> 1917079 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:46212_"} current state version: 4
   [junit4]   2> 1917080 INFO  (zkCallback-4828-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917080 INFO  (zkCallback-4846-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917080 INFO  (zkCallback-4870-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917080 INFO  (zkCallback-4854-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917080 INFO  (zkCallback-4861-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917080 INFO  (zkCallback-4835-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 1917098 INFO  (zkConnectionManagerCallback-4878-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1917099 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1917100 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready
   [junit4]   2> 1917101 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 1917121 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1917131 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1917131 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1917133 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores
   [junit4]   2> 1917168 INFO  (qtp1443383752-70379) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params 
node=127.0.0.1:46212_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1917170 INFO  
(OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000)
 [n:127.0.0.1:37722_    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1917170 INFO  
(OverseerThreadFactory-7558-thread-4-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_ c:collection1 s:shard3  ] o.a.s.c.a.c.AddReplicaCmd Node 
Identified 127.0.0.1:46212_ for creating new replica of shard shard3 for 
collection collection1
   [junit4]   2> 1917173 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_    
x:collection1_shard3_replica_n63] o.a.s.h.a.CoreAdminOperation core create 
command 
qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1917175 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 2, message 
= {
   [junit4]   2>   "core":"collection1_shard3_replica_n63",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46212";,
   [junit4]   2>   "node_name":"127.0.0.1:46212_",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 4
   [junit4]   2> 1918190 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1918203 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema 
[collection1_shard3_replica_n63] Schema name=test
   [junit4]   2> 1918282 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1918294 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1_shard3_replica_n63' using configuration from 
collection collection1, trusted=true
   [junit4]   2> 1918295 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1.shard3.replica_n63' (registry 
'solr.core.collection1.shard3.replica_n63') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1918295 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore 
[[collection1_shard3_replica_n63] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores/collection1_shard3_replica_n63],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores/collection1_shard3_replica_n63/data/]
   [junit4]   2> 1918297 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0]
   [junit4]   2> 1918298 WARN  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1918335 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1918335 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1918337 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1918337 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1918337 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, 
floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, 
segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924
   [junit4]   2> 1918338 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@379b9080[collection1_shard3_replica_n63] main]
   [junit4]   2> 1918339 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1918339 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1918340 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1918340 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1613569495134109696
   [junit4]   2> 1918340 INFO  
(searcherExecutor-7591-thread-1-processing-n:127.0.0.1:46212_ 
x:collection1_shard3_replica_n63 c:collection1 s:shard3) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SolrCore 
[collection1_shard3_replica_n63] Registered new searcher 
Searcher@379b9080[collection1_shard3_replica_n63] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1918345 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ZkShardTerms 
Successful update of terms at /collections/collection1/terms/shard3 to 
Terms{values={core_node64=0}, version=0}
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:46212/collection1_shard3_replica_n63/
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy 
http://127.0.0.1:46212/collection1_shard3_replica_n63/ has no replicas
   [junit4]   2> 1918347 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1918350 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:46212";,
   [junit4]   2>   "core":"collection1_shard3_replica_n63",
   [junit4]   2>   "state":"active"} current state version: 5
   [junit4]   2> 1918350 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:46212/collection1_shard3_replica_n63/ shard3
   [junit4]   2> 1918501 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 1918503 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1_shard3_replica_n63",
   [junit4]   2>   "core_node_name":"core_node64",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46212";,
   [junit4]   2>   "node_name":"127.0.0.1:46212_",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard3",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 6
   [junit4]   2> 1918506 INFO  (qtp321590564-70469) [n:127.0.0.1:46212_ 
c:collection1 s:shard3  x:collection1_shard3_replica_n63] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT}
 status=0 QTime=1333
   [junit4]   2> 1918508 INFO  (qtp1443383752-70379) [n:127.0.0.1:37722_ 
c:collection1   ] o.a.s.s.HttpSolrCall [admin] webapp=null 
path=/admin/collections 
params={node=127.0.0.1:46212_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2}
 status=0 QTime=1340
   [junit4]   2> 1918620 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001
 of type NRT
   [junit4]   2> 1918621 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: 
d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
   [junit4]   2> 1918623 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 1918623 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 1918623 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.h.ContextHandler Started 
o.e.j.s.ServletContextHandler@6a0e8493{/,null,AVAILABLE}
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.AbstractConnector Started 
ServerConnector@4f3b8cbc{HTTP/1.1,[http/1.1]}{127.0.0.1:40466}
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.e.j.s.Server Started @1918663ms
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.e.JettySolrRunner Jetty properties: 
{solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty3,
 replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40466, 
coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores}
   [junit4]   2> 1918624 ERROR 
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be 
missing or incomplete.
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Using logger factory 
org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 
8.0.0
   [junit4]   2> 1918624 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1918625 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1918625 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 
2018-10-06T09:56:25.531Z
   [junit4]   2> 1918626 INFO  (zkConnectionManagerCallback-4881-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1918627 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in 
ZooKeeper)
   [junit4]   2> 1918627 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Loading container configuration from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/solr.xml
   [junit4]   2> 1918631 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay 
is ignored
   [junit4]   2> 1918631 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig Configuration parameter 
autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 1918632 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.SolrXmlConfig MBean server found: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were 
configured - adding default JMX reporter.
   [junit4]   2> 1918875 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr
   [junit4]   2> 1918876 INFO  (zkConnectionManagerCallback-4885-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1918878 INFO  (zkConnectionManagerCallback-4887-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1918884 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 1918888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 
transient cores
   [junit4]   2> 1918888 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.ZkController Register node as live in 
ZooKeeper:/live_nodes/127.0.0.1:40466_
   [junit4]   2> 1918888 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:40466_"} current state version: 7
   [junit4]   2> 1918889 INFO  (zkCallback-4835-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918889 INFO  (zkCallback-4854-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918889 INFO  (zkCallback-4846-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918889 INFO  (zkCallback-4870-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918892 INFO  (zkCallback-4861-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918893 INFO  (zkCallback-4886-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918893 INFO  (zkCallback-4877-thread-1) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918893 INFO  (zkCallback-4828-thread-2) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 1918907 INFO  (zkConnectionManagerCallback-4894-thread-1) [    
] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 1918909 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 1918910 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready
   [junit4]   2> 1918910 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history 
in memory.
   [junit4]   2> 1918929 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1918940 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') 
enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1918940 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 
'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1918941 INFO  
(TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [    ] 
o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores
   [junit4]   2> 1919000 INFO  (qtp1443383752-70373) [n:127.0.0.1:37722_    ] 
o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params 
node=127.0.0.1:40466_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2
 and sendToOCPQueue=true
   [junit4]   2> 1919002 INFO  
(OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000)
 [n:127.0.0.1:37722_    ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
   [junit4]   2> 1919005 INFO  
(OverseerThreadFactory-7558-thread-5-processing-n:127.0.0.1:37722_) 
[n:127.0.0.1:37722_ c:collection1 s:shard1  ] o.a.s.c.a.c.AddReplicaCmd Node 
Identified 127.0.0.1:40466_ for creating new replica of shard shard1 for 
collection collection1
   [junit4]   2> 1919009 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_    
x:collection1_shard1_replica_n65] o.a.s.h.a.CoreAdminOperation core create 
command 
qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 1919013 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1_shard1_replica_n65",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40466";,
   [junit4]   2>   "node_name":"127.0.0.1:40466_",
   [junit4]   2>   "numShards":"3",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "operation":"state"} current state version: 7
   [junit4]   2> 1920026 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrConfig 
Using Lucene MatchVersion: 8.0.0
   [junit4]   2> 1920037 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema 
[collection1_shard1_replica_n65] Schema name=test
   [junit4]   2> 1920120 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema 
Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 1920132 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.CoreContainer 
Creating SolrCore 'collection1_shard1_replica_n65' using configuration from 
collection collection1, trusted=true
   [junit4]   2> 1920133 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.m.r.SolrJmxReporter JMX monitoring for 
'solr.core.collection1.shard1.replica_n65' (registry 
'solr.core.collection1.shard1.replica_n65') enabled at server: 
com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75
   [junit4]   2> 1920133 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore 
[[collection1_shard1_replica_n65] ] Opening new SolrCore at 
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores/collection1_shard1_replica_n65],
 
dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores/collection1_shard1_replica_n65/data/]
   [junit4]   2> 1920135 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: 
minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, 
maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, 
maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0]
   [junit4]   2> 1920141 WARN  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = 
requestHandler,name = /dump,class = DumpRequestHandler,attributes = 
{initParams=a, name=/dump, class=DumpRequestHandler},args = 
{defaults={a=A,b=B}}}
   [junit4]   2> 1920178 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateHandler 
Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1920178 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog 
Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 
maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1920179 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker 
Hard AutoCommit: disabled
   [junit4]   2> 1920179 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker 
Soft AutoCommit: disabled
   [junit4]   2> 1920180 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class 
org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: 
maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, 
floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, 
segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, 
noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924
   [junit4]   2> 1920180 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.s.SolrIndexSearcher Opening 
[Searcher@b6f9712[collection1_shard1_replica_n65] main]
   [junit4]   2> 1920181 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: 
/configs/conf1
   [junit4]   2> 1920181 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using 
ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1920181 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 1920182 INFO  
(searcherExecutor-7605-thread-1-processing-n:127.0.0.1:40466_ 
x:collection1_shard1_replica_n65 c:collection1 s:shard1) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SolrCore 
[collection1_shard1_replica_n65] Registered new searcher 
Searcher@b6f9712[collection1_shard1_replica_n65] 
main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1920182 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog 
Could not find max version in index or recent updates, using new clock 
1613569497065586688
   [junit4]   2> 1920190 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ZkShardTerms 
Successful update of terms at /collections/collection1/terms/shard1 to 
Terms{values={core_node66=0}, version=0}
   [junit4]   2> 1920193 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1920193 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1920193 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy 
Sync replicas to http://127.0.0.1:40466/collection1_shard1_replica_n65/
   [junit4]   2> 1920194 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy 
Sync Success - now sync replicas to me
   [junit4]   2> 1920194 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy 
http://127.0.0.1:40466/collection1_shard1_replica_n65/ has no replicas
   [junit4]   2> 1920194 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.c.ShardLeaderElectionContext Found all replicas participating in 
election, clear LIR
   [junit4]   2> 1920198 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] 
o.a.s.c.ShardLeaderElectionContext I am the new leader: 
http://127.0.0.1:40466/collection1_shard1_replica_n65/ shard1
   [junit4]   2> 1920198 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "base_url":"http://127.0.0.1:40466";,
   [junit4]   2>   "core":"collection1_shard1_replica_n65",
   [junit4]   2>   "state":"active"} current state version: 8
   [junit4]   2> 1920353 INFO  (qtp333792483-70506) [n:127.0.0.1:40466_ 
c:collection1 s:shard1  x:collection1_shard1_replica_n65] o.a.s.c.ZkController 
I am the leader, no recovery necessary
   [junit4]   2> 1920354 DEBUG 
(OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) 
[n:127.0.0.1:37722_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message 
= {
   [junit4]   2>   "core":"collection1_shard1_replica_n65",
   [junit4]   2>   "core_node_name":"core_node66",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40466";,
   [junit4]   2>   "node_name":"127.0.0.1:40466_",
   [junit4]   2>   "numShards":"3",
   [junit4]

[...truncated too long message...]

(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> oct. 06, 2018 8:24:26 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFOS: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=82102, 
name=TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[690FCFDA6B8134D2],
 state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue]
   [junit4]   2> 10799173 INFO  
(TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[690FCFDA6B8134D2])
 [    ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue {#2}
   [junit4]   2> NOTE: leaving temporary files on disk at: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_690FCFDA6B8134D2-001
   [junit4]   2> Oct 06, 2018 12:24:26 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene80), 
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1daa8152),
 locale=fr-CA, timezone=Asia/Hong_Kong
   [junit4]   2> NOTE: Linux 4.4.0-130-generic amd64/Oracle Corporation 
1.8.0_172 (64-bit)/cpus=4,threads=1,free=188333088,total=497025024
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudDeleteByQuery, 
BadIndexSchemaTest, TestDistributedStatsComponentCardinality, 
TestExclusionRuleCollectionAccess, 
DistributedFacetSimpleRefinementLongTailTest, DefaultValueUpdateProcessorTest, 
TestStressInPlaceUpdates, ShardRoutingTest, ImplicitSnitchTest, 
TestAuthorizationFramework, TestClusterProperties, DisMaxRequestHandlerTest, 
TestLuceneIndexBackCompat, TestCloudJSONFacetSKG, 
TriLevelCompositeIdRoutingTest, UpdateRequestProcessorFactoryTest, 
CopyFieldTest, TestConfig, AdminHandlersProxyTest, SuggesterWFSTTest, 
SimplePostToolTest, DeleteNodeTest, WrapperMergePolicyFactoryTest, 
SpellCheckCollatorTest, TestConfigSetProperties, 
CoreMergeIndexesAdminHandlerTest, MultiTermTest, TestSimNodeAddedTrigger, 
TestRealTimeGet, ZkCLITest, ReplicationFactorTest, 
OpenExchangeRatesOrgProviderTest, SchemaApiFailureTest, 
SpellingQueryConverterTest, TestFaceting, TestCloudNestedDocsSort, 
TestSolr4Spatial, TestBulkSchemaAPI, TestClusterStateMutator, 
TestHdfsCloudBackupRestore, TestUnifiedSolrHighlighter, 
URLClassifyProcessorTest, TestOmitPositions, V2StandaloneTest, SOLR749Test, 
TestComplexPhraseQParserPlugin, ResponseHeaderTest, ScriptEngineTest, 
UtilsToolTest, IndexSchemaRuntimeFieldTest, TestHashQParserPlugin, 
BlockJoinFacetRandomTest, TestQueryUtils, BadComponentTest, TestFieldCacheSort, 
TriggerSetPropertiesIntegrationTest, TestConfigSetsAPIZkFailure, 
HealthCheckHandlerTest, TestNRTOpen, JsonLoaderTest, DateMathParserTest, 
TestPayloadScoreQParserPlugin, SolrGangliaReporterTest, NodeMutatorTest, 
ScheduledTriggerIntegrationTest, TestSolrQueryResponse, 
TestMiniSolrCloudClusterSSL, TestMaxTokenLenTokenizer, DateRangeFieldTest, 
MinimalSchemaTest, QueryParsingTest, TestSolrFieldCacheBean, 
TestSolrCloudSnapshots, SimpleFacetsTest, TestSchemalessBufferedUpdates, 
TestSimDistribStateManager, TestValueSourceCache, Tagger2Test, BlockCacheTest, 
TestBulkSchemaConcurrent, TestStressVersions, TestSolrIndexConfig, 
TestExceedMaxTermLength, SearchHandlerTest, 
DistributedQueryComponentCustomSortTest, TestEmbeddedSolrServerSchemaAPI, 
TestSSLRandomization, TestMinMaxOnMultiValuedField, TestDFISimilarityFactory, 
TestJsonFacetRefinement, SimpleCollectionCreateDeleteTest, 
TestReversedWildcardFilterFactory, SolrShardReporterTest, XmlInterpolationTest, 
OverseerCollectionConfigSetProcessorTest, 
ClassificationUpdateProcessorIntegrationTest, TestCursorMarkWithoutUniqueKey, 
MetricsHistoryHandlerTest, AutoScalingHandlerTest, SyncSliceTest, 
TestJsonFacetsWithNestedObjects, TestPivotHelperCode, CreateRoutedAliasTest, 
TestImplicitCoreProperties, TestSizeLimitedDistributedMap, StatsComponentTest, 
TestJavabinTupleStreamParser, HdfsDirectoryTest, TestExactSharedStatsCache, 
NodeAddedTriggerTest, ProtectedTermFilterFactoryTest, 
MissingSegmentRecoveryTest, AlternateDirectoryTest, AssignTest, 
TestTolerantUpdateProcessorCloud, TestAddFieldRealTimeGet, 
SimpleMLTQParserTest, CollectionStateFormat2Test, TestFiltering, 
MoveReplicaTest, TestSubQueryTransformer, TestFieldCacheWithThreads, 
TestNumericTerms64, SuggesterFSTTest, RequestLoggingTest, 
TestSolrConfigHandler, TestCoreDiscovery, OverseerStatusTest, 
TestRandomRequestDistribution, DistributedSpellCheckComponentTest, 
HdfsChaosMonkeyNothingIsSafeTest, SmileWriterTest, HdfsUnloadDistributedZkTest, 
TestDocSet, TestCollationField, NumberUtilsTest, TestSimTriggerIntegration, 
TestBackupRepositoryFactory, DirectSolrSpellCheckerTest, 
BasicFunctionalityTest, ConvertedLegacyTest, CursorPagingTest, 
SolrInfoBeanTest, SolrTestCaseJ4Test, TestDistributedSearch, 
TestDocumentBuilder, TestGroupingSearch, TestHighlightDedupGrouping, TestJoin, 
TestRandomFaceting, TestSimpleTrackingShardHandler, TestSolrCoreProperties, 
TestTolerantSearch, TestTrie, PathHierarchyTokenizerFactoryTest, 
TestLuceneMatchVersion, TokenizerChainTest, ActionThrottleTest, AddReplicaTest, 
AliasIntegrationTest, AssignBackwardCompatibilityTest, 
AsyncCallRequestStatusResponseTest, BasicDistributedZkTest, 
ChaosMonkeyNothingIsSafeWithPullReplicasTest, ClusterStateTest, 
ClusterStateUpdateTest, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, 
ConnectionManagerTest, CreateCollectionCleanupTest, DistribCursorPagingTest, 
DistribJoinFromCollectionTest, DocValuesNotIndexedTest, ForceLeaderTest, 
FullSolrCloudDistribCmdsTest, OverseerTest, RecoveryAfterSoftCommitTest, 
RecoveryZkTest, RemoteQueryErrorTest, ReplaceNodeNoTargetTest, ReplaceNodeTest, 
SolrCloudExampleTest, TestCloudRecovery, TestConfigSetsAPIExclusivity, 
TestDeleteCollectionOnDownNodes, TestDistributedMap, 
TestLeaderElectionWithEmptyReplica, TestPullReplica, TestRequestForwarding, 
TestShortCircuitedRequests, TestSolrCloudWithDelegationTokens, 
TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, 
TestStressCloudBlindAtomicUpdates, TestZkChroot, 
TlogReplayBufferedWhileIndexingTest, CollectionsAPIAsyncDistributedZkTest, 
ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, 
TestReplicaProperties, TestRequestStatusCollectionAPI, 
AutoAddReplicasIntegrationTest, ExecutePlanActionTest, 
HdfsAutoAddReplicasIntegrationTest, NodeLostTriggerTest, 
RestoreTriggerStateTest, SearchRateTriggerTest, TriggerCooldownIntegrationTest, 
TestSimExtremeIndexing, TestSimGenericDistributedQueue]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  
-Dtestcase=TestSimGenericDistributedQueue -Dtests.seed=690FCFDA6B8134D2 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=fr-CA -Dtests.timezone=Asia/Hong_Kong -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | TestSimGenericDistributedQueue (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 
7200000 msec).
   [junit4]    >        at 
__randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0)
   [junit4] Completed [835/835 (2!)] on J1 in 7280.38s, 4 tests, 2 errors <<< 
FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20181006_092426_814545575240458603412.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid24953.hprof
 ...
   [junit4] Heap dump file created [594292755 bytes in 8.552 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 8961 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:651:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:585:
 Some of the tests produced a heap dump, but did not fail. Maybe a suppressed 
OutOfMemoryError? Dumps created:
* java_pid24953.hprof

Total time: 369 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to