Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1659/
3 tests failed. FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test Error Message: Java heap space Stack Trace: java.lang.OutOfMemoryError: Java heap space FAILED: org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue {#2} Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0) Build Log: [...truncated 14339 lines...] [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/init-core-data-001 [junit4] 2> 1911582 WARN (SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=42 numCloses=42 [junit4] 2> 1911582 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true [junit4] 2> 1911584 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776) [junit4] 2> 1911584 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 1911584 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[690FCFDA6B8134D2]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: / [junit4] 2> 1911589 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 1911589 INFO (Thread-55391) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1911589 INFO (Thread-55391) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 1911591 ERROR (Thread-55391) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit4] 2> 1911689 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkTestServer start zk server on port:42889 [junit4] 2> 1911697 INFO (zkConnectionManagerCallback-4819-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1911701 INFO (zkConnectionManagerCallback-4821-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1911705 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 1911707 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml [junit4] 2> 1911708 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1911709 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 1911710 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 1911711 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 1911712 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 1911713 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 1911714 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 1911715 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 1911716 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 1911718 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise [junit4] 2> 1911831 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11 [junit4] 2> 1911832 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 1911832 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 1911832 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session node0 Scavenging every 660000ms [junit4] 2> 1911832 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7858dc0d{/,null,AVAILABLE} [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@10e2e8ff{HTTP/1.1,[http/1.1]}{127.0.0.1:37722} [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server Started @1911872ms [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=37722, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores} [junit4] 2> 1911833 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0 [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1911833 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1911834 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-10-06T09:56:18.739Z [junit4] 2> 1911835 INFO (zkConnectionManagerCallback-4823-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1911836 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1911836 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/solr.xml [junit4] 2> 1911839 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored [junit4] 2> 1911839 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored [junit4] 2> 1911841 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 1911973 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr [junit4] 2> 1911974 INFO (zkConnectionManagerCallback-4827-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1911976 INFO (zkConnectionManagerCallback-4829-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1912061 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:37722_ [junit4] 2> 1912062 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer Overseer (id=72457920339378180-127.0.0.1:37722_-n_0000000000) starting [junit4] 2> 1912069 INFO (zkConnectionManagerCallback-4836-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1912070 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1912071 INFO (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:37722_ [junit4] 2> 1912072 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37722_ [junit4] 2> 1912076 INFO (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1912085 INFO (zkCallback-4835-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1912093 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1912108 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1912117 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1912117 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1912118 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [n:127.0.0.1:37722_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores [junit4] 2> 1912138 INFO (zkConnectionManagerCallback-4842-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1912138 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1912139 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1912140 INFO (qtp1443383752-70377) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37722_&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 1912142 INFO (OverseerThreadFactory-7558-thread-1-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection [junit4] 2> 1912145 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "name":"control_collection", [junit4] 2> "fromApi":"true", [junit4] 2> "collection.configName":"conf1", [junit4] 2> "numShards":"1", [junit4] 2> "createNodeSet":"127.0.0.1:37722_", [junit4] 2> "nrtReplicas":"1", [junit4] 2> "stateFormat":"2", [junit4] 2> "replicationFactor":"1", [junit4] 2> "operation":"create"} current state version: 0 [junit4] 2> 1912249 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT [junit4] 2> 1912250 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores [junit4] 2> 1912253 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"control_collection_shard1_replica_n1", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:37722", [junit4] 2> "node_name":"127.0.0.1:37722_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"down", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 0 [junit4] 2> 1913274 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0 [junit4] 2> 1913286 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test [junit4] 2> 1913355 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1913371 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true [junit4] 2> 1913371 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1913372 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/control-001/cores/control_collection_shard1_replica_n1/data/] [junit4] 2> 1913374 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0] [junit4] 2> 1913375 WARN (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}} [junit4] 2> 1913413 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1913413 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1913414 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1913414 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1913414 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924 [junit4] 2> 1913415 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6d8f0158[control_collection_shard1_replica_n1] main] [junit4] 2> 1913416 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1913416 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1913416 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1913417 INFO (searcherExecutor-7563-thread-1-processing-n:127.0.0.1:37722_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6d8f0158[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1913418 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613569489973018624 [junit4] 2> 1913423 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0} [junit4] 2> 1913426 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1913426 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1913426 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37722/control_collection_shard1_replica_n1/ [junit4] 2> 1913426 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1913426 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:37722/control_collection_shard1_replica_n1/ has no replicas [junit4] 2> 1913427 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1913430 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37722/control_collection_shard1_replica_n1/ shard1 [junit4] 2> 1913430 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "base_url":"http://127.0.0.1:37722", [junit4] 2> "core":"control_collection_shard1_replica_n1", [junit4] 2> "state":"active"} current state version: 0 [junit4] 2> 1913532 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1913534 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"control_collection_shard1_replica_n1", [junit4] 2> "core_node_name":"core_node2", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:37722", [junit4] 2> "node_name":"127.0.0.1:37722_", [junit4] 2> "numShards":"1", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 0 [junit4] 2> 1913543 INFO (qtp1443383752-70372) [n:127.0.0.1:37722_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1294 [junit4] 2> 1913547 INFO (qtp1443383752-70377) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas [junit4] 2> 1913635 INFO (zkCallback-4828-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1]) [junit4] 2> 1914147 INFO (OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1914547 INFO (qtp1443383752-70377) [n:127.0.0.1:37722_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:37722_&wt=javabin&version=2} status=0 QTime=2407 [junit4] 2> 1914552 INFO (zkConnectionManagerCallback-4847-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1914556 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1914557 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1914557 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 1914558 INFO (qtp1443383752-70379) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 1914561 INFO (OverseerThreadFactory-7558-thread-2-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1 [junit4] 2> 1914565 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "name":"collection1", [junit4] 2> "fromApi":"true", [junit4] 2> "collection.configName":"conf1", [junit4] 2> "numShards":"3", [junit4] 2> "createNodeSet":"", [junit4] 2> "stateFormat":"1", [junit4] 2> "nrtReplicas":"1", [junit4] 2> "replicationFactor":"1", [junit4] 2> "operation":"create"} current state version: 0 [junit4] 2> 1914771 WARN (OverseerThreadFactory-7558-thread-2-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores. [junit4] 2> 1914773 INFO (qtp1443383752-70379) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas [junit4] 2> 1914774 INFO (qtp1443383752-70379) [n:127.0.0.1:37722_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=215 [junit4] 2> 1914884 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001 of type NRT [junit4] 2> 1914885 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11 [junit4] 2> 1914885 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 1914886 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 1914886 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session node0 Scavenging every 660000ms [junit4] 2> 1914886 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22b7039a{/,null,AVAILABLE} [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@18b2b7b7{HTTP/1.1,[http/1.1]}{127.0.0.1:41583} [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server Started @1914931ms [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=41583, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores} [junit4] 2> 1914892 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0 [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1914892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-10-06T09:56:21.798Z [junit4] 2> 1914904 INFO (zkConnectionManagerCallback-4849-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1914905 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1914905 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/solr.xml [junit4] 2> 1914908 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored [junit4] 2> 1914908 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored [junit4] 2> 1914910 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 1915151 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr [junit4] 2> 1915152 INFO (zkConnectionManagerCallback-4853-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1915155 INFO (zkConnectionManagerCallback-4855-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1915162 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1915165 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores [junit4] 2> 1915165 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41583_ [junit4] 2> 1915166 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"downnode", [junit4] 2> "node_name":"127.0.0.1:41583_"} current state version: 1 [junit4] 2> 1915166 INFO (zkCallback-4835-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1915166 INFO (zkCallback-4846-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1915166 INFO (zkCallback-4828-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1915168 INFO (zkCallback-4854-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1915186 INFO (zkConnectionManagerCallback-4862-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1915187 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] 2> 1915188 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1915188 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1915206 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1915216 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1915216 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1915217 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores [junit4] 2> 1915245 INFO (qtp1443383752-70373) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:41583_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 1915248 INFO (OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1915249 INFO (OverseerThreadFactory-7558-thread-3-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:41583_ for creating new replica of shard shard2 for collection collection1 [junit4] 2> 1915251 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ x:collection1_shard2_replica_n61] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT [junit4] 2> 1915253 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 2, message = { [junit4] 2> "core":"collection1_shard2_replica_n61", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:41583", [junit4] 2> "node_name":"127.0.0.1:41583_", [junit4] 2> "numShards":"3", [junit4] 2> "state":"down", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"collection1", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 1 [junit4] 2> 1916266 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0 [junit4] 2> 1916314 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema [collection1_shard2_replica_n61] Schema name=test [junit4] 2> 1916419 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1916480 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n61' using configuration from collection collection1, trusted=true [junit4] 2> 1916481 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n61' (registry 'solr.core.collection1.shard2.replica_n61') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1916481 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [[collection1_shard2_replica_n61] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores/collection1_shard2_replica_n61], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-1-001/cores/collection1_shard2_replica_n61/data/] [junit4] 2> 1916483 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0] [junit4] 2> 1916497 WARN (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}} [junit4] 2> 1916553 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1916554 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1916555 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1916555 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1916555 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924 [junit4] 2> 1916556 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.SolrIndexSearcher Opening [Searcher@74dde355[collection1_shard2_replica_n61] main] [junit4] 2> 1916557 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1916557 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1916558 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1916558 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613569493265547264 [junit4] 2> 1916560 INFO (searcherExecutor-7577-thread-1-processing-n:127.0.0.1:41583_ x:collection1_shard2_replica_n61 c:collection1 s:shard2) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [collection1_shard2_replica_n61] Registered new searcher Searcher@74dde355[collection1_shard2_replica_n61] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1916575 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node62=0}, version=0} [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41583/collection1_shard2_replica_n61/ [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy http://127.0.0.1:41583/collection1_shard2_replica_n61/ has no replicas [junit4] 2> 1916578 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1916582 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41583/collection1_shard2_replica_n61/ shard2 [junit4] 2> 1916584 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"http://127.0.0.1:41583", [junit4] 2> "core":"collection1_shard2_replica_n61", [junit4] 2> "state":"active"} current state version: 2 [junit4] 2> 1916733 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1916734 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1_shard2_replica_n61", [junit4] 2> "core_node_name":"core_node62", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:41583", [junit4] 2> "node_name":"127.0.0.1:41583_", [junit4] 2> "numShards":"3", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard2", [junit4] 2> "collection":"collection1", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 3 [junit4] 2> 1916748 INFO (qtp2023914897-70432) [n:127.0.0.1:41583_ c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1497 [junit4] 2> 1916750 INFO (qtp1443383752-70373) [n:127.0.0.1:37722_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:41583_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1504 [junit4] 2> 1916886 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001 of type NRT [junit4] 2> 1916887 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11 [junit4] 2> 1916888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 1916888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 1916888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session node0 Scavenging every 660000ms [junit4] 2> 1916888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7c456d2e{/,null,AVAILABLE} [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3e21c36{HTTP/1.1,[http/1.1]}{127.0.0.1:46212} [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server Started @1916928ms [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=46212, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores} [junit4] 2> 1916889 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0 [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1916889 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-10-06T09:56:23.795Z [junit4] 2> 1916896 INFO (zkConnectionManagerCallback-4865-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1916897 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1916897 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/solr.xml [junit4] 2> 1916901 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored [junit4] 2> 1916901 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored [junit4] 2> 1916902 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 1917065 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr [junit4] 2> 1917067 INFO (zkConnectionManagerCallback-4869-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1917069 INFO (zkConnectionManagerCallback-4871-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1917074 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] 2> 1917078 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores [junit4] 2> 1917078 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46212_ [junit4] 2> 1917079 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"downnode", [junit4] 2> "node_name":"127.0.0.1:46212_"} current state version: 4 [junit4] 2> 1917080 INFO (zkCallback-4828-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917080 INFO (zkCallback-4846-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917080 INFO (zkCallback-4870-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917080 INFO (zkCallback-4854-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917080 INFO (zkCallback-4861-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917080 INFO (zkCallback-4835-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1917098 INFO (zkConnectionManagerCallback-4878-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1917099 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3) [junit4] 2> 1917100 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1917101 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1917121 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1917131 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1917131 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1917133 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores [junit4] 2> 1917168 INFO (qtp1443383752-70379) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:46212_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 1917170 INFO (OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1917170 INFO (OverseerThreadFactory-7558-thread-4-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ c:collection1 s:shard3 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:46212_ for creating new replica of shard shard3 for collection collection1 [junit4] 2> 1917173 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ x:collection1_shard3_replica_n63] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT [junit4] 2> 1917175 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 2, message = { [junit4] 2> "core":"collection1_shard3_replica_n63", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:46212", [junit4] 2> "node_name":"127.0.0.1:46212_", [junit4] 2> "numShards":"3", [junit4] 2> "state":"down", [junit4] 2> "shard":"shard3", [junit4] 2> "collection":"collection1", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 4 [junit4] 2> 1918190 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0 [junit4] 2> 1918203 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema [collection1_shard3_replica_n63] Schema name=test [junit4] 2> 1918282 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1918294 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n63' using configuration from collection collection1, trusted=true [junit4] 2> 1918295 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n63' (registry 'solr.core.collection1.shard3.replica_n63') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1918295 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [[collection1_shard3_replica_n63] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores/collection1_shard3_replica_n63], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-2-001/cores/collection1_shard3_replica_n63/data/] [junit4] 2> 1918297 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0] [junit4] 2> 1918298 WARN (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}} [junit4] 2> 1918335 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1918335 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1918337 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1918337 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1918337 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924 [junit4] 2> 1918338 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.SolrIndexSearcher Opening [Searcher@379b9080[collection1_shard3_replica_n63] main] [junit4] 2> 1918339 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1918339 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1918340 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1918340 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613569495134109696 [junit4] 2> 1918340 INFO (searcherExecutor-7591-thread-1-processing-n:127.0.0.1:46212_ x:collection1_shard3_replica_n63 c:collection1 s:shard3) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [collection1_shard3_replica_n63] Registered new searcher Searcher@379b9080[collection1_shard3_replica_n63] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1918345 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard3 to Terms{values={core_node64=0}, version=0} [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:46212/collection1_shard3_replica_n63/ [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy http://127.0.0.1:46212/collection1_shard3_replica_n63/ has no replicas [junit4] 2> 1918347 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1918350 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard3", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"http://127.0.0.1:46212", [junit4] 2> "core":"collection1_shard3_replica_n63", [junit4] 2> "state":"active"} current state version: 5 [junit4] 2> 1918350 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:46212/collection1_shard3_replica_n63/ shard3 [junit4] 2> 1918501 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1918503 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1_shard3_replica_n63", [junit4] 2> "core_node_name":"core_node64", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:46212", [junit4] 2> "node_name":"127.0.0.1:46212_", [junit4] 2> "numShards":"3", [junit4] 2> "state":"active", [junit4] 2> "shard":"shard3", [junit4] 2> "collection":"collection1", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 6 [junit4] 2> 1918506 INFO (qtp321590564-70469) [n:127.0.0.1:46212_ c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1333 [junit4] 2> 1918508 INFO (qtp1443383752-70379) [n:127.0.0.1:37722_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:46212_&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1340 [junit4] 2> 1918620 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001 of type NRT [junit4] 2> 1918621 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11 [junit4] 2> 1918623 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0 [junit4] 2> 1918623 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults [junit4] 2> 1918623 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.session node0 Scavenging every 600000ms [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6a0e8493{/,null,AVAILABLE} [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4f3b8cbc{HTTP/1.1,[http/1.1]}{127.0.0.1:40466} [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.e.j.s.Server Started @1918663ms [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40466, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores} [junit4] 2> 1918624 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0 [junit4] 2> 1918624 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1918625 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 1918625 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-10-06T09:56:25.531Z [junit4] 2> 1918626 INFO (zkConnectionManagerCallback-4881-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1918627 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1918627 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/solr.xml [junit4] 2> 1918631 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored [junit4] 2> 1918631 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored [junit4] 2> 1918632 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 1918875 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42889/solr [junit4] 2> 1918876 INFO (zkConnectionManagerCallback-4885-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1918878 INFO (zkConnectionManagerCallback-4887-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1918884 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3) [junit4] 2> 1918888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores [junit4] 2> 1918888 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40466_ [junit4] 2> 1918888 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"downnode", [junit4] 2> "node_name":"127.0.0.1:40466_"} current state version: 7 [junit4] 2> 1918889 INFO (zkCallback-4835-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918889 INFO (zkCallback-4854-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918889 INFO (zkCallback-4846-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918889 INFO (zkCallback-4870-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918892 INFO (zkCallback-4861-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918893 INFO (zkCallback-4886-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918893 INFO (zkCallback-4877-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918893 INFO (zkCallback-4828-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1918907 INFO (zkConnectionManagerCallback-4894-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1918909 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4) [junit4] 2> 1918910 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42889/solr ready [junit4] 2> 1918910 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1918929 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1918940 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1918940 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1918941 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[690FCFDA6B8134D2]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores [junit4] 2> 1919000 INFO (qtp1443383752-70373) [n:127.0.0.1:37722_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40466_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true [junit4] 2> 1919002 INFO (OverseerCollectionConfigSetProcessor-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1919005 INFO (OverseerThreadFactory-7558-thread-5-processing-n:127.0.0.1:37722_) [n:127.0.0.1:37722_ c:collection1 s:shard1 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40466_ for creating new replica of shard shard1 for collection collection1 [junit4] 2> 1919009 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ x:collection1_shard1_replica_n65] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT [junit4] 2> 1919013 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1_shard1_replica_n65", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:40466", [junit4] 2> "node_name":"127.0.0.1:40466_", [junit4] 2> "numShards":"3", [junit4] 2> "state":"down", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "type":"NRT", [junit4] 2> "operation":"state"} current state version: 7 [junit4] 2> 1920026 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0 [junit4] 2> 1920037 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema [collection1_shard1_replica_n65] Schema name=test [junit4] 2> 1920120 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1920132 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n65' using configuration from collection collection1, trusted=true [junit4] 2> 1920133 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n65' (registry 'solr.core.collection1.shard1.replica_n65') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@67453e75 [junit4] 2> 1920133 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [[collection1_shard1_replica_n65] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores/collection1_shard1_replica_n65], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/../../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_690FCFDA6B8134D2-001/shard-3-001/cores/collection1_shard1_replica_n65/data/] [junit4] 2> 1920135 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.541015625, noCFSRatio=0.0] [junit4] 2> 1920141 WARN (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}} [junit4] 2> 1920178 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1920178 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536 [junit4] 2> 1920179 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 1920179 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1920180 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=0.7861328125, floorSegmentMB=1.216796875, forceMergeDeletesPctAllowed=0.6729811829427002, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47373358407785326, deletesPctAllowed=45.33519292309924 [junit4] 2> 1920180 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.s.SolrIndexSearcher Opening [Searcher@b6f9712[collection1_shard1_replica_n65] main] [junit4] 2> 1920181 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1920181 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 1920181 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms. [junit4] 2> 1920182 INFO (searcherExecutor-7605-thread-1-processing-n:127.0.0.1:40466_ x:collection1_shard1_replica_n65 c:collection1 s:shard1) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SolrCore [collection1_shard1_replica_n65] Registered new searcher Searcher@b6f9712[collection1_shard1_replica_n65] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1920182 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1613569497065586688 [junit4] 2> 1920190 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node66=0}, version=0} [junit4] 2> 1920193 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1920193 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1920193 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40466/collection1_shard1_replica_n65/ [junit4] 2> 1920194 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1920194 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.SyncStrategy http://127.0.0.1:40466/collection1_shard1_replica_n65/ has no replicas [junit4] 2> 1920194 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1920198 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40466/collection1_shard1_replica_n65/ shard1 [junit4] 2> 1920198 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"http://127.0.0.1:40466", [junit4] 2> "core":"collection1_shard1_replica_n65", [junit4] 2> "state":"active"} current state version: 8 [junit4] 2> 1920353 INFO (qtp333792483-70506) [n:127.0.0.1:40466_ c:collection1 s:shard1 x:collection1_shard1_replica_n65] o.a.s.c.ZkController I am the leader, no recovery necessary [junit4] 2> 1920354 DEBUG (OverseerStateUpdate-72457920339378180-127.0.0.1:37722_-n_0000000000) [n:127.0.0.1:37722_ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = { [junit4] 2> "core":"collection1_shard1_replica_n65", [junit4] 2> "core_node_name":"core_node66", [junit4] 2> "roles":null, [junit4] 2> "base_url":"http://127.0.0.1:40466", [junit4] 2> "node_name":"127.0.0.1:40466_", [junit4] 2> "numShards":"3", [junit4] [...truncated too long message...] (NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) [junit4] 2> at java.lang.Thread.run(Thread.java:748) [junit4] 2> oct. 06, 2018 8:24:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFOS: Starting to interrupt leaked threads: [junit4] 2> 1) Thread[id=82102, name=TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[690FCFDA6B8134D2], state=TIMED_WAITING, group=TGRP-TestSimGenericDistributedQueue] [junit4] 2> 10799173 INFO (TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[690FCFDA6B8134D2]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueue {#2} [junit4] 2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_690FCFDA6B8134D2-001 [junit4] 2> Oct 06, 2018 12:24:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll [junit4] 2> INFO: All leaked threads terminated. [junit4] 2> NOTE: test params are: codec=Asserting(Lucene80), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1daa8152), locale=fr-CA, timezone=Asia/Hong_Kong [junit4] 2> NOTE: Linux 4.4.0-130-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=4,threads=1,free=188333088,total=497025024 [junit4] 2> NOTE: All tests run in this JVM: [TestCloudDeleteByQuery, BadIndexSchemaTest, TestDistributedStatsComponentCardinality, TestExclusionRuleCollectionAccess, DistributedFacetSimpleRefinementLongTailTest, DefaultValueUpdateProcessorTest, TestStressInPlaceUpdates, ShardRoutingTest, ImplicitSnitchTest, TestAuthorizationFramework, TestClusterProperties, DisMaxRequestHandlerTest, TestLuceneIndexBackCompat, TestCloudJSONFacetSKG, TriLevelCompositeIdRoutingTest, UpdateRequestProcessorFactoryTest, CopyFieldTest, TestConfig, AdminHandlersProxyTest, SuggesterWFSTTest, SimplePostToolTest, DeleteNodeTest, WrapperMergePolicyFactoryTest, SpellCheckCollatorTest, TestConfigSetProperties, CoreMergeIndexesAdminHandlerTest, MultiTermTest, TestSimNodeAddedTrigger, TestRealTimeGet, ZkCLITest, ReplicationFactorTest, OpenExchangeRatesOrgProviderTest, SchemaApiFailureTest, SpellingQueryConverterTest, TestFaceting, TestCloudNestedDocsSort, TestSolr4Spatial, TestBulkSchemaAPI, TestClusterStateMutator, TestHdfsCloudBackupRestore, TestUnifiedSolrHighlighter, URLClassifyProcessorTest, TestOmitPositions, V2StandaloneTest, SOLR749Test, TestComplexPhraseQParserPlugin, ResponseHeaderTest, ScriptEngineTest, UtilsToolTest, IndexSchemaRuntimeFieldTest, TestHashQParserPlugin, BlockJoinFacetRandomTest, TestQueryUtils, BadComponentTest, TestFieldCacheSort, TriggerSetPropertiesIntegrationTest, TestConfigSetsAPIZkFailure, HealthCheckHandlerTest, TestNRTOpen, JsonLoaderTest, DateMathParserTest, TestPayloadScoreQParserPlugin, SolrGangliaReporterTest, NodeMutatorTest, ScheduledTriggerIntegrationTest, TestSolrQueryResponse, TestMiniSolrCloudClusterSSL, TestMaxTokenLenTokenizer, DateRangeFieldTest, MinimalSchemaTest, QueryParsingTest, TestSolrFieldCacheBean, TestSolrCloudSnapshots, SimpleFacetsTest, TestSchemalessBufferedUpdates, TestSimDistribStateManager, TestValueSourceCache, Tagger2Test, BlockCacheTest, TestBulkSchemaConcurrent, TestStressVersions, TestSolrIndexConfig, TestExceedMaxTermLength, SearchHandlerTest, DistributedQueryComponentCustomSortTest, TestEmbeddedSolrServerSchemaAPI, TestSSLRandomization, TestMinMaxOnMultiValuedField, TestDFISimilarityFactory, TestJsonFacetRefinement, SimpleCollectionCreateDeleteTest, TestReversedWildcardFilterFactory, SolrShardReporterTest, XmlInterpolationTest, OverseerCollectionConfigSetProcessorTest, ClassificationUpdateProcessorIntegrationTest, TestCursorMarkWithoutUniqueKey, MetricsHistoryHandlerTest, AutoScalingHandlerTest, SyncSliceTest, TestJsonFacetsWithNestedObjects, TestPivotHelperCode, CreateRoutedAliasTest, TestImplicitCoreProperties, TestSizeLimitedDistributedMap, StatsComponentTest, TestJavabinTupleStreamParser, HdfsDirectoryTest, TestExactSharedStatsCache, NodeAddedTriggerTest, ProtectedTermFilterFactoryTest, MissingSegmentRecoveryTest, AlternateDirectoryTest, AssignTest, TestTolerantUpdateProcessorCloud, TestAddFieldRealTimeGet, SimpleMLTQParserTest, CollectionStateFormat2Test, TestFiltering, MoveReplicaTest, TestSubQueryTransformer, TestFieldCacheWithThreads, TestNumericTerms64, SuggesterFSTTest, RequestLoggingTest, TestSolrConfigHandler, TestCoreDiscovery, OverseerStatusTest, TestRandomRequestDistribution, DistributedSpellCheckComponentTest, HdfsChaosMonkeyNothingIsSafeTest, SmileWriterTest, HdfsUnloadDistributedZkTest, TestDocSet, TestCollationField, NumberUtilsTest, TestSimTriggerIntegration, TestBackupRepositoryFactory, DirectSolrSpellCheckerTest, BasicFunctionalityTest, ConvertedLegacyTest, CursorPagingTest, SolrInfoBeanTest, SolrTestCaseJ4Test, TestDistributedSearch, TestDocumentBuilder, TestGroupingSearch, TestHighlightDedupGrouping, TestJoin, TestRandomFaceting, TestSimpleTrackingShardHandler, TestSolrCoreProperties, TestTolerantSearch, TestTrie, PathHierarchyTokenizerFactoryTest, TestLuceneMatchVersion, TokenizerChainTest, ActionThrottleTest, AddReplicaTest, AliasIntegrationTest, AssignBackwardCompatibilityTest, AsyncCallRequestStatusResponseTest, BasicDistributedZkTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ClusterStateTest, ClusterStateUpdateTest, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, DocValuesNotIndexedTest, ForceLeaderTest, FullSolrCloudDistribCmdsTest, OverseerTest, RecoveryAfterSoftCommitTest, RecoveryZkTest, RemoteQueryErrorTest, ReplaceNodeNoTargetTest, ReplaceNodeTest, SolrCloudExampleTest, TestCloudRecovery, TestConfigSetsAPIExclusivity, TestDeleteCollectionOnDownNodes, TestDistributedMap, TestLeaderElectionWithEmptyReplica, TestPullReplica, TestRequestForwarding, TestShortCircuitedRequests, TestSolrCloudWithDelegationTokens, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressCloudBlindAtomicUpdates, TestZkChroot, TlogReplayBufferedWhileIndexingTest, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, TestReplicaProperties, TestRequestStatusCollectionAPI, AutoAddReplicasIntegrationTest, ExecutePlanActionTest, HdfsAutoAddReplicasIntegrationTest, NodeLostTriggerTest, RestoreTriggerStateTest, SearchRateTriggerTest, TriggerCooldownIntegrationTest, TestSimExtremeIndexing, TestSimGenericDistributedQueue] [junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSimGenericDistributedQueue -Dtests.seed=690FCFDA6B8134D2 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=fr-CA -Dtests.timezone=Asia/Hong_Kong -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.00s J1 | TestSimGenericDistributedQueue (suite) <<< [junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). [junit4] > at __randomizedtesting.SeedInfo.seed([690FCFDA6B8134D2]:0) [junit4] Completed [835/835 (2!)] on J1 in 7280.38s, 4 tests, 2 errors <<< FAILURES! [...truncated 1 lines...] [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/solr/build/solr-core/test/temp/junit4-J1-20181006_092426_814545575240458603412.sysout [junit4] >>> JVM J1 emitted unexpected output (verbatim) ---- [junit4] java.lang.OutOfMemoryError: Java heap space [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/heapdumps/java_pid24953.hprof ... [junit4] Heap dump file created [594292755 bytes in 8.552 secs] [junit4] <<< JVM J1: EOF ---- [...truncated 8961 lines...] BUILD FAILED /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:651: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/build.xml:585: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created: * java_pid24953.hprof Total time: 369 minutes 25 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org