Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3509/ Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed. FAILED: org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry Error Message: Stack Trace: java.lang.AssertionError at __randomizedtesting.SeedInfo.seed([F49C4FF42EEF100B:22436EAE18F20393]:0) at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry(TestLeaderElectionZkExpiry.java:95) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) at java.base/java.lang.Thread.run(Thread.java:834) Build Log: [...truncated 14326 lines...] [junit4] Suite: org.apache.solr.cloud.TestLeaderElectionZkExpiry [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001/init-core-data-001 [junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false [junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0) [junit4] 2> 1313771 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom [junit4] 2> 1313773 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLeaderElectionWithZkExpiry [junit4] 2> 1313776 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2, but no JMX reporters were configured - adding default JMX reporter. [junit4] 2> 1313854 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true] [junit4] 2> 1313867 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores [junit4] 2> 1313867 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. [junit4] 2> 1313877 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2 [junit4] 2> 1313882 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2 [junit4] 2> 1313882 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0414f2 [junit4] 2> 1313883 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001/testLeaderElectionWithZkExpiry-solr-001 [junit4] 2> 1313884 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 1313884 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1313885 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Starting server [junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer start zk server on port:43509 [junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43509 [junit4] 2> 1313985 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43509 [junit4] 2> 1313987 INFO (zkConnectionManagerCallback-8818-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1313989 INFO (zkConnectionManagerCallback-8820-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1313990 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1314017 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1314017 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) starting [junit4] 2> 1314019 INFO (zkConnectionManagerCallback-8829-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1314021 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:43509/solr ready [junit4] 2> 1314021 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr [junit4] 2> 1314021 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 1314022 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1314022 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1314023 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1314123 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1314123 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1315946 WARN (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0002 has expired [junit4] 2> 1315946 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1315946 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1315946 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 1315946 WARN (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Solr cannot talk to ZK, exiting Overseer main queue loop [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:446) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:212) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1315947 INFO (OverseerStateUpdate-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 1315948 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1316047 WARN (zkCallback-8821-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 1316047 WARN (ScheduledTrigger-7314-thread-1) [ ] o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger .scheduled_maintenance [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:509) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:150) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:611) [java/:?] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1316047 ERROR (OverseerAutoScalingTriggerThread-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1316048 WARN (OverseerAutoScalingTriggerThread-72060270075838466-dummy.host.com:8984_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1316048 WARN (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1316049 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1316050 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1316051 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72060270075838466-dummy.host.com:8984_solr-n_0000000000) closing [junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1316052 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) starting [junit4] 2> 1316054 INFO (OverseerStateUpdate-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr [junit4] 2> 1316054 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN [junit4] 2> 1316055 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn CancelledKeyException causing close of session 0x100026f104a0004 [junit4] 2> 1316155 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1316155 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0004 has expired [junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1317409 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1317409 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 1317409 INFO (OverseerStateUpdate-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 1317420 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1317655 WARN (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Could not publish node as down: KeeperErrorCode = Session expired for /overseer/queue/qn- [junit4] 2> 1317655 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 1317655 ERROR (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .scheduled_maintenance [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1317655 ERROR (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:215) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:201) [junit4] 2> at org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1082) [junit4] 2> at org.apache.solr.cloud.ZkController.access$500(ZkController.java:149) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:376) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:285) [junit4] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [junit4] 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834) [junit4] 2> [junit4] 2> 1317655 ERROR (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1317656 ERROR (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:428) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:285) [junit4] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [junit4] 2> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) [junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:215) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.getAutoScalingConfig(ZkStateReader.java:201) [junit4] 2> at org.apache.solr.cloud.ZkController.createEphemeralLiveNode(ZkController.java:1082) [junit4] 2> at org.apache.solr.cloud.ZkController.access$500(ZkController.java:149) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:376) [junit4] 2> ... 10 more [junit4] 2> [junit4] 2> 1317656 WARN (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 1317656 INFO (zkConnectionManagerCallback-8822-thread-1) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:false [junit4] 2> 1317656 WARN (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1317656 WARN (OverseerAutoScalingTriggerThread-72060270075838468-dummy.host.com:8984_solr-n_0000000001) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1317656 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1317657 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838468-dummy.host.com:8984_solr-n_0000000001) closing [junit4] 2> 1317659 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1317660 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) starting [junit4] 2> 1317661 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN [junit4] 2> 1317661 INFO (OverseerStateUpdate-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr [junit4] 2> 1317661 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 1317662 INFO (zkCallback-8821-thread-5) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1317662 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper [junit4] 2> 1317662 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true [junit4] 2> 1317763 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1317763 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0005 has expired [junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1319246 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1319246 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 1319247 INFO (OverseerStateUpdate-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 1319247 ERROR (OverseerCollectionConfigSetProcessor-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer [junit4] 2> java.lang.InterruptedException: sleep interrupted [junit4] 2> at java.lang.Thread.sleep(Native Method) ~[?:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:126) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:81) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:68) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:187) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1319247 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1319263 WARN (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 1319263 ERROR (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .scheduled_maintenance [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1319264 ERROR (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1319264 WARN (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1319264 WARN (OverseerAutoScalingTriggerThread-72060270075838469-dummy.host.com:8984_solr-n_0000000002) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1319265 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1319265 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1319266 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838469-dummy.host.com:8984_solr-n_0000000002) closing [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1319267 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) starting [junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN [junit4] 2> 1319269 INFO (OverseerStateUpdate-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr [junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper [junit4] 2> 1319269 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true [junit4] 2> 1319270 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1319275 INFO (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1319275 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1319375 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1319376 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0006 has expired [junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1320563 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1320563 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1320563 WARN (OverseerCollectionConfigSetProcessor-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Overseer cannot talk to ZK [junit4] 2> 1320563 INFO (OverseerStateUpdate-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 1320564 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer Error communicating with ZooKeeper [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:319) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1320875 WARN (zkCallback-8821-thread-6) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes] [junit4] 2> 1320875 WARN (ScheduledTrigger-7326-thread-1) [ ] o.a.s.c.a.TriggerEventQueue Exception peeking queue of trigger .scheduled_maintenance [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.scheduled_maintenance [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.fetchZkChildren(ZkDistributedQueue.java:415) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstChild(ZkDistributedQueue.java:392) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.firstElement(ZkDistributedQueue.java:509) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.peek(ZkDistributedQueue.java:150) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.peekEvent(TriggerEventQueue.java:69) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:611) [java/:?] [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] [junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?] [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1320875 ERROR (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.ScheduledTriggers Failed to add trigger .auto_add_replicas [junit4] 2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:119) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:115) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:111) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueueFactory.makeQueue(ZkDistributedQueueFactory.java:36) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.TriggerEventQueue.<init>(TriggerEventQueue.java:49) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.<init>(ScheduledTriggers.java:566) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:220) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:245) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/events/.auto_add_replicas [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?] [junit4] 2> ... 9 more [junit4] 2> 1320877 WARN (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:261) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1320877 WARN (OverseerAutoScalingTriggerThread-72060270075838470-dummy.host.com:8984_solr-n_0000000003) [ ] o.a.s.c.a.OverseerTriggerThread Error removing old markers [junit4] 2> org.apache.solr.common.AlreadyClosedException: null [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:74) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.impl.ZkDistribStateManager.listData(ZkDistribStateManager.java:80) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.listTree(DistribStateManager.java:104) ~[java/:?] [junit4] 2> at org.apache.solr.client.solrj.cloud.DistribStateManager.removeRecursively(DistribStateManager.java:123) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.removeMarkers(OverseerTriggerThread.java:269) [java/:?] [junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:262) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1320877 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1320878 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0) [junit4] 2> 1320983 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1320983 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0007 has expired [junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1322315 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1322315 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1322424 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1322424 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0008 has expired [junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1323900 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1323900 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1323901 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1324001 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$2(SolrZkClient.java:322) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:322) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.addSecurityNodeWatcher(ZkStateReader.java:485) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:468) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:343) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:533) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) [junit4] 2> [junit4] 2> 1324001 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed:org.apache.solr.common.cloud.ZooKeeperException: [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:428) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:177) [junit4] 2> at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:58) [junit4] 2> at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:151) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:533) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /security.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:130) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$2(SolrZkClient.java:322) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:322) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.addSecurityNodeWatcher(ZkStateReader.java:485) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:468) [junit4] 2> at org.apache.solr.cloud.ZkController$1.command(ZkController.java:343) [junit4] 2> ... 5 more [junit4] 2> [junit4] 2> 1324001 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnect to ZooKeeper failed [junit4] 2> 1324001 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient Connected:true [junit4] 2> 1324002 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None [junit4] 2> 1324002 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected [junit4] 2> 1324024 INFO (zkConnectionManagerCallback-8831-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-SendThread(localhost.localdomain:43509)) [ ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session 0x100026f104a0009 has expired [junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@787984be name: ZooKeeperConnection Watcher:127.0.0.1:43509/solr got event WatchedEvent state:Expired type:None path:null path: null type: None [junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... [junit4] 2> 1325098 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1325098 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... [junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager zkClient has connected [junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. [junit4] 2> 1325099 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. [junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838470-dummy.host.com:8984_solr-n_0000000003) closing [junit4] 2> 1325504 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1325505 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) starting [junit4] 2> 1325506 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325506 INFO (OverseerStateUpdate-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Starting to work on the main queue : dummy.host.com:8984_solr [junit4] 2> 1325506 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Publish node=dummy.host.com:8984_solr as DOWN [junit4] 2> 1325506 INFO (OverseerStateUpdate-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer Loop exiting : dummy.host.com:8984_solr [junit4] 2> 1325508 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/dummy.host.com:8984_solr [junit4] 2> 1325508 ERROR (OverseerCollectionConfigSetProcessor-72060270075838475-dummy.host.com:8984_solr-n_0000000004) [n:dummy.host.com:8984_solr ] o.a.s.c.OverseerTaskProcessor Unable to prioritize overseer [junit4] 2> java.lang.InterruptedException: null [junit4] 2> at java.lang.Object.wait(Native Method) ~[?:?] [junit4] 2> at java.lang.Object.wait(Object.java:328) ~[?:?] [junit4] 2> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1411) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1106) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?] [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:334) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.OverseerNodePrioritizer.prioritizeOverseerNodes(OverseerNodePrioritizer.java:68) ~[java/:?] [junit4] 2> at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:187) [java/:?] [junit4] 2> at java.lang.Thread.run(Thread.java:834) [?:?] [junit4] 2> 1325508 WARN (OverseerExitThread) [ ] o.a.s.c.Overseer I (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) am exiting, but I'm still the leader [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkCallback-8828-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1325509 INFO (zkCallback-8821-thread-5) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [n:dummy.host.com:8984_solr ] o.a.s.c.c.ConnectionManager zkClient Connected:true [junit4] 2> 1325509 INFO (OverseerExitThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000004) closing [junit4] 2> 1325509 WARN (OverseerExitThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000005 [junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000006 [junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000005 [junit4] 2> 1325509 WARN (OverseerExitThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000006 [junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000007 [junit4] 2> 1325509 WARN (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector Deleting duplicate registration: /overseer_elect/election/72060270075838475-dummy.host.com:8984_solr-n_0000000007 [junit4] 2> 1325510 INFO (OverseerExitThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1325510 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1325510 INFO (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader dummy.host.com:8984_solr [junit4] 2> 1325510 INFO (OverseerExitThread) [n:dummy.host.com:8984_solr ] o.a.s.c.Overseer Overseer (id=72060270075838475-dummy.host.com:8984_solr-n_0000000007) starting [junit4] 2> 1325510 ERROR (zkConnectionManagerCallback-8822-thread-1-EventThread) [ ] o.a.s.c.LeaderElector node exists [junit4] 2> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:122) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03] [junit4] [...truncated too long message...] ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null [junit4] 2> 1384033 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3e2e79f7: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3ededc45 [junit4] 2> 1384034 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer. [junit4] 2> 1384035 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations: [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2> [junit4] 2> 9 /solr/aliases.json [junit4] 2> 9 /solr/clusterprops.json [junit4] 2> 7 /solr/security.json [junit4] 2> [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2> [junit4] 2> 9 /solr/clusterstate.json [junit4] 2> 4 /solr/autoscaling.json [junit4] 2> [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2> [junit4] 2> 9 /solr/collections [junit4] 2> 8 /solr/live_nodes [junit4] 2> 3 /solr/overseer/queue [junit4] 2> 3 /solr/overseer/queue-work [junit4] 2> 2 /solr/overseer/collection-queue-work [junit4] 2> [junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43509 [junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43509 [junit4] 2> 1384037 INFO (TEST-TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry-seed#[F49C4FF42EEF100B]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testLeaderElectionWithZkExpiry [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] FAILURE 70.3s J1 | TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry <<< [junit4] > Throwable #1: java.lang.AssertionError [junit4] > at __randomizedtesting.SeedInfo.seed([F49C4FF42EEF100B:22436EAE18F20393]:0) [junit4] > at org.apache.solr.cloud.TestLeaderElectionZkExpiry.testLeaderElectionWithZkExpiry(TestLeaderElectionZkExpiry.java:95) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566) [junit4] > at java.base/java.lang.Thread.run(Thread.java:834) [junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=922172301 [junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null [junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null [junit4] 2> 1384040 INFO (SUITE-TestLeaderElectionZkExpiry-seed#[F49C4FF42EEF100B]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null [junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestLeaderElectionZkExpiry_F49C4FF42EEF100B-001 [junit4] 2> Feb 06, 2019 2:42:05 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 5 leaked thread(s). [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=dsb, timezone=America/Asuncion [junit4] 2> NOTE: Linux 4.15.0-45-generic amd64/Oracle Corporation 11 (64-bit)/cpus=8,threads=1,free=245205472,total=536870912 [junit4] 2> NOTE: All tests run in this JVM: [MetricsHistoryHandlerTest, RestartWhileUpdatingTest, TestTestInjection, TestReversedWildcardFilterFactory, DeleteStatusTest, TestTrieFacet, MultiSolrCloudTestCaseTest, TestSolrCoreParser, UtilsToolTest, BasicFunctionalityTest, TestSimLargeCluster, TestManagedResourceStorage, CollectionStateFormat2Test, TestCloudPivotFacet, SubstringBytesRefFilterTest, TestExpandComponent, SuggestComponentContextFilterQueryTest, TestFuzzyAnalyzedSuggestions, TestLRUCache, TestIndexSearcher, TestDocTermOrdsUninvertLimit, TestSearchPerf, OutOfBoxZkACLAndCredentialsProvidersTest, CSVRequestHandlerTest, URLClassifyProcessorTest, TestManagedSchemaAPI, TestMissingGroups, TestCloudNestedDocsSort, TestPartialUpdateDeduplication, SegmentsInfoRequestHandlerTest, JsonLoaderTest, TestBinaryResponseWriter, MergeStrategyTest, SolrCloudExampleTest, BadComponentTest, BasicAuthIntegrationTest, TestTolerantSearch, TestReRankQParserPlugin, BooleanFieldTest, TestShortCircuitedRequests, ZkSolrClientTest, DistributedUpdateProcessorTest, TestCloudDeleteByQuery, TestCollationFieldDocValues, SimpleMLTQParserTest, TestUniqueKeyFieldResource, TestMacros, TestRecoveryHdfs, SharedFSAutoReplicaFailoverTest, TestOverriddenPrefixQueryForCustomFieldType, AdminHandlersProxyTest, TransactionLogTest, TestJsonFacetsWithNestedObjects, SpellingQueryConverterTest, TermVectorComponentTest, TestPerFieldSimilarity, TriggerIntegrationTest, BlockJoinFacetRandomTest, TemplateUpdateProcessorTest, TestBlobHandler, TestPrepRecovery, SolrGraphiteReporterTest, IndexSchemaTest, TestCopyFieldCollectionResource, DateMathParserTest, TestSolrDeletionPolicy1, TestRandomFaceting, TestCorePropertiesReload, DistributedFacetExistsSmallTest, TestPolicyCloud, HttpPartitionOnCommitTest, LeaderTragicEventTest, ResponseHeaderTest, SpatialHeatmapFacetsTest, LukeRequestHandlerTest, DistributedQueryElevationComponentTest, DirectSolrSpellCheckerTest, TestSolrXml, TestSubQueryTransformerCrossCore, TestIntervalFaceting, DistribJoinFromCollectionTest, HdfsRestartWhileUpdatingTest, TestPointFields, TestConfigSets, TestDynamicLoading, DisMaxRequestHandlerTest, TestRawTransformer, TestShardHandlerFactory, RulesTest, UUIDFieldTest, TestLegacyFieldReuse, RequestLoggingTest, TestDownShardTolerantSearch, TestBlendedInfixSuggestions, ShardSplitTest, ActionThrottleTest, HLLUtilTest, CdcrWithNodesRestartsTest, DistributedMLTComponentTest, RankQueryTest, TestRTGBase, ClusterStateTest, DirectSolrConnectionTest, DistributedTermsComponentTest, RecoveryAfterSoftCommitTest, TestCursorMarkWithoutUniqueKey, PolyFieldTest, TestReplicaProperties, PluginInfoTest, TestManagedSynonymGraphFilterFactory, TestChildDocTransformer, TestStressCloudBlindAtomicUpdates, TestComponentsName, TestSQLHandler, SchemaVersionSpecificBehaviorTest, DocValuesNotIndexedTest, HdfsNNFailoverTest, SaslZkACLProviderTest, PeerSyncWithLeaderAndIndexFingerprintCachingTest, V2StandaloneTest, PeerSyncTest, TestSolrQueryParser, TestFieldSortValues, TestIndexingPerformance, TestPHPSerializedResponseWriter, SuggesterWFSTTest, TestStressLucene, FastVectorHighlighterTest, PhrasesIdentificationComponentTest, TestDFISimilarityFactory, PrimitiveFieldTypeTest, CoreMergeIndexesAdminHandlerTest, TestMaxTokenLenTokenizer, TestLockTree, VersionInfoTest, TestAuthorizationFramework, AssignBackwardCompatibilityTest, BasicDistributedZkTest, BasicZkTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeySafeLeaderTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DistributedVersionInfoTest, LeaderElectionTest, LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, OverseerModifyCollectionTest, OverseerTaskQueueTest, PeerSyncReplicationTest, TestCloudConsistency, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDistributedMap, TestHashPartitioner, TestLeaderElectionZkExpiry] [junit4] Completed [513/838 (1!)] on J1 in 71.05s, 1 test, 1 failure <<< FAILURES! [...truncated 43636 lines...] [repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3509/consoleText [repro] Revision: 2835e086537ee708dc6037e19b33c5aa3c287a15 [repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" [repro] Repro line: ant test -Dtestcase=TestLeaderElectionZkExpiry -Dtests.method=testLeaderElectionWithZkExpiry -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [repro] ant clean [...truncated 6 lines...] [repro] Test suites by module: [repro] solr/core [repro] TestLeaderElectionZkExpiry [repro] ant compile-test [...truncated 3588 lines...] [repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.TestLeaderElectionZkExpiry" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=F49C4FF42EEF100B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=dsb -Dtests.timezone=America/Asuncion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [...truncated 75 lines...] [repro] Failures: [repro] 0/5 failed: org.apache.solr.cloud.TestLeaderElectionZkExpiry [repro] Exiting with code 0 [...truncated 43 lines...]
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org