Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10977/ Java: 32bit/jdk1.7.0_65 -client -XX:+UseG1GC
1 tests failed. REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure Error Message: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed Stack Trace: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed at __randomizedtesting.SeedInfo.seed([66EAD145A0F8583E:62E25EB6B25DB71F]:0) at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221) at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155) at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398) at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136) at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31) at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131) ... 47 more Build Log: [...truncated 11494 lines...] [junit4] Suite: org.apache.solr.cloud.OverseerTest [junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-66EAD145A0F8583E-001/init-core-data-001 [junit4] 2> 1571545 T4229 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false) [junit4] 2> 1571546 T4229 oas.SolrTestCaseJ4.initCore ####initCore [junit4] 2> 1571546 T4229 oas.SolrTestCaseJ4.initCore ####initCore end [junit4] 2> 1571551 T4229 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure [junit4] 2> 1571552 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1571553 T4230 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 1571652 T4229 oasc.ZkTestServer.run start zk server on port:44278 [junit4] 2> 1571654 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1571656 T4236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437f97 name:ZooKeeperConnection Watcher:127.0.0.1:44278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1571656 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1571658 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1571659 T4238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59d421 name:ZooKeeperConnection Watcher:127.0.0.1:44278 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1571659 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1571659 T4229 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 1571662 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1571663 T4240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b191d7 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1571663 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1571664 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 1571666 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 1571668 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 1571669 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1571672 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1571673 T4242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1915eda name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1571673 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1571674 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1571677 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 1571680 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1571681 T4246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b706d5 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1571681 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1571682 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1571684 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1571687 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44278_solr [junit4] 2> 1571687 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1571689 T4229 oasc.Overseer.start Overseer (id=92241659301527556-127.0.0.1:44278_solr-n_0000000000) starting [junit4] 2> 1571690 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1571692 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1571694 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1571697 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1571698 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1571700 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1571703 T4248 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1571704 T4247 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1572705 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1572706 T4247 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1572707 T4247 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"recovering", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 1572707 T4247 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1] [junit4] 2> 1572708 T4247 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1572713 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1572713 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1573205 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1573211 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1573215 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1573215 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1573216 T4247 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1573218 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1573219 T4247 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 1573219 T4247 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 1573220 T4249 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1573221 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1573221 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1573705 T4248 oasc.OverseerCollectionProcessor.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:514) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:269) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 1573706 T4248 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92241659301527556-127.0.0.1:44278_solr-n_0000000000) am no longer a leader. [junit4] 2> 1574276 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1574280 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1574280 T4251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@989290 name:ZooKeeperConnection Watcher:127.0.0.1:44278/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1574281 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1574282 T4247 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44278_solr [junit4] 2> 1574282 T4252 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 1574284 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44278_solr [junit4] 2> 1574284 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1574286 T4229 oasc.Overseer.start Overseer (id=92241659301527557-127.0.0.1:44278_solr-n_0000000001) starting [junit4] 2> 1574292 T4254 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1574292 T4253 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1574294 T4253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1574295 T4253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"recovering", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 1574295 T4253 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 1574297 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1574297 T4244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1574297 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1574350 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1574350 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1574351 T4253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1574354 T4229 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/92241659301527555-node1_core1-n_0000000001 [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:148) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314) [junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221) [junit4] 2> at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155) [junit4] 2> at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [junit4] 2> at java.lang.reflect.Method.invoke(Method.java:606) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 1574356 T4256 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1574355 T4253 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:184) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:181) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:181) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1094) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:359) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:242) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 1574354 T4244 oasc.LeaderElector.checkIfIamLeader WARN Our node is no longer in line to be leader [junit4] 2> 1574358 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1574361 T4255 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1574362 T4253 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1 [junit4] 2> 1574465 T4256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1574465 T4243 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1575361 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1576364 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1577367 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1578370 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1579373 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1580378 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1581380 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1582383 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1583386 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1584389 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1585391 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1586394 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1587397 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1588400 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1589403 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1589411 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44278 44278 [junit4] 2> 1590247 T4230 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44278 44278 [junit4] 2> 1590248 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure [junit4] 2> 1590249 T4253 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44278_solr [junit4] 2> 1590249 T4257 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=66EAD145A0F8583E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in -Dtests.timezone=Etc/GMT+2 -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 18.7s J1 | OverseerTest.testOverseerFailure <<< [junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed [junit4] > at __randomizedtesting.SeedInfo.seed([66EAD145A0F8583E:62E25EB6B25DB71F]:0) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144) [junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163) [junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125) [junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155) [junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314) [junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221) [junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155) [junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1 [junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) [junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) [junit4] > at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457) [junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136) [junit4] > at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131) [junit4] > ... 47 more [junit4] 2> 1590285 T4229 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem [junit4] 2> 1590286 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1590287 T4258 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 1590387 T4229 oasc.ZkTestServer.run start zk server on port:38232 [junit4] 2> 1590388 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1590390 T4264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4721e2 name:ZooKeeperConnection Watcher:127.0.0.1:38232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1590390 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1590393 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1590394 T4266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@156c819 name:ZooKeeperConnection Watcher:127.0.0.1:38232 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1590394 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1590395 T4229 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 1590398 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1590399 T4268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@111fde2 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1590399 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1590400 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 1590402 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1590403 T4270 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dc221 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1590403 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1590404 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1590405 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1590408 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38232_solr [junit4] 2> 1590408 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1590409 T4229 oasc.Overseer.start Overseer (id=92241660529278979-127.0.0.1:38232_solr-n_0000000000) starting [junit4] 2> 1590411 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1590412 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1590414 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1590416 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1590417 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1590419 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1590422 T4272 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1590423 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 1590423 T4271 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1590424 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 1590425 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1590427 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1590428 T4274 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3db668 name:ZooKeeperConnection Watcher:127.0.0.1:38232/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1590428 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1590429 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1590430 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1 [junit4] 2> 1590433 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1590434 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3] [junit4] 2> 1590435 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2 [junit4] 2> 1590437 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1590437 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1590933 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election [junit4] 2> 1590939 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2 [junit4] 2> 1590943 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1590944 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core2", [junit4] 2> "core_node_name":"node2", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 1590946 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1590948 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1590949 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1590949 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1591444 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 1591449 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 1591452 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1591453 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1591455 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core3", [junit4] 2> "core_node_name":"node3", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1591456 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 1591456 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3 [junit4] 2> 1591457 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1591458 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1591458 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1591954 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election [junit4] 2> 1591959 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3 [junit4] 2> 1591962 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1591963 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1592065 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1592065 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1592113 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1592115 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1592116 T4271 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: { [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core0", [junit4] 2> "core_node_name":"node0", [junit4] 2> "collection":"", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1592118 T4271 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: { [junit4] 2> "operation":"deletecore", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core0", [junit4] 2> "core_node_name":"node0", [junit4] 2> "collection":""} [junit4] 2> 1592120 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1592120 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection2", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1592121 T4271 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3] [junit4] 2> 1592121 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2 [junit4] 2> 1592224 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1592224 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1592617 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election [junit4] 2> 1592624 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2 [junit4] 2> 1592631 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1592633 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1592638 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core2", [junit4] 2> "core_node_name":"node2", [junit4] 2> "collection":"collection2", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1592639 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 1592639 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1592641 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1592642 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1592642 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1593132 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election [junit4] 2> 1593138 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1 [junit4] 2> 1593141 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1593142 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1593144 T4271 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core3", [junit4] 2> "core_node_name":"node3", [junit4] 2> "collection":"collection2", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 1593144 T4271 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 1593145 T4271 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3 [junit4] 2> 1593146 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1593148 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1593148 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1593642 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election [junit4] 2> 1593648 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3 [junit4] 2> 1593651 T4277 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1593653 T4271 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1593758 T4276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1593758 T4275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1593806 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38232 38232 [junit4] 2> 1594253 T4258 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:38232 38232 [junit4] 2> 1594254 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem [junit4] 2> 1594254 T4271 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:38232_solr [junit4] 2> 1594255 T4278 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 1594258 T4229 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders [junit4] 2> 1594259 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1594260 T4279 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 1594360 T4229 oasc.ZkTestServer.run start zk server on port:48272 [junit4] 2> 1594361 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1594367 T4285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6845b9 name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1594368 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1594369 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1594370 T4287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9e54c name:ZooKeeperConnection Watcher:127.0.0.1:48272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1594370 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1594372 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1594373 T4289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be2ae5 name:ZooKeeperConnection Watcher:127.0.0.1:48272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1594373 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1594374 T4229 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 1594376 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 1594378 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 1594380 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 1594381 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1594383 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1594384 T4291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ca9788 name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1594384 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1594385 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1594387 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 1594394 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1594394 T4295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e0354b name:ZooKeeperConnection Watcher:127.0.0.1:48272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1594395 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1594397 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1594398 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1594401 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48272_solr [junit4] 2> 1594401 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1594403 T4229 oasc.Overseer.start Overseer (id=92241660789653508-127.0.0.1:48272_solr-n_0000000000) starting [junit4] 2> 1594404 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1594406 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1594408 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1594410 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1594412 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1594414 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1594417 T4297 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1594418 T4296 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1594420 T4296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"recovering", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"12", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12] [junit4] 2> 1594421 T4296 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2 [junit4] 2> 1594424 T4298 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1594424 T4292 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1594431 T4293 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 1594920 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election [junit4] 2> 1594926 T4229 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2 [junit4] 2> 1594929 T4298 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1594929 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1594930 T4296 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1594931 T4296 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterstate.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ClusterState.load(ClusterState.java:239) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:445) [junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:242) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:234) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 1594932 T4296 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48272_solr [junit4] 2> 1594933 T4299 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 1596420 T4297 oasc.OverseerCollectionProcessor.amILeader org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:514) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:269) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 1596421 T4297 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92241660789653508-127.0.0.1:48272_solr-n_0000000000) am no longer a leader. [junit4] 2> 1596432 T4292 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 1596435 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48272 48272 [junit4] 2> 1597028 T4279 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48272 48272 [junit4] 2> 1597029 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders [junit4] 2> 1597033 T4229 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger [junit4] 2> 1597034 T4229 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1597035 T4300 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 1597135 T4229 oasc.ZkTestServer.run start zk server on port:37174 [junit4] 2> 1597136 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597138 T4306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f86206 name:ZooKeeperConnection Watcher:127.0.0.1:37174 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597138 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597140 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597141 T4308 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@911f76 name:ZooKeeperConnection Watcher:127.0.0.1:37174 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597141 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597142 T4229 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 1597147 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597148 T4310 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6814f3 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597148 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597149 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 1597151 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597151 T4312 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1886595 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597152 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597153 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1597154 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1597157 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:37174_solr [junit4] 2> 1597157 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1597159 T4229 oasc.Overseer.start Overseer (id=92241660971515907-127.0.0.1:37174_solr-n_0000000000) starting [junit4] 2> 1597160 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1597162 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1597164 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1597165 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1597167 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1597169 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1597172 T4314 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1597173 T4313 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1597172 T4229 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 1597174 T4229 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 1597175 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597177 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597178 T4316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c651c name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597179 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597180 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597181 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node0 [junit4] 2> 1597183 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597184 T4320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b6ce8a name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597185 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597187 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597189 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 1597191 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597192 T4323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bd290 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597192 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597193 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597195 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node2 [junit4] 2> 1597198 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597198 T4326 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@633195 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597199 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597201 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597203 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node3 [junit4] 2> 1597206 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597206 T4329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56716b name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597207 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597208 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597209 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node4 [junit4] 2> 1597211 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597212 T4332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bd2054 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597213 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597213 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597215 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node5 [junit4] 2> 1597218 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597218 T4335 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@880735 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597219 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597220 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597221 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node6 [junit4] 2> 1597224 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597225 T4338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5666ad name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597226 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597227 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597228 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node7 [junit4] 2> 1597231 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597232 T4341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ec740 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597232 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597233 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597234 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node8 [junit4] 2> 1597238 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597239 T4344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f44fb2 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597239 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597240 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597241 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node9 [junit4] 2> 1597245 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597246 T4347 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e4b02a name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597246 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597247 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597249 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node10 [junit4] 2> 1597253 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597254 T4350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@757e3f name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597254 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597255 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597256 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node11 [junit4] 2> 1597260 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597261 T4353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14a871 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597262 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597262 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597264 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node12 [junit4] 2> 1597268 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597269 T4356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8cb2cf name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597269 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597270 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597272 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node13 [junit4] 2> 1597276 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597277 T4359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16dc3b5 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597277 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597278 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597279 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node14 [junit4] 2> 1597284 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597285 T4362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12dae6d name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597286 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597287 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597289 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node15 [junit4] 2> 1597293 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597294 T4365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1316bf8 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597295 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597296 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597297 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node16 [junit4] 2> 1597304 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597304 T4368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdd2df name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597305 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597306 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597308 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node17 [junit4] 2> 1597312 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597313 T4371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ff248 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597313 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597314 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597316 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node18 [junit4] 2> 1597320 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597321 T4374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da4bb0 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597321 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597322 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597324 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node19 [junit4] 2> 1597328 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597329 T4377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ec993 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597329 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597330 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597331 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node20 [junit4] 2> 1597335 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597336 T4380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@183989d name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597336 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597337 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597339 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node21 [junit4] 2> 1597342 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597344 T4382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8244 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597344 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597345 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597346 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node22 [junit4] 2> 1597351 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597352 T4386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b11909 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597352 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597353 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597354 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node23 [junit4] 2> 1597358 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597359 T4389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ca9531 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597360 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597360 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597362 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node24 [junit4] 2> 1597367 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597368 T4392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@baba79 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597368 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597369 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597370 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node25 [junit4] 2> 1597375 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597376 T4395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dbc447 name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597376 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597377 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597379 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node26 [junit4] 2> 1597385 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597386 T4398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d1fcaa name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1597386 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1597387 T4229 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 1597388 T4229 oascc.SolrZkClient.makePath makePath: /live_nodes/node27 [junit4] 2> 1597395 T4229 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 1597396 T4401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bdc83f name:ZooKeeperConnection Watcher:127.0.0.1:37174/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] [...truncated too long message...] cc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19cc474 name:ZooKeeperConnection Watcher:127.0.0.1:47578/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 1648911 T4229 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 1648912 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 1648913 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 1648915 T4229 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47578_solr [junit4] 2> 1648915 T4229 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 1648917 T4229 oasc.Overseer.start Overseer (id=92241664363397123-127.0.0.1:47578_solr-n_0000000000) starting [junit4] 2> 1648918 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 1648920 T4229 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 1648922 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 1648923 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 1648925 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 1648927 T4229 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 1648931 T4927 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 1648931 T4926 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 1648932 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1648933 T4926 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1648934 T4926 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={ [junit4] 2> "operation":"state", [junit4] 2> "base_url":"http://127.0.0.1/solr", [junit4] 2> "node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "core":"core1", [junit4] 2> "roles":"", [junit4] 2> "state":"recovering"} [junit4] 2> 1648934 T4926 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 1648935 T4928 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1648936 T4929 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0) [junit4] 2> 1648984 T4229 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1648986 T4928 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 1648988 T4926 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 1648989 T4926 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={ [junit4] 2> "operation":"state", [junit4] 2> "base_url":"http://127.0.0.1/solr", [junit4] 2> "node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "core":"core1", [junit4] 2> "roles":"", [junit4] 2> "state":"active"} [junit4] 2> 1648990 T4926 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered [junit4] 2> 1648990 T4926 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 1649095 T4929 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0) [junit4] 2> 1649140 T4229 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47578 47578 [junit4] 2> 1649712 T4913 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47578 47578 [junit4] 2> 1649716 T4229 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange [junit4] 2> 1649717 T4926 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:47578_solr [junit4] 2> 1649717 T4930 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 1652719 T4229 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.OverseerTest-66EAD145A0F8583E-001 [junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=in, timezone=Etc/GMT+2 [junit4] 2> NOTE: Linux 3.13.0-32-generic i386/Oracle Corporation 1.7.0_65 (32-bit)/cpus=8,threads=1,free=311994352,total=525336576 [junit4] 2> NOTE: All tests run in this JVM: [HttpPartitionTest, OpenCloseCoreStressTest, TestUtils, DistribCursorPagingTest, TestImplicitCoreProperties, BlockDirectoryTest, TestFreeTextSuggestions, SortByFunctionTest, TestDistribDocBasedVersion, TestPostingsSolrHighlighter, DocValuesMissingTest, QueryParsingTest, TestNonNRTOpen, RequestHandlersTest, TermVectorComponentDistributedTest, ModifyConfFileTest, TestCSVResponseWriter, TestCollapseQParserPlugin, CachingDirectoryFactoryTest, QueryFacetTest, RollingRestartTest, DeleteShardTest, HdfsUnloadDistributedZkTest, TestSimpleQParserPlugin, TestExpandComponent, HdfsChaosMonkeySafeLeaderTest, UpdateRequestProcessorFactoryTest, DistributedQueryComponentCustomSortTest, HdfsRecoveryZkTest, DistributedQueryComponentOptimizationTest, TestBinaryField, AlternateDirectoryTest, ScriptEngineTest, TestQueryUtils, FieldFacetExtrasTest, SampleTest, TestModifyConfFiles, TestGroupingSearch, TestPseudoReturnFields, TestSerializedLuceneMatchVersion, TestUniqueKeyFieldResource, TestRandomFaceting, EnumFieldTest, MultiThreadedOCPTest, AliasIntegrationTest, TestSolrQueryParserResource, TestRecoveryHdfs, TestReversedWildcardFilterFactory, SolrIndexConfigTest, AutoCommitTest, AnalysisAfterCoreReloadTest, TermVectorComponentTest, TestLFUCache, RangeFacetTest, CSVRequestHandlerTest, TestDistributedGrouping, TestSolrQueryParserDefaultOperatorResource, NumericFieldsTest, DebugComponentTest, TestCloudManagedSchema, TestSchemaNameResource, TestSolrXmlPersistor, StressHdfsTest, PreAnalyzedFieldTest, TestFunctionQuery, DateMathParserTest, LegacyHTMLStripCharFilterTest, TestQueryTypes, TestRestManager, TestSweetSpotSimilarityFactory, TestIndexSearcher, ResourceLoaderTest, TestCoreContainer, ConvertedLegacyTest, TestIndexingPerformance, TestValueSourceCache, TestBadConfig, TestPhraseSuggestions, TestXIncludeConfig, TestMultiCoreConfBootstrap, HardAutoCommitTest, SoftAutoCommitTest, TestNRTOpen, LukeRequestHandlerTest, SpatialFilterTest, CollectionsAPIAsyncDistributedZkTest, TestFieldSortValues, FileBasedSpellCheckerTest, IndexSchemaRuntimeFieldTest, SynonymTokenizerTest, TestJoin, TestZkChroot, TestOmitPositions, UUIDFieldTest, NoFacetTest, DistributedSuggestComponentTest, TestCSVLoader, ClusterStateTest, PrimUtilsTest, UnloadDistributedZkTest, SyncSliceTest, OverseerTest] [junit4] Completed on J1 in 81.21s, 9 tests, 1 error <<< FAILURES! [...truncated 642 lines...] BUILD FAILED /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:474: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:454: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 421 suites, 1761 tests, 1 error, 39 ignored (18 assumptions) Total time: 104 minutes 26 seconds Build step 'Invoke Ant' marked build as failure [description-setter] Description set: Java: 32bit/jdk1.7.0_65 -client -XX:+UseG1GC Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org