Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2299/
2 tests failed. REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch Error Message: Test abandoned because suite timeout was reached. Stack Trace: java.lang.Exception: Test abandoned because suite timeout was reached. at __randomizedtesting.SeedInfo.seed([AB1A6338BDB4224D]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest Error Message: Suite timeout exceeded (>= 7200000 msec). Stack Trace: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). at __randomizedtesting.SeedInfo.seed([AB1A6338BDB4224D]:0) Build Log: [...truncated 10336 lines...] [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest [junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/init-core-data-001 [junit4] 2> 833519 T1096 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false) [junit4] 2> 833520 T1096 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: / [junit4] 2> 833528 T1096 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch [junit4] 2> 833529 T1096 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 833530 T1097 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 833630 T1096 oasc.ZkTestServer.run start zk server on port:38373 [junit4] 2> 833631 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 833632 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 833637 T1104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6db61fa5 name:ZooKeeperConnection Watcher:127.0.0.1:38373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 833638 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 833638 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 833639 T1096 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 833642 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 833643 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 833645 T1107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@768d21c3 name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 833645 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 833646 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 833646 T1096 oascc.SolrZkClient.makePath makePath: /collections/collection1 [junit4] 2> 833649 T1096 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards [junit4] 2> 833650 T1096 oascc.SolrZkClient.makePath makePath: /collections/control_collection [junit4] 2> 833652 T1096 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards [junit4] 2> 833654 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 833654 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml [junit4] 2> 833657 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml [junit4] 2> 833658 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml [junit4] 2> 833761 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 833762 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 833765 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 833765 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt [junit4] 2> 833767 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 833768 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt [junit4] 2> 833770 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 833771 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml [junit4] 2> 833773 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 833774 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml [junit4] 2> 833776 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json [junit4] 2> 833777 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json [junit4] 2> 833780 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 833780 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt [junit4] 2> 833782 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 833783 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt [junit4] 2> 833786 T1096 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 833787 T1096 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt [junit4] 2> 833790 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 833791 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 833792 T1110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45832fc name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 833793 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 833793 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 835085 T1096 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 835089 T1096 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 835093 T1096 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38379 [junit4] 2> 835096 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 835096 T1096 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 835097 T1096 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002 [junit4] 2> 835097 T1096 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/' [junit4] 2> 835135 T1096 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/solr.xml [junit4] 2> 835201 T1096 oasc.CoreContainer.<init> New CoreContainer 578260863 [junit4] 2> 835201 T1096 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/] [junit4] 2> 835202 T1096 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 835203 T1096 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 835203 T1096 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 835203 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 835203 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 835204 T1096 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 835204 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 835204 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 835205 T1096 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 835205 T1096 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 835207 T1096 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 835209 T1096 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 835209 T1096 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 835210 T1096 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 835210 T1096 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38373/solr [junit4] 2> 835210 T1096 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 835210 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 835211 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 835213 T1122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bef3aa0 name:ZooKeeperConnection Watcher:127.0.0.1:38373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 835214 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 835214 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 835216 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 835217 T1125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22b0cedf name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 835217 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 835218 T1096 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 835220 T1096 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 835222 T1096 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 835223 T1096 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 835225 T1096 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 835227 T1096 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 835228 T1096 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38379_ [junit4] 2> 835228 T1096 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38379_ [junit4] 2> 835230 T1096 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 835231 T1096 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 835232 T1096 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 835233 T1096 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38379_ [junit4] 2> 835233 T1096 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 835235 T1096 oasc.Overseer.start Overseer (id=92935654231703556-127.0.0.1:38379_-n_0000000000) starting [junit4] 2> 835236 T1096 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 835240 T1096 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000 [junit4] 2> 835241 T1127 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 835241 T1096 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 835243 T1096 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 835244 T1096 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 835246 T1126 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 835251 T1129 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection [junit4] 2> 835252 T1129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 835252 T1129 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 835253 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 835254 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"1", [junit4] 2> "collection":"control_collection", [junit4] 2> "state":"down", [junit4] 2> "node_name":"127.0.0.1:38379_", [junit4] 2> "shard":null, [junit4] 2> "base_url":"https://127.0.0.1:38379"} [junit4] 2> 835255 T1126 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1] [junit4] 2> 835255 T1126 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1 [junit4] 2> 835255 T1126 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 835257 T1125 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 836253 T1129 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 836254 T1129 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection [junit4] 2> 836255 T1129 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 836255 T1129 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection [junit4] 2> 836256 T1129 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 836256 T1129 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/collection1/' [junit4] 2> 836258 T1129 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/collection1/lib/classes/' to classloader [junit4] 2> 836259 T1129 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/collection1/lib/README' to classloader [junit4] 2> 836260 T1129 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/collection1/lib/.svn/' to classloader [junit4] 2> 836297 T1129 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 836300 T1129 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 836338 T1129 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0 [junit4] 2> 836387 T1129 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 836489 T1129 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 836516 T1129 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 836880 T1129 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 836883 T1129 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 836885 T1129 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 836890 T1129 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 836920 T1129 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection [junit4] 2> 836921 T1129 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 837000 T1129 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/ [junit4] 2> 837001 T1129 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@36f1aaba [junit4] 2> 837004 T1129 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data [junit4] 2> 837005 T1129 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/index/ [junit4] 2> 837006 T1129 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/index' doesn't exist. Creating new index... [junit4] 2> 837007 T1129 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/index [junit4] 2> 837008 T1129 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=94.775390625, floorSegmentMB=0.408203125, forceMergeDeletesPctAllowed=11.827659262500351, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0 [junit4] 2> 837012 T1129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1} [junit4] 2> 837012 T1129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 837018 T1129 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 837019 T1129 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 837019 T1129 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 837019 T1129 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 837020 T1129 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 837020 T1129 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 837021 T1129 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 837021 T1129 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 837021 T1129 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 837022 T1129 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 837023 T1129 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 837023 T1129 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 837024 T1129 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 837024 T1129 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler [junit4] 2> 837025 T1129 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler [junit4] 2> 837026 T1129 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler [junit4] 2> 837026 T1129 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler [junit4] 2> 837027 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler [junit4] 2> 837028 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler [junit4] 2> 837029 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler [junit4] 2> 837029 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler [junit4] 2> 837030 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler [junit4] 2> 837030 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler [junit4] 2> 837031 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler [junit4] 2> 837032 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler [junit4] 2> 837032 T1129 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4] 2> 837033 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler [junit4] 2> 837033 T1129 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler [junit4] 2> 837053 T1129 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 837055 T1129 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 837058 T1129 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 837060 T1129 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 837063 T1129 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 837064 T1129 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 837065 T1129 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=75.15234375, floorSegmentMB=0.9521484375, forceMergeDeletesPctAllowed=9.248015382300197, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4901446280233296 [junit4] 2> 837066 T1129 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1} [junit4] 2> 837067 T1129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 837069 T1129 oass.SolrIndexSearcher.<init> Opening Searcher@4e8b6c1e[collection1] main [junit4] 2> 837069 T1129 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection [junit4] 2> 837070 T1129 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper [junit4] 2> 837070 T1129 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 837071 T1129 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 837071 T1129 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 837071 T1129 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 837072 T1129 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json [junit4] 2> 837072 T1129 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 837073 T1129 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json [junit4] 2> 837075 T1129 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json [junit4] 2> 837075 T1129 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 837075 T1129 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 837076 T1129 oash.ReplicationHandler.inform Commits will be reserved for 10000 [junit4] 2> 837077 T1130 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e8b6c1e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 837078 T1129 oasc.CoreContainer.registerCore registering core: collection1 [junit4] 2> 837078 T1133 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38379 collection:control_collection shard:shard1 [junit4] 2> 837079 T1096 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1 [junit4] 2> 837079 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 837080 T1133 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election [junit4] 2> 837085 T1133 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1 [junit4] 2> 837085 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 837086 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 837087 T1133 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 837086 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 837087 T1133 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> ASYNC NEW_CORE C152 name=collection1 org.apache.solr.core.SolrCore@58b31c75 url=https://127.0.0.1:38379/collection1 node=127.0.0.1:38379_ C152_STATE=coll:control_collection core:collection1 props:{core=collection1, state=down, node_name=127.0.0.1:38379_, base_url=https://127.0.0.1:38379} [junit4] 2> 837088 T1133 C152 P38379 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38379/collection1/ [junit4] 2> 837088 T1136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30316703 name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 837088 T1133 C152 P38379 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 837089 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 837089 T1133 C152 P38379 oasc.SyncStrategy.syncToMe https://127.0.0.1:38379/collection1/ has no replicas [junit4] 2> 837090 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 837090 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection"} [junit4] 2> 837090 T1133 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38379/collection1/ shard1 [junit4] 2> 837091 T1133 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1 [junit4] 2> 837091 T1096 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 837094 T1125 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837095 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0) [junit4] 2> 837096 T1096 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false [junit4] 2> 837097 T1096 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2 [junit4] 2> 837097 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 837098 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 837100 T1139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e964ea3 name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 837100 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 837101 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 837103 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 837106 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"create", [junit4] 2> "name":"collection1", [junit4] 2> "numShards":"3", [junit4] 2> "stateFormat":2} [junit4] 2> 837107 T1126 oasc.Overseer$ClusterStateUpdater.buildCollection Building a new collection: collection1 [junit4] 2> 837107 T1126 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3] [junit4] 2> 837107 T1126 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2 [junit4] 2> 837110 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json [junit4] 2> 837111 T1125 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837112 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837115 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"control_collection", [junit4] 2> "base_url":"https://127.0.0.1:38379", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} [junit4] 2> 837117 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 837118 T1125 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837118 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837158 T1133 oasc.ZkController.register We are https://127.0.0.1:38379/collection1/ and leader is https://127.0.0.1:38379/collection1/ [junit4] 2> 837159 T1133 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38379 [junit4] 2> 837159 T1133 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 837159 T1133 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection [junit4] 2> 837159 T1133 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 837161 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 837164 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"3", [junit4] 2> "collection":"control_collection", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1:38379_", [junit4] 2> "shard":"shard1", [junit4] 2> "base_url":"https://127.0.0.1:38379"} [junit4] 2> 837267 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 837267 T1125 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 838359 T1096 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 [junit4] 2> 838360 T1096 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 838365 T1096 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 838367 T1096 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38401 [junit4] 2> 838369 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 838369 T1096 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 838370 T1096 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003 [junit4] 2> 838370 T1096 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/' [junit4] 2> 838402 T1096 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/solr.xml [junit4] 2> 838458 T1096 oasc.CoreContainer.<init> New CoreContainer 1200631133 [junit4] 2> 838459 T1096 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/] [junit4] 2> 838460 T1096 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 838460 T1096 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 838460 T1096 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 838461 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 838461 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 838461 T1096 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 838461 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 838462 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 838462 T1096 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 838462 T1096 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 838465 T1096 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 838466 T1096 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 838467 T1096 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 838467 T1096 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 838467 T1096 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38373/solr [junit4] 2> 838468 T1096 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 838468 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 838469 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 838471 T1151 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a4f293e name:ZooKeeperConnection Watcher:127.0.0.1:38373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 838472 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 838472 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 838474 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 838476 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@125682fb name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 838476 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 838480 T1096 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 839484 T1096 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38401_ [junit4] 2> 839485 T1096 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38401_ [junit4] 2> 839488 T1096 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 839497 T1155 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1 [junit4] 2> 839497 T1155 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 839498 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 839499 T1155 oasc.ZkController.preRegister Registering watch for external collection collection1 [junit4] 2> 839499 T1155 oascc.ZkStateReader.addZkWatch addZkWatch collection1 [junit4] 2> 839499 T1155 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... [junit4] 2> 839500 T1155 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 [junit4] 2> 839501 T1155 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 839503 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"3", [junit4] 2> "collection":"collection1", [junit4] 2> "state":"down", [junit4] 2> "node_name":"127.0.0.1:38401_", [junit4] 2> "shard":null, [junit4] 2> "base_url":"https://127.0.0.1:38401"} [junit4] 2> 839506 T1126 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 839506 T1126 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3 [junit4] 2> 839510 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 839510 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 839511 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 839511 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 [junit4] 2> 840501 T1155 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 840502 T1155 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1 [junit4] 2> 840503 T1155 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 840503 T1155 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 840504 T1155 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 840504 T1155 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/collection1/' [junit4] 2> 840506 T1155 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/collection1/lib/README' to classloader [junit4] 2> 840507 T1155 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/collection1/lib/.svn/' to classloader [junit4] 2> 840508 T1155 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/collection1/lib/classes/' to classloader [junit4] 2> 840541 T1155 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 840544 T1155 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 840579 T1155 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0 [junit4] 2> 840625 T1155 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 840727 T1155 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 840738 T1155 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 841077 T1155 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 841080 T1155 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 841082 T1155 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 841088 T1155 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 841114 T1155 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 841114 T1155 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 841115 T1155 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/ [junit4] 2> 841115 T1155 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@36f1aaba [junit4] 2> 841117 T1155 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1 [junit4] 2> 841118 T1155 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/index/ [junit4] 2> 841119 T1155 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/index' doesn't exist. Creating new index... [junit4] 2> 841120 T1155 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/index [junit4] 2> 841121 T1155 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=94.775390625, floorSegmentMB=0.408203125, forceMergeDeletesPctAllowed=11.827659262500351, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0 [junit4] 2> 841145 T1155 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1} [junit4] 2> 841146 T1155 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 841153 T1155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 841154 T1155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 841154 T1155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 841154 T1155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 841155 T1155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 841155 T1155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 841156 T1155 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 841156 T1155 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 841156 T1155 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 841157 T1155 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 841158 T1155 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 841159 T1155 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 841159 T1155 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 841160 T1155 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler [junit4] 2> 841161 T1155 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler [junit4] 2> 841163 T1155 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler [junit4] 2> 841164 T1155 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler [junit4] 2> 841165 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler [junit4] 2> 841166 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler [junit4] 2> 841167 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler [junit4] 2> 841168 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler [junit4] 2> 841168 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler [junit4] 2> 841169 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler [junit4] 2> 841170 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler [junit4] 2> 841170 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler [junit4] 2> 841171 T1155 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4] 2> 841172 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler [junit4] 2> 841172 T1155 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler [junit4] 2> 841191 T1155 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 841193 T1155 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 841195 T1155 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 841197 T1155 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 841201 T1155 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 841201 T1155 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 841202 T1155 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=75.15234375, floorSegmentMB=0.9521484375, forceMergeDeletesPctAllowed=9.248015382300197, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4901446280233296 [junit4] 2> 841204 T1155 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1} [junit4] 2> 841204 T1155 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 841205 T1155 oass.SolrIndexSearcher.<init> Opening Searcher@220a6e21[collection1] main [junit4] 2> 841205 T1155 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 841206 T1155 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 841206 T1155 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 841207 T1155 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 841207 T1155 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 841207 T1155 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 841208 T1155 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json [junit4] 2> 841209 T1155 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 841209 T1155 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed [junit4] 2> 841209 T1155 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 841209 T1155 oash.ReplicationHandler.inform Commits will be reserved for 10000 [junit4] 2> 841211 T1156 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@220a6e21[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 841211 T1155 oasc.CoreContainer.registerCore registering core: collection1 [junit4] 2> 841212 T1159 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38401 collection:collection1 shard:shard3 [junit4] 2> 841212 T1096 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1 [junit4] 2> 841213 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 841213 T1159 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election [junit4] 2> 841218 T1159 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3 [junit4] 2> 841219 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 841219 T1159 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 841220 T1159 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> ASYNC NEW_CORE C153 name=collection1 org.apache.solr.core.SolrCore@7b139a31 url=https://127.0.0.1:38401/collection1 node=127.0.0.1:38401_ C153_STATE=coll:collection1 core:collection1 props:{core=collection1, state=down, node_name=127.0.0.1:38401_, base_url=https://127.0.0.1:38401} [junit4] 2> 841220 T1159 C153 P38401 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38401/collection1/ [junit4] 2> 841220 T1159 C153 P38401 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 841221 T1159 C153 P38401 oasc.SyncStrategy.syncToMe https://127.0.0.1:38401/collection1/ has no replicas [junit4] 2> 841221 T1159 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38401/collection1/ shard3 [junit4] 2> 841221 T1159 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3 [junit4] 2> 841222 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard3", [junit4] 2> "collection":"collection1"} [junit4] 2> 841224 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 841225 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 841226 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 [junit4] 2> 841228 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard3", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"https://127.0.0.1:38401", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} [junit4] 2> 841231 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 841231 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 841232 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 841232 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 [junit4] 2> 841276 T1159 oasc.ZkController.register We are https://127.0.0.1:38401/collection1/ and leader is https://127.0.0.1:38401/collection1/ [junit4] 2> 841277 T1159 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38401 [junit4] 2> 841277 T1159 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 841277 T1159 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1 [junit4] 2> 841277 T1159 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 841279 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 841281 T1159 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 [junit4] 2> 841281 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"3", [junit4] 2> "collection":"collection1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1:38401_", [junit4] 2> "shard":"shard3", [junit4] 2> "base_url":"https://127.0.0.1:38401"} [junit4] 2> 841285 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 841286 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 841287 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 [junit4] 2> 842473 T1096 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 [junit4] 2> 842473 T1096 oejs.Server.doStart jetty-8.1.10.v20130312 [junit4] 2> 842477 T1096 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] [junit4] 2> 842479 T1096 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38408 [junit4] 2> 842482 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() [junit4] 2> 842482 T1096 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx) [junit4] 2> 842482 T1096 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004 [junit4] 2> 842483 T1096 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/' [junit4] 2> 842512 T1096 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/solr.xml [junit4] 2> 842565 T1096 oasc.CoreContainer.<init> New CoreContainer 751140324 [junit4] 2> 842566 T1096 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/] [junit4] 2> 842567 T1096 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000 [junit4] 2> 842567 T1096 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: [junit4] 2> 842567 T1096 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000 [junit4] 2> 842568 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20 [junit4] 2> 842568 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000 [junit4] 2> 842568 T1096 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0 [junit4] 2> 842568 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647 [junit4] 2> 842569 T1096 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5 [junit4] 2> 842569 T1096 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1 [junit4] 2> 842569 T1096 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false [junit4] 2> 842571 T1096 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false [junit4] 2> 842573 T1096 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory [junit4] 2> 842573 T1096 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] [junit4] 2> 842574 T1096 oasc.CoreContainer.load Host Name: 127.0.0.1 [junit4] 2> 842574 T1096 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38373/solr [junit4] 2> 842574 T1096 oasc.ZkController.checkChrootPath zkHost includes chroot [junit4] 2> 842575 T1096 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider [junit4] 2> 842575 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 842577 T1171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f389671 name:ZooKeeperConnection Watcher:127.0.0.1:38373 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 842578 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 842578 T1096 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider [junit4] 2> 842580 T1096 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 842581 T1174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25fff5ee name:ZooKeeperConnection Watcher:127.0.0.1:38373/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 842581 T1096 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 842585 T1096 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 843588 T1096 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38408_ [junit4] 2> 843589 T1096 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38408_ [junit4] 2> 843593 T1096 oasc.Overseer.close Overseer (id=null) closing [junit4] 2> 843602 T1175 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1 [junit4] 2> 843602 T1175 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 843603 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 843604 T1175 oasc.ZkController.preRegister Registering watch for external collection collection1 [junit4] 2> 843604 T1175 oascc.ZkStateReader.addZkWatch addZkWatch collection1 [junit4] 2> 843605 T1175 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... [junit4] 2> 843606 T1175 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 [junit4] 2> 843606 T1175 oasc.ZkController.waitForCoreNodeName look for our core node name [junit4] 2> 843606 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"3", [junit4] 2> "collection":"collection1", [junit4] 2> "state":"down", [junit4] 2> "node_name":"127.0.0.1:38408_", [junit4] 2> "shard":null, [junit4] 2> "base_url":"https://127.0.0.1:38408"} [junit4] 2> 843609 T1126 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 843610 T1126 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 843614 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 843614 T1174 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 843614 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 843615 T1174 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 [junit4] 2> 843615 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 [junit4] 2> 844606 T1175 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1 [junit4] 2> 844607 T1175 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1 [junit4] 2> 844608 T1175 oasc.ZkController.createCollectionZkNode Collection zkNode exists [junit4] 2> 844608 T1175 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 844609 T1175 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 844609 T1175 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/collection1/' [junit4] 2> 844611 T1175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/collection1/lib/README' to classloader [junit4] 2> 844612 T1175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/collection1/lib/.svn/' to classloader [junit4] 2> 844612 T1175 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/collection1/lib/classes/' to classloader [junit4] 2> 844649 T1175 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1 [junit4] 2> 844652 T1175 oasc.Config.<init> loaded config solrconfig.xml with version 0 [junit4] 2> 844690 T1175 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0 [junit4] 2> 844738 T1175 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml [junit4] 2> 844841 T1175 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml [junit4] 2> 844852 T1175 oass.IndexSchema.readSchema [collection1] Schema name=test [junit4] 2> 845199 T1175 oass.IndexSchema.readSchema default search field in schema is text [junit4] 2> 845202 T1175 oass.IndexSchema.readSchema unique key field: id [junit4] 2> 845204 T1175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 845209 T1175 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml [junit4] 2> 845236 T1175 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1 [junit4] 2> 845236 T1175 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory [junit4] 2> 845237 T1175 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/ [junit4] 2> 845237 T1175 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@36f1aaba [junit4] 2> 845239 T1175 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2 [junit4] 2> 845240 T1175 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/index/ [junit4] 2> 845241 T1175 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/index' doesn't exist. Creating new index... [junit4] 2> 845242 T1175 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/index [junit4] 2> 845243 T1175 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=94.775390625, floorSegmentMB=0.408203125, forceMergeDeletesPctAllowed=11.827659262500351, segmentsPerTier=26.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0 [junit4] 2> 845248 T1175 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1} [junit4] 2> 845248 T1175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 845256 T1175 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib" [junit4] 2> 845256 T1175 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe" [junit4] 2> 845257 T1175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe" [junit4] 2> 845257 T1175 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig" [junit4] 2> 845257 T1175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig" [junit4] 2> 845258 T1175 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit" [junit4] 2> 845258 T1175 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 845259 T1175 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit" [junit4] 2> 845259 T1175 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default [junit4] 2> 845260 T1175 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 845261 T1175 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 845261 T1175 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 845262 T1175 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler [junit4] 2> 845263 T1175 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler [junit4] 2> 845264 T1175 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler [junit4] 2> 845266 T1175 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler [junit4] 2> 845266 T1175 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler [junit4] 2> 845267 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler [junit4] 2> 845268 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler [junit4] 2> 845269 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler [junit4] 2> 845270 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler [junit4] 2> 845271 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler [junit4] 2> 845271 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler [junit4] 2> 845272 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler [junit4] 2> 845273 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler [junit4] 2> 845274 T1175 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler [junit4] 2> 845274 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler [junit4] 2> 845275 T1175 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler [junit4] 2> 845294 T1175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 845296 T1175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 845298 T1175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 845300 T1175 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60 [junit4] 2> 845304 T1175 oasu.CommitTracker.<init> Hard AutoCommit: disabled [junit4] 2> 845304 T1175 oasu.CommitTracker.<init> Soft AutoCommit: disabled [junit4] 2> 845306 T1175 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=28, maxMergedSegmentMB=75.15234375, floorSegmentMB=0.9521484375, forceMergeDeletesPctAllowed=9.248015382300197, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4901446280233296 [junit4] 2> 845307 T1175 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1 [junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AB1A6338BDB4224D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1} [junit4] 2> 845308 T1175 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1 [junit4] 2> 845308 T1175 oass.SolrIndexSearcher.<init> Opening Searcher@a13606c[collection1] main [junit4] 2> 845309 T1175 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1 [junit4] 2> 845310 T1175 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper [junit4] 2> 845310 T1175 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1 [junit4] 2> 845310 T1175 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 845311 T1175 oasr.RestManager.init Initializing RestManager with initArgs: {} [junit4] 2> 845311 T1175 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 845312 T1175 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json [junit4] 2> 845312 T1175 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1 [junit4] 2> 845313 T1175 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed [junit4] 2> 845313 T1175 oasr.RestManager.init Initializing 0 registered ManagedResources [junit4] 2> 845313 T1175 oash.ReplicationHandler.inform Commits will be reserved for 10000 [junit4] 2> 845315 T1176 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a13606c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 845315 T1175 oasc.CoreContainer.registerCore registering core: collection1 [junit4] 2> 845316 T1179 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38408 collection:collection1 shard:shard1 [junit4] 2> 845316 T1096 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build/solr-core/test/J1 [junit4] 2> 845317 T1096 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done [junit4] 2> 845317 T1179 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 845322 T1179 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1 [junit4] 2> 845324 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 845324 T1179 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue. [junit4] 2> 845324 T1179 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync [junit4] 2> ASYNC NEW_CORE C154 name=collection1 org.apache.solr.core.SolrCore@7a485b1e url=https://127.0.0.1:38408/collection1 node=127.0.0.1:38408_ C154_STATE=coll:collection1 core:collection1 props:{core=collection1, state=down, node_name=127.0.0.1:38408_, base_url=https://127.0.0.1:38408} [junit4] 2> 845324 T1179 C154 P38408 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38408/collection1/ [junit4] 2> 845325 T1179 C154 P38408 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me [junit4] 2> 845325 T1179 C154 P38408 oasc.SyncStrategy.syncToMe https://127.0.0.1:38408/collection1/ has no replicas [junit4] 2> 845325 T1179 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38408/collection1/ shard1 [junit4] 2> 845326 T1179 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 845327 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1"} [junit4] 2> 845330 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 845331 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 845331 T1174 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 845331 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 [junit4] 2> 845332 T1174 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 [junit4] 2> 845335 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "operation":"leader", [junit4] 2> "shard":"shard1", [junit4] 2> "collection":"collection1", [junit4] 2> "base_url":"https://127.0.0.1:38408", [junit4] 2> "core":"collection1", [junit4] 2> "state":"active"} [junit4] 2> 845337 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 845337 T1126 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json [junit4] 2> 845338 T1174 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 845338 T1154 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... [junit4] 2> 845339 T1174 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 [junit4] 2> 845339 T1154 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 [junit4] 2> 845380 T1179 oasc.ZkController.register We are https://127.0.0.1:38408/collection1/ and leader is https://127.0.0.1:38408/collection1/ [junit4] 2> 845381 T1179 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38408 [junit4] 2> 845381 T1179 oasc.ZkController.checkRecovery I am the leader, no recovery necessary [junit4] 2> 845381 T1179 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1 [junit4] 2> 845381 T1179 oasc.ZkController.publish numShards not found on descriptor - reading it from system property [junit4] 2> 845383 T1125 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected [junit4] 2> 845385 T1179 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 [junit4] 2> 845386 T1126 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = { [junit4] 2> "roles":null, [junit4] 2> "core":"collection1", [junit4] 2> "operation":"state", [junit4] 2> "numShards":"3", [junit4] 2> "collection":"collection1", [junit4] 2> "core_node_nam [...truncated too long message...] testing.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) [junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) [junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) [junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 11) Thread[id=1200, name=HashSessionScavenger-19, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552) [junit4] 2> at java.util.TimerThread.run(Timer.java:505) [junit4] 2> 12) Thread[id=1180, name=HashSessionScavenger-18, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552) [junit4] 2> at java.util.TimerThread.run(Timer.java:505) [junit4] 2> 13) Thread[id=1196, name=searcherExecutor-352-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at sun.misc.Unsafe.park(Native Method) [junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) [junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) [junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 14) Thread[id=1231, name=commitScheduler-355-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at sun.misc.Unsafe.park(Native Method) [junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) [junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) [junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> 15) Thread[id=1217, name=Thread-377, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Object.wait(Object.java:503) [junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:924) [junit4] 2> 16) Thread[id=1192, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[AB1A6338BDB4224D]-SendThread(127.0.0.1:38373), state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:82) [junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) [junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) [junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [junit4] 2> 17) Thread[id=1197, name=Thread-366, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] [junit4] 2> at java.lang.Object.wait(Native Method) [junit4] 2> at java.lang.Object.wait(Object.java:503) [junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:924) [junit4] 2> NOTE: test params are: codec=Asserting(Lucene50), sim=DefaultSimilarity, locale=no_NO_NY, timezone=America/Guadeloupe [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=19,free=77339776,total=187170816 [junit4] 2> NOTE: All tests run in this JVM: [HttpPartitionTest, HdfsBasicDistributedZk2Test, MigrateRouteKeyTest, SSLMigrationTest, HdfsCollectionsAPIDistributedZkTest, OverseerCollectionProcessorTest, SolrPluginUtilsTest, TestFieldSortValues, TestXIncludeConfig, TestFunctionQuery, TestSchemaNameResource, SimpleCollectionCreateDeleteTest, TestPivotHelperCode, DocumentAnalysisRequestHandlerTest, DistanceFunctionTest, CollectionsAPIAsyncDistributedZkTest, TestExceedMaxTermLength, PreAnalyzedUpdateProcessorTest, TestRestManager, ChaosMonkeySafeLeaderTest] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=AB1A6338BDB4224D -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=no_NO_NY -Dtests.timezone=America/Guadeloupe -Dtests.asserts=true -Dtests.file.encoding=UTF-8 [junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<< [junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec). [junit4] > at __randomizedtesting.SeedInfo.seed([AB1A6338BDB4224D]:0) [junit4] Completed on J1 in 7223.68s, 1 test, 2 errors <<< FAILURES! [...truncated 135 lines...] BUILD FAILED /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:529: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:477: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/build.xml:61: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/extra-targets.xml:39: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/build.xml:190: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/solr/common-build.xml:508: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:1363: The following error occurred while executing this line: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-5.x-Java7/lucene/common-build.xml:970: There were test failures: 450 suites, 1844 tests, 1 suite-level error, 1 error, 211 ignored (20 assumptions) Total time: 154 minutes 1 second Build step 'Invoke Ant' marked build as failure Archiving artifacts Sending artifact delta relative to Lucene-Solr-Tests-5.x-Java7 #2298 Archived 5 artifacts Archive block size is 32768 Received 0 blocks and 215494173 bytes Compression is 0.0% Took 42 sec Recording test results Email was triggered for: Failure Sending email for trigger: Failure
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org