I am working on it, I have identified at least one problem which is that the overseer is killed too often in that test, i'll let the test run locally for a bit and if everything looks good commit a fix tomorrow.
-- Sami Siren On Tue, Jun 12, 2012 at 6:32 PM, Mark Miller <[email protected]> wrote: > While working on the collections api, I have seen this on the odd occasion > locally as well. > > On Jun 12, 2012, at 10:43 AM, [email protected] wrote: > >> Build: >> http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Windows-Java7-64/300/ >> >> 1 tests failed. >> FAILED: org.apache.solr.cloud.OverseerTest.testShardLeaderChange >> >> Error Message: >> Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but >> was:<core[1]> >> >> Stack Trace: >> org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 >> shard:shard1 expected:<core[4]> but was:<core[1]> >> at >> __randomizedtesting.SeedInfo.seed([195A5E746C7F55C0:C709D98376E7A031]:0) >> at org.junit.Assert.assertEquals(Assert.java:125) >> at >> org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:522) >> at >> org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:677) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:601) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1969) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:814) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:875) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:889) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) >> at >> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) >> at >> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) >> at >> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:821) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:669) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:695) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:734) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:745) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) >> at >> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) >> at >> org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) >> at >> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) >> at >> org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51) >> at >> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) >> at >> org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53) >> at >> org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52) >> at >> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36) >> at >> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) >> at >> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:56) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132) >> at >> com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551) >> >> >> >> >> Build Log: >> [...truncated 11002 lines...] >> [junit4] 2> at >> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:289) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:286) >> [junit4] 2> at >> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:286) >> [junit4] 2> at >> org.apache.solr.cloud.Overseer$CloudStateUpdater.amILeader(Overseer.java:186) >> [junit4] 2> at >> org.apache.solr.cloud.Overseer$CloudStateUpdater.run(Overseer.java:111) >> [junit4] 2> at java.lang.Thread.run(Thread.java:722) >> [junit4] 2> >> [junit4] 2> 4792 T1150 oasc.Overseer$CloudStateUpdater.amILeader >> According to ZK I (id=87786223260336139-127.0.0.1:57511_solr-n_0000000008) >> am no longer a leader. >> [junit4] 2> 4937 T960 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server 127.0.0.1/127.0.0.1:57322 >> [junit4] 2> 4963 T1153 oasc.Overseer$CloudStateUpdater.run WARNING >> Overseer cannot talk to ZK >> [junit4] 2> 4964 T1120 oazs.PrepRequestProcessor.pRequest Processed >> session termination for sessionid: 0x137e11add9b000c >> [junit4] 2> 4981 T1117 oazs.NIOServerCnxn.closeSock Closed socket >> connection for client /127.0.0.1:57551 which had sessionid 0x137e11add9b000c >> [junit4] 2> 4982 T1127 oaz.ZooKeeper.close Session: 0x137e11add9b000c >> closed >> [junit4] 2> 4982 T1127 oaz.ZooKeeper.<init> Initiating client >> connection, connectString=127.0.0.1:57511/solr sessionTimeout=10000 >> watcher=org.apache.solr.common.cloud.ConnectionManager@7d16b6ef >> [junit4] 2> 4982 T1152 oaz.ClientCnxn$EventThread.run EventThread shut >> down >> [junit4] 2> 4982 T1154 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server /127.0.0.1:57511 >> [junit4] 2> 4982 T1154 oaz.ClientCnxn$SendThread.primeConnection Socket >> connection established to 127.0.0.1/127.0.0.1:57511, initiating session >> [junit4] 2> 4982 T1117 oazs.NIOServerCnxn$Factory.run Accepted socket >> connection from /127.0.0.1:57555 >> [junit4] 2> 4982 T1117 oazs.NIOServerCnxn.readConnectRequest Client >> attempting to establish new session at /127.0.0.1:57555 >> [junit4] 2> 5007 T1119 oazs.NIOServerCnxn.finishSessionInit Established >> session 0x137e11add9b000d with negotiated timeout 10000 for client >> /127.0.0.1:57555 >> [junit4] 2> 5008 T1154 oaz.ClientCnxn$SendThread.readConnectResult >> Session establishment complete on server 127.0.0.1/127.0.0.1:57511, >> sessionid = 0x137e11add9b000d, negotiated timeout = 10000 >> [junit4] 2> 5008 T1155 oascc.ConnectionManager.process Watcher >> org.apache.solr.common.cloud.ConnectionManager@7d16b6ef >> name:ZooKeeperConnection Watcher:127.0.0.1:57511/solr got event WatchedEvent >> state:SyncConnected type:None path:null path:null type:None >> [junit4] 2> 5028 T1127 oascc.SolrZkClient.makePath makePath: >> /overseer_elect/leader >> [junit4] 2> 5047 T1127 oasc.Overseer.<init> Overseer >> (id=87786223260336141-127.0.0.1:57511_solr-n_0000000012) starting >> [junit4] 2> 5047 T1120 oazs.PrepRequestProcessor.pRequest Got user-level >> KeeperException when processing sessionid:0x137e11add9b000d type:create >> cxid:0x8 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error >> Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer >> [junit4] 2> 5060 T1120 oazs.PrepRequestProcessor.pRequest Got user-level >> KeeperException when processing sessionid:0x137e11add9b000d type:create >> cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error >> Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer >> [junit4] 2> 5069 T1120 oazs.PrepRequestProcessor.pRequest Got user-level >> KeeperException when processing sessionid:0x137e11add9b000d type:create >> cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error >> Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer >> [junit4] 2> 5091 T1156 oasc.Overseer$CloudStateUpdater.run Starting to >> work on the main queue >> [junit4] 2> 5788 T960 oaz.ClientCnxn$SendThread.run WARNING Session >> 0x137e118aa7c0010 for server null, unexpected error, closing socket >> connection and attempting reconnect java.net.ConnectException: Connection >> refused: no further information >> [junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >> Method) >> [junit4] 2> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> [junit4] 2> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146) >> [junit4] 2> >> [junit4] 2> 6956 T960 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server 127.0.0.1/127.0.0.1:57322 >> [junit4] 2> 7895 T960 oaz.ClientCnxn$SendThread.run WARNING Session >> 0x137e118aa7c0010 for server null, unexpected error, closing socket >> connection and attempting reconnect java.net.ConnectException: Connection >> refused: no further information >> [junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >> Method) >> [junit4] 2> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> [junit4] 2> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146) >> [junit4] 2> >> [junit4] 2> 9652 T960 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server 127.0.0.1/127.0.0.1:57322 >> [junit4] 2> 10640 T960 oaz.ClientCnxn$SendThread.run WARNING Session >> 0x137e118aa7c0010 for server null, unexpected error, closing socket >> connection and attempting reconnect java.net.ConnectException: Connection >> refused: no further information >> [junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >> Method) >> [junit4] 2> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> [junit4] 2> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146) >> [junit4] 2> >> [junit4] 2> 12479 T960 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server 127.0.0.1/127.0.0.1:57322 >> [junit4] 2> 13046 T1120 oazs.PrepRequestProcessor.pRequest Processed >> session termination for sessionid: 0x137e11add9b000d >> [junit4] 2> 13065 T1117 oazs.NIOServerCnxn.doIO WARNING >> EndOfStreamException: Unable to read additional data from client sessionid >> 0x137e11add9b000d, likely client has closed socket >> [junit4] 2> 13065 T1127 oaz.ZooKeeper.close Session: 0x137e11add9b000d >> closed >> [junit4] 2> 13065 T1155 oaz.ClientCnxn$EventThread.run EventThread shut >> down >> [junit4] 2> 13066 T1117 oazs.NIOServerCnxn.closeSock Closed socket >> connection for client /127.0.0.1:57555 which had sessionid 0x137e11add9b000d >> [junit4] 2> 13069 T1122 oascc.ZkStateReader$3.process Updating live nodes >> [junit4] 2> 13070 T1140 oascc.ZkStateReader$3.process Updating live nodes >> [junit4] 2> 13071 T1120 oazs.PrepRequestProcessor.pRequest Processed >> session termination for sessionid: 0x137e11add9b0008 >> [junit4] 2> 13073 T1117 oazs.NIOServerCnxn.closeSock Closed socket >> connection for client /127.0.0.1:57538 which had sessionid 0x137e11add9b0008 >> [junit4] 2> 13073 T1140 oaz.ClientCnxn$EventThread.run EventThread shut >> down >> [junit4] 2> 13073 T1115 oaz.ZooKeeper.close Session: 0x137e11add9b0008 >> closed >> [junit4] 2> 13074 T1120 oazs.PrepRequestProcessor.pRequest Processed >> session termination for sessionid: 0x137e11add9b0000 >> [junit4] 2> 13076 T1117 oazs.NIOServerCnxn.doIO WARNING >> EndOfStreamException: Unable to read additional data from client sessionid >> 0x137e11add9b0000, likely client has closed socket >> [junit4] 2> 13076 T1115 oaz.ZooKeeper.close Session: 0x137e11add9b0000 >> closed >> [junit4] 2> 13076 T1122 oaz.ClientCnxn$EventThread.run EventThread shut >> down >> [junit4] 2> 13076 T1117 oazs.NIOServerCnxn.closeSock Closed socket >> connection for client /127.0.0.1:57514 which had sessionid 0x137e11add9b0000 >> [junit4] 2> 13079 T1120 oazs.PrepRequestProcessor.run >> PrepRequestProcessor exited loop! >> [junit4] 2> 13079 T1119 oazs.SyncRequestProcessor.run >> SyncRequestProcessor exited! >> [junit4] 2> 13080 T1115 oazs.FinalRequestProcessor.shutdown shutdown of >> request processor complete >> [junit4] 2> 13171 T1156 oasc.Overseer$CloudStateUpdater.amILeader >> WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: >> KeeperErrorCode = Session expired for /overseer_elect/leader >> [junit4] 2> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:118) >> [junit4] 2> at >> org.apache.zookeeper.KeeperException.create(KeeperException.java:42) >> [junit4] 2> at >> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:289) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:286) >> [junit4] 2> at >> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) >> [junit4] 2> at >> org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:286) >> [junit4] 2> at >> org.apache.solr.cloud.Overseer$CloudStateUpdater.amILeader(Overseer.java:186) >> [junit4] 2> at >> org.apache.solr.cloud.Overseer$CloudStateUpdater.run(Overseer.java:111) >> [junit4] 2> at java.lang.Thread.run(Thread.java:722) >> [junit4] 2> >> [junit4] 2> 13171 T1156 oasc.Overseer$CloudStateUpdater.amILeader >> According to ZK I (id=87786223260336141-127.0.0.1:57511_solr-n_0000000012) >> am no longer a leader. >> [junit4] 2> 13422 T960 oaz.ClientCnxn$SendThread.run WARNING Session >> 0x137e118aa7c0010 for server null, unexpected error, closing socket >> connection and attempting reconnect java.net.ConnectException: Connection >> refused: no further information >> [junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native >> Method) >> [junit4] 2> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) >> [junit4] 2> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1146) >> [junit4] 2> >> [junit4] 2> 14960 T1118 oazs.SessionTrackerImpl.run SessionTrackerImpl >> exited loop! >> [junit4] 2> 15006 T960 oaz.ClientCnxn$SendThread.startConnect Opening >> socket connection to server 127.0.0.1/127.0.0.1:57322 >> [junit4] 2> 15559 T1117 oazs.NIOServerCnxn$Factory.run NIOServerCnxn >> factory exited run method >> [junit4] 2> 15561 T1115 oazs.FinalRequestProcessor.shutdown shutdown of >> request processor complete >> [junit4] 2> 15561 T1115 oas.SolrTestCaseJ4.tearDown ###Ending >> testShardLeaderChange >> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest >> -Dtests.method=testShardLeaderChange -Dtests.seed=195A5E746C7F55C0 >> -Dtests.locale=es_CR -Dtests.timezone=Asia/Colombo >> -Dargs="-Dfile.encoding=Cp1252" >> [junit4] 2> >> [junit4] > (@AfterClass output) >> [junit4] 2> 76626 T1115 oas.SolrTestCaseJ4.initCore ####initCore >> [junit4] 2> 76626 T1115 oas.SolrTestCaseJ4.initCore ####initCore end >> [junit4] 2> 76626 T1115 oas.SolrTestCaseJ4.deleteCore ###deleteCore >> [junit4] 2> NOTE: test params are: codec=Lucene40: {}, >> sim=DefaultSimilarity, locale=es_CR, timezone=Asia/Colombo >> [junit4] 2> NOTE: Windows 7 6.1 amd64/Oracle Corporation 1.7.0_04 >> (64-bit)/cpus=2,threads=1,free=113734152,total=259325952 >> [junit4] 2> NOTE: All tests run in this JVM: [UUIDFieldTest, >> LengthFilterTest, TestNGramFilters, IndexReaderFactoryTest, TestQueryUtils, >> OutputWriterTest, SolrRequestParserTest, >> DistributedQueryElevationComponentTest, MultiTermTest, >> TestDFRSimilarityFactory, CloudStateUpdateTest, TestCSVLoader, >> JSONWriterTest, SuggesterTSTTest, TestArbitraryIndexDir, TestRecovery, >> StatsComponentTest, TestSpanishLightStemFilterFactory, TestPropInject, >> LeaderElectionTest, TestGermanNormalizationFilterFactory, >> URLClassifyProcessorTest, TestPseudoReturnFields, TestSolrCoreProperties, >> PeerSyncTest, DisMaxRequestHandlerTest, TestPortugueseStemFilterFactory, >> TestDistributedSearch, TestGreekStemFilterFactory, >> TestGalicianStemFilterFactory, BadIndexSchemaTest, TestShingleFilterFactory, >> PrimitiveFieldTypeTest, SignatureUpdateProcessorFactoryTest, >> RequestHandlersTest, TestGalicianMinimalStemFilterFactory, >> DocumentAnalysisRequestHandlerTest, TestTrie, DirectSolrSpellCheckerTest, >> LeaderElectionIntegrationTest, TestMappingCharFilterFactory, >> DistanceFunctionTest, TestBrazilianStemFilterFactory, >> UniqFieldsUpdateProcessorFactoryTest, TestJapaneseBaseFormFilterFactory, >> TestFaceting, DistributedTermsComponentTest, TestJmxIntegration, >> TestPHPSerializedResponseWriter, FileBasedSpellCheckerTest, >> TestBulgarianStemFilterFactory, SnowballPorterFilterFactoryTest, >> TestGermanStemFilterFactory, FullSolrCloudTest, TimeZoneUtilsTest, >> LukeRequestHandlerTest, SystemInfoHandlerTest, CoreAdminHandlerTest, >> TestLMJelinekMercerSimilarityFactory, TestGroupingSearch, >> TestRemoteStreaming, SolrIndexConfigTest, TestItalianLightStemFilterFactory, >> CSVRequestHandlerTest, FieldMutatingUpdateProcessorTest, >> TestRussianLightStemFilterFactory, TestPluginEnable, TestJmxMonitoredMap, >> TestTurkishLowerCaseFilterFactory, TestStemmerOverrideFilterFactory, >> ConvertedLegacyTest, TestWriterPerf, TestGermanMinimalStemFilterFactory, >> TestExtendedDismaxParser, TestHyphenationCompoundWordTokenFilterFactory, >> TestFinnishLightStemFilterFactory, FieldAnalysisRequestHandlerTest, >> TestIndexingPerformance, FastVectorHighlighterTest, AlternateDirectoryTest, >> TestPersianNormalizationFilterFactory, BinaryUpdateRequestHandlerTest, >> CacheHeaderTest, TestCJKBigramFilterFactory, TestDocSet, TestNumberUtils, >> XmlUpdateRequestHandlerTest, TestRemoveDuplicatesTokenFilterFactory, >> TestSolrXMLSerializer, TestUpdate, TestBadConfig, MBeansHandlerTest, >> NoCacheHeaderTest, BasicZkTest, TestPatternReplaceCharFilterFactory, >> UpdateParamsTest, TestCapitalizationFilterFactory, >> TestDelimitedPayloadTokenFilterFactory, NotRequiredUniqueKeyTest, >> SuggesterTest, OverseerTest] >> [junit4] 2> >> [junit4] Completed in 76.66s, 8 tests, 1 failure <<< FAILURES! >> [...truncated 831 lines...] >> >> [...truncated 11934 lines...] >> >> [...truncated 11934 lines...] >> >> [...truncated 11934 lines...] >> >> [...truncated 11934 lines...] >> >> [...truncated 11934 lines...] >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > > - Mark Miller > lucidimagination.com > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
