[ https://issues.apache.org/jira/browse/SOLR-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324726#comment-14324726 ]
Shalin Shekhar Mangar commented on SOLR-7118: --------------------------------------------- Okay, here's what's happening at http://jenkins.thetaphi.de/job/Lucene-Solr-5.0-Linux/151/consoleText # collection has numShards=1 and replicationFactor=2 # collection1 has stateFormat=2 so CloudSolrClient will cache cluster state and update when the server returns a stale state error # ChaosMonkey kills the leader when state version = 7 # The leader publishes itself as down and unsets the leader flag. Overseer writes new state(s) and new state version = 9 # A Query request returns stale state error. CloudSolrClient evicts cached state and retries after refreshing+caching the state from ZK {code} [junit4] 2> 1646950 T8921 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38608/collection1: STATE STALE: collection1:7valid : false, retry? 0 [junit4] 2> 1646951 T8921 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1646954 T8903 C3533 P38608 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=to+come&_stateVer_=collection1:9&wt=javabin&version=2} hits=0 status=0 QTime=1 {code} # An indexing request comes in but there is no leader in the cached state. Every subsequent indexing request will continue to fail until the cache entry expires i.e. 60 seconds {code} [junit4] 2> 1647581 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 0 [junit4] 2> 1647582 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1647583 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 1 [junit4] 2> 1647583 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1647584 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 2 [junit4] 2> 1647584 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1647585 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 3 [junit4] 2> 1647585 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1647586 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 4 [junit4] 2> 1647587 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState WARN Re-trying request to collection(s) collection1 after stale state error from server. [junit4] 2> 1647588 T8920 oascsi.CloudSolrClient.requestWithRetryOnStaleState ERROR Request to collection collection1 failed due to (510) org.apache.solr.common.SolrException: Not enough nodes to handle the request, retry? 5 [junit4] 2> REQUEST FAILED for id=0-107 [junit4] 2> org.apache.solr.client.solrj.SolrServerException: org.apache.solr.common.SolrException: Not enough nodes to handle the request [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:873) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:738) [junit4] 2> at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:174) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103) [junit4] 2> Caused by: org.apache.solr.common.SolrException: Not enough nodes to handle the request [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1002) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:795) [junit4] 2> ... 10 more [junit4] 2> ROOT CAUSE for id=0-107 [junit4] 2> org.apache.solr.common.SolrException: Not enough nodes to handle the request [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1002) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:795) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:866) [junit4] 2> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:738) [junit4] 2> at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:174) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158) [junit4] 2> at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103) {code} > ChaosMonkeyNothingIsSafeTest fails with too many update fails > ------------------------------------------------------------- > > Key: SOLR-7118 > URL: https://issues.apache.org/jira/browse/SOLR-7118 > Project: Solr > Issue Type: Bug > Components: SolrCloud, Tests > Affects Versions: 5.0 > Reporter: Shalin Shekhar Mangar > Fix For: Trunk, 5.1 > > Attachments: SOLR-7118.patch > > > There are frequent failures on both trunk and branch_5x with the following > message: > {code} > java.lang.AssertionError: There were too many update fails - we expect it can > happen, but shouldn't easily > at > __randomizedtesting.SeedInfo.seed([786DB0FD42626C16:F98B3EE5353D0C2A]:0) > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertFalse(Assert.java:68) > at > org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:224) > at > org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:878) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org