[
https://issues.apache.org/jira/browse/SOLR-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097476#comment-18097476
]
Serhiy Bzhezytskyy commented on SOLR-17764:
-------------------------------------------
[~dsmiley] , thanks — I dug into this a fair bit (landed here from a red CI on
#4637). One thing that helped me was that the failures under discussion split
into two different mechanisms, both surfacing around shutdown:
- ChaosMonkey / the tests here fail with a 503 (RemoteSolrException) — Solr
throws it when it notices shutdown mid-request (zkCheck), and CloudSolrClient
doesn't treat a 503 as retryable. That's the "should we retry on 503" question
you and Mark raised.
- Separately, TestGracefulJettyShutdown (and the closed-proxy cases on
SOLR-18188) fail with ClosedChannelException — a different thing: the
connection drops and the client never gets the response. The root cause there
is a Jetty HTTP/2 client bug, jetty#15368 — it discards an already-delivered
response when the session closes. I have a fix up at jetty/jetty.project#15435,
so that path stops producing the CCE at the source.
On the retry side, both come back to CloudSolrClient.wasCommError. #4643 adds
ClosedChannelException there (the CCE half); the 503 half is the separate call
you and Mark were weighing. So I think the CloudSolrClient retry-classification
piece fits this ticket; the pure test-framework flakiness
(ReplicationFactorTest etc.) is probably better on SOLR-18295 where Jan already
has it — but you know the layout better, happy either way.
(One aside if useful: CloudSolrClient's retry path and LBSolrClient don't treat
updates the same — LBSolrClient won't retry an update except on a connect
exception, while the CloudSolrClient path doesn't guard updates. Happy to dig
into whether that's worth reconciling, but that's a separate thread from
where-to-track-this.)
> "graceful" jetty shutdown causes ChaosMonkeySafeLeaderWithPullReplicasTest
> failures
> -----------------------------------------------------------------------------------
>
> Key: SOLR-17764
> URL: https://issues.apache.org/jira/browse/SOLR-17764
> Project: Solr
> Issue Type: Bug
> Reporter: Chris M. Hostetter
> Priority: Major
> Labels: pull-request-available
> Attachments:
> E7F93005B9386058.OUTPUT-org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Reviewing recent jenkins test failure metrics, I noticed that (Nightly) test
> ChaosMonkeySafeLeaderWithPullReplicasTest started failing ~60% of the time
> right around the time that SOLR-17744 was committed.
> Things i have observed:
> * Seeds from failing runs seem to reliably reproduce the failure
> ** These failures do *NOT* reproduce if i revert to just before SOLR-17744
> * Ad-hoc testing I've done of seeds that do _not_ fail on first attempt seem
> to reliably succeed on all subsequent attempts
> ** Suggesting that the root cause is something deterministic in the
> {{{}random(){}}}-ness of the test, and not something dependent on timing or
> concurrency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
