[
https://issues.apache.org/jira/browse/SOLR-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18096708#comment-18096708
]
Serhiy Bzhezytskyy commented on SOLR-17764:
-------------------------------------------
Ran into this from a CI failure on an unrelated PR and did some digging — a few
findings that might help move it forward.
Another affected test:
TestGracefulJettyShutdown.testSingleShardInFlightRequestsDuringShutDown
reproduces it on current main — fails ~60% (3 of 5) with seed 284231B7A8B0E347.
Same signature as the ChaosMonkey/LeaderTragic ones: the server handler
completes ("Finishing request" logs), but the client gets
ClosedChannelException via HTTP2Session.sendGoAwayAndTerminate → onClose. This
matches Jan's Jetty issue
[jetty#15368|https://github.com/jetty/jetty.project/issues/15368] (HTTP/2
client discards an already-sent response on connection close).
Jetty 12.1.11 does not fix it. I bumped Jetty 12.1.10 → 12.1.11 (regenerated
lockfiles) and reran the failing seed 10x — still fails with the same
ClosedChannelException. So a version bump won't unblock this; it needs the
Jetty-side fix
([jetty#15368|https://github.com/jetty/jetty.project/issues/15368] is still
open). Pinning back to 12.1.9 is reported clean on that issue, though I didn't
verify 12.1.9 myself.
On the retry angle raised earlier in this thread:
CloudSolrClient.wasCommError() doesn't treat ClosedChannelException as a
communication error, so it never retries it (commError=false in the logs). I
opened a PR to fix that classification:
https://github.com/apache/solr/pull/4643 — to be clear it does not fix this
issue (for the single-node cluster in TestGracefulJettyShutdown there's no
other node to retry on, and for the Jetty bug the response is already lost),
but retrying a dropped connection seems correct on its own regardless.
> "graceful" jetty shutdown causes ChaosMonkeySafeLeaderWithPullReplicasTest
> failures
> -----------------------------------------------------------------------------------
>
> Key: SOLR-17764
> URL: https://issues.apache.org/jira/browse/SOLR-17764
> Project: Solr
> Issue Type: Bug
> Reporter: Chris M. Hostetter
> Priority: Major
> Labels: pull-request-available
> Attachments:
> E7F93005B9386058.OUTPUT-org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.txt
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Reviewing recent jenkins test failure metrics, I noticed that (Nightly) test
> ChaosMonkeySafeLeaderWithPullReplicasTest started failing ~60% of the time
> right around the time that SOLR-17744 was committed.
> Things i have observed:
> * Seeds from failing runs seem to reliably reproduce the failure
> ** These failures do *NOT* reproduce if i revert to just before SOLR-17744
> * Ad-hoc testing I've done of seeds that do _not_ fail on first attempt seem
> to reliably succeed on all subsequent attempts
> ** Suggesting that the root cause is something deterministic in the
> {{{}random(){}}}-ness of the test, and not something dependent on timing or
> concurrency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
