Dario — A few things to disentangle here, because "GOAWAY" is a transport-layer signal with several distinct triggers, and the fix depends on which one is firing.
What GOAWAY actually means. Per RFC 7540 §6.8, GOAWAY is the server telling the client: "I will not accept any new streams above this last-stream-id on this connection; streams already open below that id may still complete." It is not itself an error — it's Jetty (Solr's embedded server) telling the client to stop multiplexing more requests onto this particular TCP connection and open a new one for anything further. The two most common reasons Jetty emits it under concurrent indexing load are: Connection-level idle/lifecycle limits — Jetty proactively recycles HTTP/2 connections after they've carried a configured number of streams or been open for a configured duration, specifically so no single connection accumulates unbounded state or pins traffic to one node behind a load balancer. Concurrent stream limits — Jetty's HTTP/2 connector caps concurrent streams per connection (maxConcurrentStreams, server-side). If your indexer is firing many requests in a tight burst over few connections, you can hit that ceiling and get told to back off onto a new connection rather than getting queued. I don't want to hand you an exact numeric threshold from memory — those Jetty server settings vary by version and by Solr's jetty.xml/solr.xml config, and guessing a number here would be worse than useless. The fastest way to know which of the two is happening in your case: turn on Jetty's HTTP/2 debug logging (org.eclipse.jetty.http2 at DEBUG) on the Solr node and look at the actual GOAWAY error code in the frame — NO_ERROR typically means (1), a graceful recycle; anything else points to a protocol-level problem worth its own investigation. Does the request that triggered GOAWAY still get indexed? This is the part that actually matters operationally, and the honest answer is: it depends on whether Solr had already accepted and begun processing that stream before the GOAWAY was sent. GOAWAY only forbids new streams above the announced id — streams already in flight below it are explicitly allowed to complete per the RFC. So: If your client's request stream ID was below the last-stream-id in the GOAWAY frame, Solr should still process it and send a response normally. If it was above that id (i.e., the connection was already being drained when you tried to open it), the request was never accepted, and you must resend it — no document was written. The failure mode you need to design for is not "GOAWAY happened," it's "I don't have a response for this request."That's the actual retry trigger, independent of GOAWAY specifically — timeouts, resets, and GOAWAY-before-accept all land you in the same "unknown outcome" state. On resend safety: whether resending is safe depends on your update semantics, not on Solr's transport layer. A plain add/replace by id is idempotent — resending is harmless, worst case a duplicate write of the same state. If you're using atomic updates with increment-style operations (inc), a spurious resend after a request that actually succeeded silently would double-apply the delta. Worth checking whether your indexer's updates are idempotent before blanket-resending on any ambiguous failure. On your specific questions: Switch to HTTP/1.1? This removes the GOAWAY signal specifically (HTTP/1.1 has no equivalent stream-multiplexing concept), but it doesn't remove the underlying problem — you'll just get connection resets/timeouts instead when the server wants to shed load, harder to distinguish from a real failure. I wouldn't call it a fix, more a change of which symptom you see. Another client? HttpJdkSolrClient wraps the JDK 11+ built-in java.net.http.HttpClient, which is a different code path from Jetty's HttpClient used by the older Http2SolrClient. If you haven't already, it's worth testing whether the same burst pattern against Http2SolrClient (Jetty-client-backed) exhibits the same behavior — that would tell you whether this is server-side pressure (affects both clients equally) or something specific to the JDK client's stream/connection handling under this Solr version. Practical mitigation in the meantime: widen your client-side connection pool so the burst spreads across more physical connections instead of saturating one (reduces how often you hit per-connection stream caps), and make sure your indexer treats "no response received" (including GOAWAY-before-accept) as retryable with backoff, rather than a hard failure. If you can share the actual GOAWAY error code from the Jetty debug log, I can be a lot more precise about which of the two mechanisms above is in play. Shrey Narayan NextBricks [email protected] [email protected] Nextbrick.com On 2025/05/13 09:02:25 [email protected] wrote: > Dear Solr People > > On our system we have an indexer-service that indexes documents based on > incoming events. Sometimes a lot of these events happen at the same time (or > at least very close in time to each other). If that happens our indexer > receives GOAWAY signals from solr. > We're using the HttpJdkSolrClient with HTTP/2. Solr Version is 9.5.0 > > When exactly do these GOAWAY signals show up. In other words, how many > requests need to be made to solr in a certain amount of timespan that solr > starts emitting this signal. And how exactly should it be handled or fixed. > > Should we switch to HTTP/1.1? This version of HTTP does not specify this > signal, but this is probably not a solution? > Using another client? > > Can there be done anything else about it? Will the documents that were in the > request that caused the GOAWAY signal still get added to the index or do we > need to resend this request? And If yes, how long should we wait to resend it? > > With kind regards, > > Dario >
