Hi David, Thanks for writing this up :) so my understanding: after a distributed query you already need shards.info to know which replica served each shard, and today you still have to go correlate Luke / core admin / metrics after the fact, by which time the searcher may have rolled. In-band on that request is the right idea, especially searcher open time.
+1 on putting the same facts on the shard sub-request spans no matter where the response body lands. That's the production path that doesn't depend on someone having thought to add a debug parameter, and sampling is already how tracing controls volume. I'd treat tracing as part of the same change, not a follow-up. On the response body I lean slightly the other way: #1, then #2, and I'd drop #3 These aren't facts about how the query was parsed or scored. They're facts about the replica that served this request. That's the job shards.info already has, shardAddress is there specifically so you can tell which replica ran. Putting openedAt / segmentCount / numDocs / maxDocs next to shardAddress is the natural join. Under debug, operators still have to correlate two trees to answer the question you started with. A nested `searcher` block, included whenever shards.info is on, feels like enough of an opt-in to me. shards.info is already off by default, the payload is a handful of numbers, and the rest of that section is already diagnostic (time, error, shardAddress). A second parameter would make the common case, "add shards.info to this slow query", miss the new fields unless you already knew to ask twice. debug=all picking this up "for free" isn't actually the production win it sounds like, because debug=all also turns on explains. If we do go #2, two existing DebugComponent behaviors are worth planning around so it isn't quite the isolated add it first appears: - modifyRequest currently strips debug off every shard request that isn't PURPOSE_GET_FIELDS, then re-adds timing and track. query / results never ride on GET_TOP_IDS. Searcher info needs to come from the replica that executed the query, including rows=0 (no GET_FIELDS). A new debug=indexsearcher would need to be forwarded the way track is, not the way query/results are. - finishStage merges per-shard debug (parsed query, explains, …). Searcher facts have to stay per-replica, keyed by who produced them, the track shape, not the query/results shape. And debug=all is not quite free either: setDebug() / isDebug() / isDebugAll() are a closed set of four flags today, so a fifth value has to be wired through SolrPluginUtils.getDebugInterests as well. There's also a small precedent for #1 that keeps the producer local to the replica: QueryComponent already copies segmentTerminatedEarly / maxHitsTerminatedEarly / approximateTotalHits off the shard's responseHeader into shards.info. A SearcherDiagnostics helper on SolrIndexSearcher could fill that same role for both the response and the span attributes, so the grouping copies (including shards.info.firstPhase) stay mechanical. Small correction: debug=track is documented, briefly, in solrcloud-distributed-requests.adoc. I agree it shouldn't become a second shards.info. I'd skip "stale" for a first cut; open timestamp already answers "how old is this view." isLeader is cluster-state rather than SolrIndexSearcher, but it's cheap from CloudDescriptor and useful sitting next to the rest. I could live with #2 if that's the consensus, the debug namespace is a genuine plus, but I'd rather not add a third per-shard tree for replica identity we already have a home for. Cheers, Prithvi S On Sun, Sep 20, 2026 at 4:18 AM David Smiley <[email protected]> wrote: > Hi all, > > I'm looking to augment Solr responses with more detail about the searcher > to help diagnose a performance issue. We could expose some cheap facts per > shard: > > - searcher open timestamp - when the currently-registered searcher was > opened (what I'm most interested in right now) > - segment count > - maxDocs > - numDocs > - isLeader? > - stale? (changes are pending visibility by a new searcher, e.g. a commit. > Maybe expose staleness as a time duration instead. Disclaimer: I tacked > this on last minute; purely hypothetical addition I probably won't actually > do) > > All but the last are easily retrieved from the SolrIndexSearcher. Together > they make it easy to spot "shard2 is running a searcher from 5 minutes ago" > (therefore may be warm / performant... or return ~stale results) or "these > two shards disagree on doc counts" - currently you have to go correlate > metrics or core admin calls out-of-band, per replica, after the fact. And > in a distributed query the coordinator picks one replica per shard, so > afterward, you need shards.info to know which replica actually served you. > > Producing the data is the easy part. The conundrum is where to put it. Solr > already has multiple overlapping locations for per-shard diagnostics, so > potentially adding a third requires careful consideration. I see three > plausible homes. A dimension that distinguishes them is how you request > the information. > > Option 1: shards.info > > Add the fields to the existing shards.info section. > > This is the established spot for per-shard information - it already carries > numFound, maxScore, per-shard timing and errors, and it covers single-shard > collections too, not just multi-shard ones. The main cost is that > shards.info is currently assembled in several independent code paths, so > any new field has to be added in each of them (the grouping path even uses > a different top-level key). > > If we go this way there's a follow-on question: does the new information > simply always appear whenever shards.info is requested, or does it need > its > own parameter to opt in? These fields are more debug-flavored than the rest > of what shards.info carries, so reasonable people may differ. > > > Option 2: a new debug value, e.g. debug=indexsearcher > > Alongside the existing debug=query / timing / results / track. > > This has a single implementation point, composes with the other debug > values, and debug=all picks it up for free. Notably it also answers the > opt-in question for free - debug is already a multi-valued parameter, so > asking for this and not asking for it is a solved problem needing no new > parameter. It keeps diagnostic payload out of shards.info too. > > The small drawback is that it adds yet another *per-shard* structure to the > response, on top of shards.info and debug=track - which is awkward given > how much those two already overlap. Perhaps debug=track may not "count" > since it's undocumented and perhaps should stay that way. > > > Option 3: a nested block in each shard's response header (new) > > Each shard could report its own searcher info in its responseHeader, and > the coordinator surfaces the per-shard copies. > > The appeal is that a non-distributed request would see it with no extra > machinery at all, since the response header is always returned. The > drawback is it would need a new parameter to ask for it explicitly, since > we wouldn't want this on by default for every request. And of course it's > existence may beg the question of why not instead add onto shards.info or > debug. > > > Regardless of which option wins: distributed tracing > > Whichever home we pick for the response body, I'd like the same information > to also be set as attributes on the distributed tracing spans. Each shard > sub-request already gets its own span, so these facts belong naturally on > the span for the shard that produced them. > > Arguably that's where the data is most valuable. Tracing already correlates > the whole request tree for you, including which replica served each shard, > and it doesn't require anyone to have thought to add a debug parameter to a > production query - by which time the searcher may well have rolled over and > the evidence is gone. Tracing already incorporates the notion of > "sampling" to avoid excess information. > > > Finally, I don't have a strong preference, but I do lean a little in this > order #2, #1, #3. I like debug because (a) I like the structured namespace > for choosing different debug categories -- just add another (b) the > implementation is more straight-forward from what I'm seeing instead of > shards.info which has some gotchas and is more spread out instead of > isolated to a component. > > ~ David >
