Hi all, I'm looking to augment Solr responses with more detail about the searcher to help diagnose a performance issue. We could expose some cheap facts per shard:
- searcher open timestamp - when the currently-registered searcher was opened (what I'm most interested in right now) - segment count - maxDocs - numDocs - isLeader? - stale? (changes are pending visibility by a new searcher, e.g. a commit. Maybe expose staleness as a time duration instead. Disclaimer: I tacked this on last minute; purely hypothetical addition I probably won't actually do) All but the last are easily retrieved from the SolrIndexSearcher. Together they make it easy to spot "shard2 is running a searcher from 5 minutes ago" (therefore may be warm / performant... or return ~stale results) or "these two shards disagree on doc counts" - currently you have to go correlate metrics or core admin calls out-of-band, per replica, after the fact. And in a distributed query the coordinator picks one replica per shard, so afterward, you need shards.info to know which replica actually served you. Producing the data is the easy part. The conundrum is where to put it. Solr already has multiple overlapping locations for per-shard diagnostics, so potentially adding a third requires careful consideration. I see three plausible homes. A dimension that distinguishes them is how you request the information. Option 1: shards.info Add the fields to the existing shards.info section. This is the established spot for per-shard information - it already carries numFound, maxScore, per-shard timing and errors, and it covers single-shard collections too, not just multi-shard ones. The main cost is that shards.info is currently assembled in several independent code paths, so any new field has to be added in each of them (the grouping path even uses a different top-level key). If we go this way there's a follow-on question: does the new information simply always appear whenever shards.info is requested, or does it need its own parameter to opt in? These fields are more debug-flavored than the rest of what shards.info carries, so reasonable people may differ. Option 2: a new debug value, e.g. debug=indexsearcher Alongside the existing debug=query / timing / results / track. This has a single implementation point, composes with the other debug values, and debug=all picks it up for free. Notably it also answers the opt-in question for free - debug is already a multi-valued parameter, so asking for this and not asking for it is a solved problem needing no new parameter. It keeps diagnostic payload out of shards.info too. The small drawback is that it adds yet another *per-shard* structure to the response, on top of shards.info and debug=track - which is awkward given how much those two already overlap. Perhaps debug=track may not "count" since it's undocumented and perhaps should stay that way. Option 3: a nested block in each shard's response header (new) Each shard could report its own searcher info in its responseHeader, and the coordinator surfaces the per-shard copies. The appeal is that a non-distributed request would see it with no extra machinery at all, since the response header is always returned. The drawback is it would need a new parameter to ask for it explicitly, since we wouldn't want this on by default for every request. And of course it's existence may beg the question of why not instead add onto shards.info or debug. Regardless of which option wins: distributed tracing Whichever home we pick for the response body, I'd like the same information to also be set as attributes on the distributed tracing spans. Each shard sub-request already gets its own span, so these facts belong naturally on the span for the shard that produced them. Arguably that's where the data is most valuable. Tracing already correlates the whole request tree for you, including which replica served each shard, and it doesn't require anyone to have thought to add a debug parameter to a production query - by which time the searcher may well have rolled over and the evidence is gone. Tracing already incorporates the notion of "sampling" to avoid excess information. Finally, I don't have a strong preference, but I do lean a little in this order #2, #1, #3. I like debug because (a) I like the structured namespace for choosing different debug categories -- just add another (b) the implementation is more straight-forward from what I'm seeing instead of shards.info which has some gotchas and is more spread out instead of isolated to a component. ~ David
