Hi David, Fair challenge, and you're right that a tracing backend can sort by duration. I've worked on distributed tracing for the last two years, and the catch I keep running into is that *it can only rank what it was sent.*
Where tracing falls short here: - *Sampling decides before anyone knows the request is slow.* Our tracing page <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html%23sampling&source=gmail&ust=1793392173181000&sa=E> says tracing every request is rarely affordable and shows a 10% ratio, and Solr follows the caller's decision <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java%23L68&source=gmail&ust=1793392173181000&sa=E>. If the gateway samples 1%, a bad query shape can hurt the cluster for minutes and leave no trace at all. - *Tail sampling costs full volume.* Every span gets exported, and the collectors need each whole trace on one instance <https://www.google.com/url?q=https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor%23scaling-collectors-with-the-tail-sampling-processor&source=gmail&ust=1793392173181000&sa=E>, held in memory until they decide. You ship 100% of traffic to keep the slow 1%. - *A search span is thin.* One span per request, no CPU time, no hit count. http.params only comes from the URL query string <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/tracing/TraceUtils.java%23L159-L161&source=gmail&ust=1793392173181000&sa=E>, so POSTed and JSON DSL <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/query-guide/json-query-dsl.html&source=gmail&ust=1793392173181000&sa=E> queries never reach it. - *Duration points at the victim.* On a saturated node, the longest traces are the cheap queries that waited, not the one burning CPU. - *The slow request log <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1793392173181000&sa=E> has the same blind spot.* One QTime threshold, a line per core on each node, no CPU ranking. What Query Insights adds: - *Every request, ranked.* Top N by latency and by CPU (from ThreadCpuTimer <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/ThreadCpuTimer.java&source=gmail&ust=1793392173181000&sa=E>) per time window. One call, GET /api/node/query-insights/top-queries?type=cpu&nodes=all, merges every node and lists any that didn't answer. - *Cheap and bounded.* About 0.35 MB per node at defaults, no background threads. A request that doesn't make the list costs one comparison. - *Works with tracing.* Each record carries the trace ID <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1793392173181000&sa=E>, so the list tells you which traces to open. - *Little to maintain.* Two lines in solr.in.sh <https://www.google.com/url?q=http://solr.in.sh&source=gmail&ust=1793392173181000&sa=E> turn it on. In Solr, it's a small listener in SearchHandler (#5238 <https://www.google.com/url?q=https://github.com/apache/solr/pull/5238&source=gmail&ust=1793392173181000&sa=E>) plus a module that's off by default. On making tracing better, agreed, and I'll take two pieces: - A JIRA to add CPU time and hit count to the search span. - SOLR-13732 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-13732&source=gmail&ust=1793392173181000&sa=E>, Dat's issue from 2019: always trace requests that carry a configured header. It pairs well with this. Find the query in Query Insights, replay it with the header, and you get its full trace even at 1% sampling. Unless you see a hole in this, I'll keep going with SOLR-18571 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1793392173181000&sa=E> . Thanks, Amrit On Thu, Oct 29, 2026 12:11 PM, David Smiley <[email protected]> wrote: > Please tell me what this solves that distributed tracing (e.g. via OTEL > standards) does not solve. I'm aware distributed tracing implies > additional/other infrastructure, but that's becoming increasingly common. > And it's stuff we/Solr don't have to maintain. Of course we should do more > to make tracing more awesome (more spans, more detailed spans). > > > None of them gives a ranked list of the most expensive queries. > > Of course tracing can do that! > > On Thu, Oct 22, 2026 at 8:31 PM Amrit Sarkar <[email protected]> > wrote: > > > Hi all, > > > > No objections came in, so I've opened the umbrella: SOLR-18571 > > < > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1792801902707000&sa=E > > >, > > with the full design attached and the five sub-tasks from my last mail. > I'm > > starting on SOLR-11994 > > < > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1792801902707000&sa=E > > >, > > the core hook. > > > > Feedback is still welcome on the umbrella or on any PR as they come up. > > > > Thanks, > > Amrit > > > > On Sun, Oct 18, 2026 09:53 AM, Amrit Sarkar <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Bumping this before I start on the code. The only reply so far was > Eric's > > > idea about load-based trade-offs > > > <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp>. > That > > > stays out of this proposal, but the two connect; more on that below. > > Nobody > > > has raised concerns about the plan itself, and I want to be sure that > > > silence means agreement and not that people missed it. > > > > > > *The use case I didn't cover.* A lot of teams at my workplace now > > > diagnose production issues with agents: Slack bots and Claude-style > > agents > > > that call observability tools through MCP > > > <https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro > > > > and > > > CLI. Someone asks, "Why was the ingress service slow at 14:05?" and the > > > agent pulls traces, metrics and logs. > > > > > > For Solr, the only source today is solr_slow_requests.log > > > < > > > https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries > > > > > > shipped to a log store. The agent can query that through something like > > the Splunk > > > MCP server > > > < > > > https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools > > >, > > > but its search tool gives up after one minute or 1,000 events, and the > > > lines are only there if they fit in your daily license volume > > > < > > > https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works > > >. > > > The agent ends up scanning raw log lines to rebuild something Solr > > already > > > had in memory when the request finished. > > > > > > *How this API closes the gap*, without Solr shipping anything > > > agent-specific: > > > > > > - Applications can already pass their own request ID to Solr through > > > the X-Trace-Id header > > > < > > > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation > > > > > > (solr.traceIdHeader), or traceparent when the OpenTelemetry module > > > < > > > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module > > > > > > is on. Solr uses it as the trace ID, and every record carries it. > > > - An agent can go from the application's request ID to the Solr > query, > > > its latency, CPU, hits and shard count in one call, straight from > the > > > cluster. > > > - It's a v2 API, so it ends up in the generated OpenAPI spec and > SolrJ > > > client <https://github.com/apache/solr/blob/main/dev-docs/apis.adoc > >. > > > Wrapping it as an MCP tool is a few lines in whatever MCP server a > > team > > > already runs. Solr itself stays out of MCP. > > > > > > To make that lookup cheap, I'm adding optional collection and traceId > > > filters to GET /api/node/query-insights/top-queries. Everything else > > > stays as described in the first mail > > > <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd>. > > > > > > How this ties into Eric's idea. Shedding LTR or vector work under load > > needs > > > to know which queries are expensive and what they ran. So each record > > > will also note the costly features used (LTR, knn, grouping, faceting, > > > paramset). The load control itself is a separate discussion. > > > > > > *How I'll deliver it.* One umbrella JIRA for Query Insights, then small > > > sub-tasks, each its own PR that can be reviewed and merged on its own: > > > > > > 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994>: > the > > > SearchRequestListener hook in core. No behaviour change when nothing > > > is registered. > > > 2. The query-insights module: per-node top-N by latency and CPU, the > > > features each query used, plus the node-local API. > > > 3. Cluster-wide view: nodes=all merging, with nodes that didn't > answer > > > listed. > > > 4. collection and traceId filters, and runtime settings through > > > cluster properties. > > > 5. Ref guide page. > > > > > > Stored history and grouping get their own sub-tasks later, after the > > first > > > five land. > > > > > > *Two direct questions:* > > > > > > 1. Any objections to the plan as described? > > > 2. Do you think this is a good fit for Solr at all? If you have > > > doubts, I'd really value hearing them before I start. I'd rather > > adjust the > > > idea now than spend reviewers' time on something the project doesn't > > want. > > > > > > If nothing comes up by Wednesday, Oct 21, I'll create the umbrella and > > > the sub-tasks, post the full design there, and start with the core > hook. > > > I'm not a committer, so I'll also need someone to shepherd it. Eric, > > would > > > you be up for that? > > > > > > Thanks, > > > Amrit > > > > > > On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar <[email protected]> > > > wrote: > > > > > >> Resending, the first copy seems stuck in moderation. > > >> > > >> Hi all, > > >> > > >> I've been away from Solr for a while. Coming back, I spent some time > > >> digging into the observability side, both what Solr exposes today and > > how > > >> other search engines handle it, to rebuild my understanding before > > >> proposing anything. One gap stood out, and I'd like to pick up > > SOLR-11994 > > >> < > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E > > > > > >> ("Expose a slow query API", open since 2018) to close it. I want to > > check > > >> the shape with you before writing code. > > >> > > >> *The problem.* When someone asks "which queries are hurting this > cluster > > >> right now?", Solr has no direct answer: > > >> > > >> - slowQueryThresholdMillis > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E > > > > > >> writes to solr_slow_requests.log, which you have to ship somewhere > > >> and grep. > > >> - Request handler metrics > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E > > > > > >> give latency per handler, not per query. > > >> - Tracing > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > > > > >> shows individual requests, if you run a backend and sample the > right > > ones. > > >> - /tasks/list > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E > > > > > >> shows only queries that are still running, and only those sent with > > >> canCancel=true. > > >> > > >> None of them gives a ranked list of the most expensive queries. > > >> > > >> OpenSearch solved this with its Query Insights > > >> < > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E > > > > > >> plugin (2.12+). Each node keeps an in-memory top-N of the searches it > > >> coordinated > > >> < > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > >, > > >> ranked by latency, CPU or memory, over a rolling window. One API call > > >> returns them from all nodes. It runs inside the OpenSearch process > > >> < > > > https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E > > >, > > >> with no sidecar. > > >> > > >> *The proposal.* A new module, solr/modules/query-insights, modelled on > > >> that plugin: > > >> > > >> - *Off by default, two steps to turn on.* List the module in > > >> SOLR_MODULES > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E > > > > > >> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is > > >> loaded. > > >> - *Small hook in core.* A SearchRequestListener interface, called > at > > >> the end of SearchHandler > > >> < > > > https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E > > > > > >> request handling on the coordinator only (shard sub-requests are > > skipped). > > >> With no listener registered, the cost is one isEmpty() check per > > >> search. > > >> - *Recording.* Per node, a top-N per metric (latency, CPU) for the > > >> current and previous clock-aligned window. Defaults follow > > OpenSearch: N=10 > > >> (max 100), 5-minute windows. A request that doesn't beat the > current > > Nth > > >> value costs a field read and nothing else. > > >> - *Records.* Collection, handler, params (redactable, truncated at > 8 > > >> KB), hits, shard count, user, latency, CPU, and the trace id > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > >. > > >> The trace id is something OpenSearch doesn't have: it links each > > entry to > > >> the request log and to traces. > > >> - *API.* GET /api/node/query-insights/top-queries?type=latency|cpu& > > >> nodes=all, needs the metrics-read > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E > > > > > >> permission. Merges across nodes and lists any node that didn't > > answer. > > >> - *Settings.* System properties for defaults, ext.queryInsights.* > > cluster > > >> properties > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E > > > > > >> for runtime changes. > > >> - *Not in the first PR.* Grouping similar queries by shape > > >> < > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > >, > > >> memory as a metric, and stored history. History would come later as > > a Time > > >> Routed Alias > > >> < > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E > > > > > >> with autoDeleteAge, which gives us retention without a custom > cleanup > > >> job. > > >> > > >> *Why not tracing?* Tracing and this work together rather than compete. > > >> Tracing answers "what happened inside this request?" once you know > which > > >> one to look at. This answers "which requests should I look at?" > without > > a > > >> tracing backend, and the trace id in each record takes you straight to > > the > > >> trace. > > >> > > >> *Questions for you:* > > >> > > >> 1. Module plus a small core listener, or would you rather see this > in > > >> core? > > >> 2. Is metrics-read the right permission for data that includes > query > > >> text, given the redaction option? > > >> 3. Anyone already working on something similar? I found SOLR-1101 > > >> < > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E > > >, > > >> SOLR-11994 > > >> < > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E > > > > > >> and Eric's User Behaviour Insights PR (#2452) > > >> < > > > https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E > > >. > > >> I'd like to line up with UBI when we get to stored history. > > >> > > >> I've written up a fuller design covering the core hook, recording, > > >> settings, security and tests, and I'll share it on the JIRA. Feedback > on > > >> the direction here first would help me get it right before I start. > > >> > > >> Thanks, > > >> Amrit > > >> > > > > > >
