Please tell me what this solves that distributed tracing (e.g. via OTEL standards) does not solve. I'm aware distributed tracing implies additional/other infrastructure, but that's becoming increasingly common. And it's stuff we/Solr don't have to maintain. Of course we should do more to make tracing more awesome (more spans, more detailed spans).
> None of them gives a ranked list of the most expensive queries. Of course tracing can do that! On Thu, Oct 22, 2026 at 8:31 PM Amrit Sarkar <[email protected]> wrote: > Hi all, > > No objections came in, so I've opened the umbrella: SOLR-18571 > < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1792801902707000&sa=E > >, > with the full design attached and the five sub-tasks from my last mail. I'm > starting on SOLR-11994 > < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1792801902707000&sa=E > >, > the core hook. > > Feedback is still welcome on the umbrella or on any PR as they come up. > > Thanks, > Amrit > > On Sun, Oct 18, 2026 09:53 AM, Amrit Sarkar <[email protected]> > wrote: > > > Hi all, > > > > Bumping this before I start on the code. The only reply so far was Eric's > > idea about load-based trade-offs > > <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp>. That > > stays out of this proposal, but the two connect; more on that below. > Nobody > > has raised concerns about the plan itself, and I want to be sure that > > silence means agreement and not that people missed it. > > > > *The use case I didn't cover.* A lot of teams at my workplace now > > diagnose production issues with agents: Slack bots and Claude-style > agents > > that call observability tools through MCP > > <https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro> > and > > CLI. Someone asks, "Why was the ingress service slow at 14:05?" and the > > agent pulls traces, metrics and logs. > > > > For Solr, the only source today is solr_slow_requests.log > > < > https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries > > > > shipped to a log store. The agent can query that through something like > the Splunk > > MCP server > > < > https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools > >, > > but its search tool gives up after one minute or 1,000 events, and the > > lines are only there if they fit in your daily license volume > > < > https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works > >. > > The agent ends up scanning raw log lines to rebuild something Solr > already > > had in memory when the request finished. > > > > *How this API closes the gap*, without Solr shipping anything > > agent-specific: > > > > - Applications can already pass their own request ID to Solr through > > the X-Trace-Id header > > < > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation > > > > (solr.traceIdHeader), or traceparent when the OpenTelemetry module > > < > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module > > > > is on. Solr uses it as the trace ID, and every record carries it. > > - An agent can go from the application's request ID to the Solr query, > > its latency, CPU, hits and shard count in one call, straight from the > > cluster. > > - It's a v2 API, so it ends up in the generated OpenAPI spec and SolrJ > > client <https://github.com/apache/solr/blob/main/dev-docs/apis.adoc>. > > Wrapping it as an MCP tool is a few lines in whatever MCP server a > team > > already runs. Solr itself stays out of MCP. > > > > To make that lookup cheap, I'm adding optional collection and traceId > > filters to GET /api/node/query-insights/top-queries. Everything else > > stays as described in the first mail > > <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd>. > > > > How this ties into Eric's idea. Shedding LTR or vector work under load > needs > > to know which queries are expensive and what they ran. So each record > > will also note the costly features used (LTR, knn, grouping, faceting, > > paramset). The load control itself is a separate discussion. > > > > *How I'll deliver it.* One umbrella JIRA for Query Insights, then small > > sub-tasks, each its own PR that can be reviewed and merged on its own: > > > > 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994>: the > > SearchRequestListener hook in core. No behaviour change when nothing > > is registered. > > 2. The query-insights module: per-node top-N by latency and CPU, the > > features each query used, plus the node-local API. > > 3. Cluster-wide view: nodes=all merging, with nodes that didn't answer > > listed. > > 4. collection and traceId filters, and runtime settings through > > cluster properties. > > 5. Ref guide page. > > > > Stored history and grouping get their own sub-tasks later, after the > first > > five land. > > > > *Two direct questions:* > > > > 1. Any objections to the plan as described? > > 2. Do you think this is a good fit for Solr at all? If you have > > doubts, I'd really value hearing them before I start. I'd rather > adjust the > > idea now than spend reviewers' time on something the project doesn't > want. > > > > If nothing comes up by Wednesday, Oct 21, I'll create the umbrella and > > the sub-tasks, post the full design there, and start with the core hook. > > I'm not a committer, so I'll also need someone to shepherd it. Eric, > would > > you be up for that? > > > > Thanks, > > Amrit > > > > On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar <[email protected]> > > wrote: > > > >> Resending, the first copy seems stuck in moderation. > >> > >> Hi all, > >> > >> I've been away from Solr for a while. Coming back, I spent some time > >> digging into the observability side, both what Solr exposes today and > how > >> other search engines handle it, to rebuild my understanding before > >> proposing anything. One gap stood out, and I'd like to pick up > SOLR-11994 > >> < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E > > > >> ("Expose a slow query API", open since 2018) to close it. I want to > check > >> the shape with you before writing code. > >> > >> *The problem.* When someone asks "which queries are hurting this cluster > >> right now?", Solr has no direct answer: > >> > >> - slowQueryThresholdMillis > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E > > > >> writes to solr_slow_requests.log, which you have to ship somewhere > >> and grep. > >> - Request handler metrics > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E > > > >> give latency per handler, not per query. > >> - Tracing > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > > >> shows individual requests, if you run a backend and sample the right > ones. > >> - /tasks/list > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E > > > >> shows only queries that are still running, and only those sent with > >> canCancel=true. > >> > >> None of them gives a ranked list of the most expensive queries. > >> > >> OpenSearch solved this with its Query Insights > >> < > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E > > > >> plugin (2.12+). Each node keeps an in-memory top-N of the searches it > >> coordinated > >> < > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E > >, > >> ranked by latency, CPU or memory, over a rolling window. One API call > >> returns them from all nodes. It runs inside the OpenSearch process > >> < > https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E > >, > >> with no sidecar. > >> > >> *The proposal.* A new module, solr/modules/query-insights, modelled on > >> that plugin: > >> > >> - *Off by default, two steps to turn on.* List the module in > >> SOLR_MODULES > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E > > > >> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is > >> loaded. > >> - *Small hook in core.* A SearchRequestListener interface, called at > >> the end of SearchHandler > >> < > https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E > > > >> request handling on the coordinator only (shard sub-requests are > skipped). > >> With no listener registered, the cost is one isEmpty() check per > >> search. > >> - *Recording.* Per node, a top-N per metric (latency, CPU) for the > >> current and previous clock-aligned window. Defaults follow > OpenSearch: N=10 > >> (max 100), 5-minute windows. A request that doesn't beat the current > Nth > >> value costs a field read and nothing else. > >> - *Records.* Collection, handler, params (redactable, truncated at 8 > >> KB), hits, shard count, user, latency, CPU, and the trace id > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > >. > >> The trace id is something OpenSearch doesn't have: it links each > entry to > >> the request log and to traces. > >> - *API.* GET /api/node/query-insights/top-queries?type=latency|cpu& > >> nodes=all, needs the metrics-read > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E > > > >> permission. Merges across nodes and lists any node that didn't > answer. > >> - *Settings.* System properties for defaults, ext.queryInsights.* > cluster > >> properties > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E > > > >> for runtime changes. > >> - *Not in the first PR.* Grouping similar queries by shape > >> < > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E > >, > >> memory as a metric, and stored history. History would come later as > a Time > >> Routed Alias > >> < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E > > > >> with autoDeleteAge, which gives us retention without a custom cleanup > >> job. > >> > >> *Why not tracing?* Tracing and this work together rather than compete. > >> Tracing answers "what happened inside this request?" once you know which > >> one to look at. This answers "which requests should I look at?" without > a > >> tracing backend, and the trace id in each record takes you straight to > the > >> trace. > >> > >> *Questions for you:* > >> > >> 1. Module plus a small core listener, or would you rather see this in > >> core? > >> 2. Is metrics-read the right permission for data that includes query > >> text, given the redaction option? > >> 3. Anyone already working on something similar? I found SOLR-1101 > >> < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E > >, > >> SOLR-11994 > >> < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E > > > >> and Eric's User Behaviour Insights PR (#2452) > >> < > https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E > >. > >> I'd like to line up with UBI when we get to stored history. > >> > >> I've written up a fuller design covering the core hook, recording, > >> settings, security and tests, and I'll share it on the JIRA. Feedback on > >> the direction here first would help me get it right before I start. > >> > >> Thanks, > >> Amrit > >> > > >
