Hi all, Bumping this before I start on the code. The only reply so far was Eric's idea about load-based trade-offs <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp>. That stays out of this proposal, but the two connect; more on that below. Nobody has raised concerns about the plan itself, and I want to be sure that silence means agreement and not that people missed it.
*The use case I didn't cover.* A lot of teams at my workplace now diagnose production issues with agents: Slack bots and Claude-style agents that call observability tools through MCP <https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro> and CLI. Someone asks, "Why was the ingress service slow at 14:05?" and the agent pulls traces, metrics and logs. For Solr, the only source today is solr_slow_requests.log <https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries> shipped to a log store. The agent can query that through something like the Splunk MCP server <https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools>, but its search tool gives up after one minute or 1,000 events, and the lines are only there if they fit in your daily license volume <https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works>. The agent ends up scanning raw log lines to rebuild something Solr already had in memory when the request finished. *How this API closes the gap*, without Solr shipping anything agent-specific: - Applications can already pass their own request ID to Solr through the X-Trace-Id header <https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation> (solr.traceIdHeader), or traceparent when the OpenTelemetry module <https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module> is on. Solr uses it as the trace ID, and every record carries it. - An agent can go from the application's request ID to the Solr query, its latency, CPU, hits and shard count in one call, straight from the cluster. - It's a v2 API, so it ends up in the generated OpenAPI spec and SolrJ client <https://github.com/apache/solr/blob/main/dev-docs/apis.adoc>. Wrapping it as an MCP tool is a few lines in whatever MCP server a team already runs. Solr itself stays out of MCP. To make that lookup cheap, I'm adding optional collection and traceId filters to GET /api/node/query-insights/top-queries. Everything else stays as described in the first mail <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd>. How this ties into Eric's idea. Shedding LTR or vector work under load needs to know which queries are expensive and what they ran. So each record will also note the costly features used (LTR, knn, grouping, faceting, paramset). The load control itself is a separate discussion. *How I'll deliver it.* One umbrella JIRA for Query Insights, then small sub-tasks, each its own PR that can be reviewed and merged on its own: 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994>: the SearchRequestListener hook in core. No behaviour change when nothing is registered. 2. The query-insights module: per-node top-N by latency and CPU, the features each query used, plus the node-local API. 3. Cluster-wide view: nodes=all merging, with nodes that didn't answer listed. 4. collection and traceId filters, and runtime settings through cluster properties. 5. Ref guide page. Stored history and grouping get their own sub-tasks later, after the first five land. *Two direct questions:* 1. Any objections to the plan as described? 2. Do you think this is a good fit for Solr at all? If you have doubts, I'd really value hearing them before I start. I'd rather adjust the idea now than spend reviewers' time on something the project doesn't want. If nothing comes up by Wednesday, Oct 21, I'll create the umbrella and the sub-tasks, post the full design there, and start with the core hook. I'm not a committer, so I'll also need someone to shepherd it. Eric, would you be up for that? Thanks, Amrit On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar <[email protected]> wrote: > Resending, the first copy seems stuck in moderation. > > Hi all, > > I've been away from Solr for a while. Coming back, I spent some time > digging into the observability side, both what Solr exposes today and how > other search engines handle it, to rebuild my understanding before > proposing anything. One gap stood out, and I'd like to pick up SOLR-11994 > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E> > ("Expose a slow query API", open since 2018) to close it. I want to check > the shape with you before writing code. > > *The problem.* When someone asks "which queries are hurting this cluster > right now?", Solr has no direct answer: > > - slowQueryThresholdMillis > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E> > writes to solr_slow_requests.log, which you have to ship somewhere and > grep. > - Request handler metrics > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E> > give latency per handler, not per query. > - Tracing > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E> > shows individual requests, if you run a backend and sample the right ones. > - /tasks/list > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E> > shows only queries that are still running, and only those sent with > canCancel=true. > > None of them gives a ranked list of the most expensive queries. > > OpenSearch solved this with its Query Insights > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E> > plugin (2.12+). Each node keeps an in-memory top-N of the searches it > coordinated > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, > ranked by latency, CPU or memory, over a rolling window. One API call > returns them from all nodes. It runs inside the OpenSearch process > <https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E>, > with no sidecar. > > *The proposal.* A new module, solr/modules/query-insights, modelled on > that plugin: > > - *Off by default, two steps to turn on.* List the module in > SOLR_MODULES > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E> > and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is > loaded. > - *Small hook in core.* A SearchRequestListener interface, called at > the end of SearchHandler > > <https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E> > request handling on the coordinator only (shard sub-requests are skipped). > With no listener registered, the cost is one isEmpty() check per > search. > - *Recording.* Per node, a top-N per metric (latency, CPU) for the > current and previous clock-aligned window. Defaults follow OpenSearch: N=10 > (max 100), 5-minute windows. A request that doesn't beat the current Nth > value costs a field read and nothing else. > - *Records.* Collection, handler, params (redactable, truncated at 8 > KB), hits, shard count, user, latency, CPU, and the trace id > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>. > The trace id is something OpenSearch doesn't have: it links each entry to > the request log and to traces. > - *API.* GET > /api/node/query-insights/top-queries?type=latency|cpu&nodes=all, needs > the metrics-read > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E> > permission. Merges across nodes and lists any node that didn't answer. > - *Settings.* System properties for defaults, ext.queryInsights.* cluster > properties > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E> > for runtime changes. > - *Not in the first PR.* Grouping similar queries by shape > > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, > memory as a metric, and stored history. History would come later as a Time > Routed Alias > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E> > with autoDeleteAge, which gives us retention without a custom cleanup > job. > > *Why not tracing?* Tracing and this work together rather than compete. > Tracing answers "what happened inside this request?" once you know which > one to look at. This answers "which requests should I look at?" without a > tracing backend, and the trace id in each record takes you straight to the > trace. > > *Questions for you:* > > 1. Module plus a small core listener, or would you rather see this in > core? > 2. Is metrics-read the right permission for data that includes query > text, given the redaction option? > 3. Anyone already working on something similar? I found SOLR-1101 > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E>, > SOLR-11994 > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E> > and Eric's User Behaviour Insights PR (#2452) > > <https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E>. > I'd like to line up with UBI when we get to stored history. > > I've written up a fuller design covering the core hook, recording, > settings, security and tests, and I'll share it on the JIRA. Feedback on > the direction here first would help me get it right before I start. > > Thanks, > Amrit >
