Hi Eric, I agree with where you want this to go.
One caution I'd like us to keep in front of us: *self-healing software can make the wrong call, and in production a wrong call can be catastrophic*. In a users@ thread from early 2024 <https://www.google.com/url?q=https://lists.apache.org/thread/qw25bzxpjyz1ncfms89y6x2pozjcnkn1&source=gmail&ust=1793408914905000&sa=E>, a query with an LTR re-ranker filled the heap and triggered a stop-the-world GC; the nodes went unhealthy, restarted, and *the entire cluster went down*. A stop-the-world pause freezes every request on the node, so anything acting on latency alone would have punished the *victims*, not the query filling the heap. There's a reason production access is denied to most engineers and changes go through a second pair of eyes. Software acting on prod by itself deserves at least that much care. So for now I'm keeping this effort to *making load visible and trustworthy*. That way, when someone does decide to act, they act on evidence. Thanks, Amrit On Thu, Oct 29, 2026 05:34 PM, Eric Pugh <[email protected]> wrote: > > > On 2026/10/29 20:31:43 Amrit Sarkar wrote: > > Hi David, > > > > Fair challenge, and you're right that a tracing backend can sort by > > duration. I've worked on distributed tracing for the last two years, and > > the catch I keep running into is that *it can only rank what it was > sent.* > > > > Where tracing falls short here: > > > > - *Sampling decides before anyone knows the request is slow.* Our > tracing > > page > > < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html%23sampling&source=gmail&ust=1793392173181000&sa=E > > > > says tracing every request is rarely affordable and shows a 10% > ratio, and > > Solr follows the caller's decision > > < > https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java%23L68&source=gmail&ust=1793392173181000&sa=E > >. > > If the gateway samples 1%, a bad query shape can hurt the cluster for > > minutes and leave no trace at all. > > - *Tail sampling costs full volume.* Every span gets exported, and > > the collectors > > need each whole trace on one instance > > < > https://www.google.com/url?q=https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor%23scaling-collectors-with-the-tail-sampling-processor&source=gmail&ust=1793392173181000&sa=E > >, > > held in memory until they decide. You ship 100% of traffic to keep > the slow > > 1%. > > - *A search span is thin.* One span per request, no CPU time, no hit > > count. http.params only comes from the URL query string > > < > https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/tracing/TraceUtils.java%23L159-L161&source=gmail&ust=1793392173181000&sa=E > >, > > so POSTed and JSON DSL > > < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/query-guide/json-query-dsl.html&source=gmail&ust=1793392173181000&sa=E > > > > queries never reach it. > > - *Duration points at the victim.* On a saturated node, the longest > > traces are the cheap queries that waited, not the one burning CPU. > > - *The slow request log > > < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1793392173181000&sa=E > > > > has the same blind spot.* One QTime threshold, a line per core on each > > node, no CPU ranking. > > > > What Query Insights adds: > > > > - *Every request, ranked.* Top N by latency and by CPU (from > > ThreadCpuTimer > > < > https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/ThreadCpuTimer.java&source=gmail&ust=1793392173181000&sa=E > >) > > per time window. One call, GET > > /api/node/query-insights/top-queries?type=cpu&nodes=all, merges every > node > > and lists any that didn't answer. > > - *Cheap and bounded.* About 0.35 MB per node at defaults, no > background > > threads. A request that doesn't make the list costs one comparison. > > - *Works with tracing.* Each record carries the trace ID > > < > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1793392173181000&sa=E > >, > > so the list tells you which traces to open. > > - *Little to maintain.* Two lines in solr.in.sh > > < > https://www.google.com/url?q=http://solr.in.sh&source=gmail&ust=1793392173181000&sa=E > > > > turn it on. In Solr, it's a small listener in SearchHandler (#5238 > > < > https://www.google.com/url?q=https://github.com/apache/solr/pull/5238&source=gmail&ust=1793392173181000&sa=E > >) > > plus a module that's off by default. > > > > On making tracing better, agreed, and I'll take two pieces: > > > > - A JIRA to add CPU time and hit count to the search span. > > - SOLR-13732 > > < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-13732&source=gmail&ust=1793392173181000&sa=E > >, > > Dat's issue from 2019: always trace requests that carry a configured > > header. It pairs well with this. Find the query in Query Insights, > replay > > it with the header, and you get its full trace even at 1% sampling. > > > > Unless you see a hole in this, I'll keep going with SOLR-18571 > > < > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1793392173181000&sa=E > > > > . > > > > Thanks, > > Amrit > > > > On Thu, Oct 29, 2026 12:11 PM, David Smiley <[email protected]> wrote: > > > > > Please tell me what this solves that distributed tracing (e.g. via OTEL > > > standards) does not solve. I'm aware distributed tracing implies > > > additional/other infrastructure, but that's becoming increasingly > common. > > > And it's stuff we/Solr don't have to maintain. Of course we should do > more > > > to make tracing more awesome (more spans, more detailed spans). > > > > > > > None of them gives a ranked list of the most expensive queries. > > > > > > Of course tracing can do that! > > > > > > On Thu, Oct 22, 2026 at 8:31 PM Amrit Sarkar <[email protected]> > > > wrote: > > > > > > > Hi all, > > > > > > > > No objections came in, so I've opened the umbrella: SOLR-18571 > > > > < > > > > > > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1792801902707000&sa=E > > > > >, > > > > with the full design attached and the five sub-tasks from my last > mail. > > > I'm > > > > starting on SOLR-11994 > > > > < > > > > > > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1792801902707000&sa=E > > > > >, > > > > the core hook. > > > > > > > > Feedback is still welcome on the umbrella or on any PR as they come > up. > > > > > > > > Thanks, > > > > Amrit > > > > > > > > On Sun, Oct 18, 2026 09:53 AM, Amrit Sarkar <[email protected]> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Bumping this before I start on the code. The only reply so far was > > > Eric's > > > > > idea about load-based trade-offs > > > > > <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp > >. > > > That > > > > > stays out of this proposal, but the two connect; more on that > below. > > > > Nobody > > > > > has raised concerns about the plan itself, and I want to be sure > that > > > > > silence means agreement and not that people missed it. > > > > > > > > > > *The use case I didn't cover.* A lot of teams at my workplace now > > > > > diagnose production issues with agents: Slack bots and Claude-style > > > > agents > > > > > that call observability tools through MCP > > > > > < > https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro > > > > > > > > and > > > > > CLI. Someone asks, "Why was the ingress service slow at 14:05?" > and the > > > > > agent pulls traces, metrics and logs. > > > > > > > > > > For Solr, the only source today is solr_slow_requests.log > > > > > < > > > > > > > > https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries > > > > > > > > > > shipped to a log store. The agent can query that through something > like > > > > the Splunk > > > > > MCP server > > > > > < > > > > > > > > https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools > > > > >, > > > > > but its search tool gives up after one minute or 1,000 events, and > the > > > > > lines are only there if they fit in your daily license volume > > > > > < > > > > > > > > https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works > > > > >. > > > > > The agent ends up scanning raw log lines to rebuild something Solr > > > > already > > > > > had in memory when the request finished. > > > > > > > > > > *How this API closes the gap*, without Solr shipping anything > > > > > agent-specific: > > > > > > > > > > - Applications can already pass their own request ID to Solr > through > > > > > the X-Trace-Id header > > > > > < > > > > > > > > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation > > > > > > > > > > (solr.traceIdHeader), or traceparent when the OpenTelemetry > module > > > > > < > > > > > > > > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module > > > > > > > > > > is on. Solr uses it as the trace ID, and every record carries > it. > > > > > - An agent can go from the application's request ID to the Solr > > > query, > > > > > its latency, CPU, hits and shard count in one call, straight > from > > > the > > > > > cluster. > > > > > - It's a v2 API, so it ends up in the generated OpenAPI spec and > > > SolrJ > > > > > client < > https://github.com/apache/solr/blob/main/dev-docs/apis.adoc > > > >. > > > > > Wrapping it as an MCP tool is a few lines in whatever MCP > server a > > > > team > > > > > already runs. Solr itself stays out of MCP. > > > > > > > > > > To make that lookup cheap, I'm adding optional collection and > traceId > > > > > filters to GET /api/node/query-insights/top-queries. Everything > else > > > > > stays as described in the first mail > > > > > <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd > >. > > > > > > > > > > How this ties into Eric's idea. Shedding LTR or vector work under > load > > > > needs > > > > > to know which queries are expensive and what they ran. So each > record > > > > > will also note the costly features used (LTR, knn, grouping, > faceting, > > > > > paramset). The load control itself is a separate discussion. > > > > > > > > > > *How I'll deliver it.* One umbrella JIRA for Query Insights, then > small > > > > > sub-tasks, each its own PR that can be reviewed and merged on its > own: > > > > > > > > > > 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994 > >: > > > the > > > > > SearchRequestListener hook in core. No behaviour change when > nothing > > > > > is registered. > > > > > 2. The query-insights module: per-node top-N by latency and > CPU, the > > > > > features each query used, plus the node-local API. > > > > > 3. Cluster-wide view: nodes=all merging, with nodes that didn't > > > answer > > > > > listed. > > > > > 4. collection and traceId filters, and runtime settings through > > > > > cluster properties. > > > > > 5. Ref guide page. > > > > > > > > > > Stored history and grouping get their own sub-tasks later, after > the > > > > first > > > > > five land. > > > > > > > > > > *Two direct questions:* > > > > > > > > > > 1. Any objections to the plan as described? > > > > > 2. Do you think this is a good fit for Solr at all? If you have > > > > > doubts, I'd really value hearing them before I start. I'd rather > > > > adjust the > > > > > idea now than spend reviewers' time on something the project > doesn't > > > > want. > > > > > > > > > > If nothing comes up by Wednesday, Oct 21, I'll create the umbrella > and > > > > > the sub-tasks, post the full design there, and start with the core > > > hook. > > > > > I'm not a committer, so I'll also need someone to shepherd it. > Eric, > > > > would > > > > > you be up for that? > > > > > > > > > > Thanks, > > > > > Amrit > > > > > > > > > > On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar < > [email protected]> > > > > > wrote: > > > > > > > > > >> Resending, the first copy seems stuck in moderation. > > > > >> > > > > >> Hi all, > > > > >> > > > > >> I've been away from Solr for a while. Coming back, I spent some > time > > > > >> digging into the observability side, both what Solr exposes today > and > > > > how > > > > >> other search engines handle it, to rebuild my understanding before > > > > >> proposing anything. One gap stood out, and I'd like to pick up > > > > SOLR-11994 > > > > >> < > > > > > > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E > > > > > > > > > >> ("Expose a slow query API", open since 2018) to close it. I want > to > > > > check > > > > >> the shape with you before writing code. > > > > >> > > > > >> *The problem.* When someone asks "which queries are hurting this > > > cluster > > > > >> right now?", Solr has no direct answer: > > > > >> > > > > >> - slowQueryThresholdMillis > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E > > > > > > > > > >> writes to solr_slow_requests.log, which you have to ship > somewhere > > > > >> and grep. > > > > >> - Request handler metrics > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E > > > > > > > > > >> give latency per handler, not per query. > > > > >> - Tracing > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> shows individual requests, if you run a backend and sample the > > > right > > > > ones. > > > > >> - /tasks/list > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> shows only queries that are still running, and only those sent > with > > > > >> canCancel=true. > > > > >> > > > > >> None of them gives a ranked list of the most expensive queries. > > > > >> > > > > >> OpenSearch solved this with its Query Insights > > > > >> < > > > > > > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> plugin (2.12+). Each node keeps an in-memory top-N of the > searches it > > > > >> coordinated > > > > >> < > > > > > > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > > > >, > > > > >> ranked by latency, CPU or memory, over a rolling window. One API > call > > > > >> returns them from all nodes. It runs inside the OpenSearch process > > > > >> < > > > > > > > > https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E > > > > >, > > > > >> with no sidecar. > > > > >> > > > > >> *The proposal.* A new module, solr/modules/query-insights, > modelled on > > > > >> that plugin: > > > > >> > > > > >> - *Off by default, two steps to turn on.* List the module in > > > > >> SOLR_MODULES > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing > is > > > > >> loaded. > > > > >> - *Small hook in core.* A SearchRequestListener interface, > called > > > at > > > > >> the end of SearchHandler > > > > >> < > > > > > > > > https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> request handling on the coordinator only (shard sub-requests > are > > > > skipped). > > > > >> With no listener registered, the cost is one isEmpty() check > per > > > > >> search. > > > > >> - *Recording.* Per node, a top-N per metric (latency, CPU) for > the > > > > >> current and previous clock-aligned window. Defaults follow > > > > OpenSearch: N=10 > > > > >> (max 100), 5-minute windows. A request that doesn't beat the > > > current > > > > Nth > > > > >> value costs a field read and nothing else. > > > > >> - *Records.* Collection, handler, params (redactable, > truncated at > > > 8 > > > > >> KB), hits, shard count, user, latency, CPU, and the trace id > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > > > >. > > > > >> The trace id is something OpenSearch doesn't have: it links > each > > > > entry to > > > > >> the request log and to traces. > > > > >> - *API.* GET > /api/node/query-insights/top-queries?type=latency|cpu& > > > > >> nodes=all, needs the metrics-read > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> permission. Merges across nodes and lists any node that didn't > > > > answer. > > > > >> - *Settings.* System properties for defaults, > ext.queryInsights.* > > > > cluster > > > > >> properties > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> for runtime changes. > > > > >> - *Not in the first PR.* Grouping similar queries by shape > > > > >> < > > > > > > > > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > > > >, > > > > >> memory as a metric, and stored history. History would come > later as > > > > a Time > > > > >> Routed Alias > > > > >> < > > > > > > > > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> with autoDeleteAge, which gives us retention without a custom > > > cleanup > > > > >> job. > > > > >> > > > > >> *Why not tracing?* Tracing and this work together rather than > compete. > > > > >> Tracing answers "what happened inside this request?" once you know > > > which > > > > >> one to look at. This answers "which requests should I look at?" > > > without > > > > a > > > > >> tracing backend, and the trace id in each record takes you > straight to > > > > the > > > > >> trace. > > > > >> > > > > >> *Questions for you:* > > > > >> > > > > >> 1. Module plus a small core listener, or would you rather see > this > > > in > > > > >> core? > > > > >> 2. Is metrics-read the right permission for data that includes > > > query > > > > >> text, given the redaction option? > > > > >> 3. Anyone already working on something similar? I found > SOLR-1101 > > > > >> < > > > > > > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E > > > > >, > > > > >> SOLR-11994 > > > > >> < > > > > > > > > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E > > > > > > > > > >> and Eric's User Behaviour Insights PR (#2452) > > > > >> < > > > > > > > > https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E > > > > >. > > > > >> I'd like to line up with UBI when we get to stored history. > > > > >> > > > > >> I've written up a fuller design covering the core hook, recording, > > > > >> settings, security and tests, and I'll share it on the JIRA. > Feedback > > > on > > > > >> the direction here first would help me get it right before I > start. > > > > >> > > > > >> Thanks, > > > > >> Amrit > > > > >> > > > > > > > > > > > > > > > > I'm actually hoping that Amrit thinks bigger and more holisticially about > the problem of running Solr in a fixed resource environment where load goes > up and down. > > We're seeing more and more delegation to agents to craft the queries our > search engines respond to. At one of our clients the load goes up and down > very spikely, and it's driven by the whims of the agents. Those agents are > built by teams outside of search, and very much have a "move fast and break > things" philosopy. They are delegating to external customers access to > MCP servers that then talk to search! And tehn we wonder why search falls > over? Especially because the elasticity of the server resources is never > quite what we want. > > Query Insights is something I'm familiar with, it was one of the first > plugins on OpenSearch that I worked on, and indeed, the User Behavior > Insights plugin (and standard!) was named that to match up with the Query > Insights name. > > Here is the thing. Query Insights is only half the battle. And > honestly, half the battle was what OpenSearch fought. It missed/never > did, the second half of the battle. I hope Solr does. Once you have > near real time understanding of query performance, then we can start > thinking about how to protect the cluster when we start seeing problems. > Query Insights can help us understand what aspects of our queries are > causing the most load, and help us dial things back when the cluster is > under pressure. I would rather turn off vectors than have my entire > cluster crash. I would rather have a simple query run then a rich query > over having no query run. I would rather disable certain capablities > temporarily if an Agent is just crushing my server... "I know you asked > for X,Y, and Z, but you are killing me, so I ran your query just with > Y"... > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
