On 2026/10/29 20:31:43 Amrit Sarkar wrote:
> Hi David,
>
> Fair challenge, and you're right that a tracing backend can sort by
> duration. I've worked on distributed tracing for the last two years, and
> the catch I keep running into is that *it can only rank what it was sent.*
>
> Where tracing falls short here:
>
> - *Sampling decides before anyone knows the request is slow.* Our tracing
> page
>
> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html%23sampling&source=gmail&ust=1793392173181000&sa=E>
> says tracing every request is rarely affordable and shows a 10% ratio, and
> Solr follows the caller's decision
>
> <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java%23L68&source=gmail&ust=1793392173181000&sa=E>.
> If the gateway samples 1%, a bad query shape can hurt the cluster for
> minutes and leave no trace at all.
> - *Tail sampling costs full volume.* Every span gets exported, and
> the collectors
> need each whole trace on one instance
>
> <https://www.google.com/url?q=https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor%23scaling-collectors-with-the-tail-sampling-processor&source=gmail&ust=1793392173181000&sa=E>,
> held in memory until they decide. You ship 100% of traffic to keep the slow
> 1%.
> - *A search span is thin.* One span per request, no CPU time, no hit
> count. http.params only comes from the URL query string
>
> <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/tracing/TraceUtils.java%23L159-L161&source=gmail&ust=1793392173181000&sa=E>,
> so POSTed and JSON DSL
>
> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/query-guide/json-query-dsl.html&source=gmail&ust=1793392173181000&sa=E>
> queries never reach it.
> - *Duration points at the victim.* On a saturated node, the longest
> traces are the cheap queries that waited, not the one burning CPU.
> - *The slow request log
>
> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1793392173181000&sa=E>
> has the same blind spot.* One QTime threshold, a line per core on each
> node, no CPU ranking.
>
> What Query Insights adds:
>
> - *Every request, ranked.* Top N by latency and by CPU (from
> ThreadCpuTimer
>
> <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/ThreadCpuTimer.java&source=gmail&ust=1793392173181000&sa=E>)
> per time window. One call, GET
> /api/node/query-insights/top-queries?type=cpu&nodes=all, merges every node
> and lists any that didn't answer.
> - *Cheap and bounded.* About 0.35 MB per node at defaults, no background
> threads. A request that doesn't make the list costs one comparison.
> - *Works with tracing.* Each record carries the trace ID
>
> <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1793392173181000&sa=E>,
> so the list tells you which traces to open.
> - *Little to maintain.* Two lines in solr.in.sh
>
> <https://www.google.com/url?q=http://solr.in.sh&source=gmail&ust=1793392173181000&sa=E>
> turn it on. In Solr, it's a small listener in SearchHandler (#5238
>
> <https://www.google.com/url?q=https://github.com/apache/solr/pull/5238&source=gmail&ust=1793392173181000&sa=E>)
> plus a module that's off by default.
>
> On making tracing better, agreed, and I'll take two pieces:
>
> - A JIRA to add CPU time and hit count to the search span.
> - SOLR-13732
>
> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-13732&source=gmail&ust=1793392173181000&sa=E>,
> Dat's issue from 2019: always trace requests that carry a configured
> header. It pairs well with this. Find the query in Query Insights, replay
> it with the header, and you get its full trace even at 1% sampling.
>
> Unless you see a hole in this, I'll keep going with SOLR-18571
> <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1793392173181000&sa=E>
> .
>
> Thanks,
> Amrit
>
> On Thu, Oct 29, 2026 12:11 PM, David Smiley <[email protected]> wrote:
>
> > Please tell me what this solves that distributed tracing (e.g. via OTEL
> > standards) does not solve. I'm aware distributed tracing implies
> > additional/other infrastructure, but that's becoming increasingly common.
> > And it's stuff we/Solr don't have to maintain. Of course we should do more
> > to make tracing more awesome (more spans, more detailed spans).
> >
> > > None of them gives a ranked list of the most expensive queries.
> >
> > Of course tracing can do that!
> >
> > On Thu, Oct 22, 2026 at 8:31 PM Amrit Sarkar <[email protected]>
> > wrote:
> >
> > > Hi all,
> > >
> > > No objections came in, so I've opened the umbrella: SOLR-18571
> > > <
> > >
> > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-18571&source=gmail&ust=1792801902707000&sa=E
> > > >,
> > > with the full design attached and the five sub-tasks from my last mail.
> > I'm
> > > starting on SOLR-11994
> > > <
> > >
> > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1792801902707000&sa=E
> > > >,
> > > the core hook.
> > >
> > > Feedback is still welcome on the umbrella or on any PR as they come up.
> > >
> > > Thanks,
> > > Amrit
> > >
> > > On Sun, Oct 18, 2026 09:53 AM, Amrit Sarkar <[email protected]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Bumping this before I start on the code. The only reply so far was
> > Eric's
> > > > idea about load-based trade-offs
> > > > <https://lists.apache.org/thread/2lvrz5qozkwbxv5j52jon66zms4hbwpp>.
> > That
> > > > stays out of this proposal, but the two connect; more on that below.
> > > Nobody
> > > > has raised concerns about the plan itself, and I want to be sure that
> > > > silence means agreement and not that people missed it.
> > > >
> > > > *The use case I didn't cover.* A lot of teams at my workplace now
> > > > diagnose production issues with agents: Slack bots and Claude-style
> > > agents
> > > > that call observability tools through MCP
> > > > <https://modelcontextprotocol.io/docs/2026-07-28/getting-started/intro
> > >
> > > and
> > > > CLI. Someone asks, "Why was the ingress service slow at 14:05?" and the
> > > > agent pulls traces, metrics and logs.
> > > >
> > > > For Solr, the only source today is solr_slow_requests.log
> > > > <
> > >
> > https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html#logging-slow-queries
> > > >
> > > > shipped to a log store. The agent can query that through something like
> > > the Splunk
> > > > MCP server
> > > > <
> > >
> > https://help.splunk.com/en/splunk-enterprise/mcp-server-for-splunk-platform/2.0/mcp-server-tools
> > > >,
> > > > but its search tool gives up after one minute or 1,000 events, and the
> > > > lines are only there if they fit in your daily license volume
> > > > <
> > >
> > https://help.splunk.com/en/splunk-enterprise/administer/admin-manual/10.0/configure-splunk-licenses/how-splunk-enterprise-licensing-works
> > > >.
> > > > The agent ends up scanning raw log lines to rebuild something Solr
> > > already
> > > > had in memory when the request finished.
> > > >
> > > > *How this API closes the gap*, without Solr shipping anything
> > > > agent-specific:
> > > >
> > > > - Applications can already pass their own request ID to Solr through
> > > > the X-Trace-Id header
> > > > <
> > >
> > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#always-on-trace-id-generation
> > > >
> > > > (solr.traceIdHeader), or traceparent when the OpenTelemetry module
> > > > <
> > >
> > https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html#open-telemetry-module
> > > >
> > > > is on. Solr uses it as the trace ID, and every record carries it.
> > > > - An agent can go from the application's request ID to the Solr
> > query,
> > > > its latency, CPU, hits and shard count in one call, straight from
> > the
> > > > cluster.
> > > > - It's a v2 API, so it ends up in the generated OpenAPI spec and
> > SolrJ
> > > > client <https://github.com/apache/solr/blob/main/dev-docs/apis.adoc
> > >.
> > > > Wrapping it as an MCP tool is a few lines in whatever MCP server a
> > > team
> > > > already runs. Solr itself stays out of MCP.
> > > >
> > > > To make that lookup cheap, I'm adding optional collection and traceId
> > > > filters to GET /api/node/query-insights/top-queries. Everything else
> > > > stays as described in the first mail
> > > > <https://lists.apache.org/thread/t7mft9dbntgtp8pjfh3zvomxh8zdobtd>.
> > > >
> > > > How this ties into Eric's idea. Shedding LTR or vector work under load
> > > needs
> > > > to know which queries are expensive and what they ran. So each record
> > > > will also note the costly features used (LTR, knn, grouping, faceting,
> > > > paramset). The load control itself is a separate discussion.
> > > >
> > > > *How I'll deliver it.* One umbrella JIRA for Query Insights, then small
> > > > sub-tasks, each its own PR that can be reviewed and merged on its own:
> > > >
> > > > 1. SOLR-11994 <https://issues.apache.org/jira/browse/SOLR-11994>:
> > the
> > > > SearchRequestListener hook in core. No behaviour change when nothing
> > > > is registered.
> > > > 2. The query-insights module: per-node top-N by latency and CPU, the
> > > > features each query used, plus the node-local API.
> > > > 3. Cluster-wide view: nodes=all merging, with nodes that didn't
> > answer
> > > > listed.
> > > > 4. collection and traceId filters, and runtime settings through
> > > > cluster properties.
> > > > 5. Ref guide page.
> > > >
> > > > Stored history and grouping get their own sub-tasks later, after the
> > > first
> > > > five land.
> > > >
> > > > *Two direct questions:*
> > > >
> > > > 1. Any objections to the plan as described?
> > > > 2. Do you think this is a good fit for Solr at all? If you have
> > > > doubts, I'd really value hearing them before I start. I'd rather
> > > adjust the
> > > > idea now than spend reviewers' time on something the project doesn't
> > > want.
> > > >
> > > > If nothing comes up by Wednesday, Oct 21, I'll create the umbrella and
> > > > the sub-tasks, post the full design there, and start with the core
> > hook.
> > > > I'm not a committer, so I'll also need someone to shepherd it. Eric,
> > > would
> > > > you be up for that?
> > > >
> > > > Thanks,
> > > > Amrit
> > > >
> > > > On Mon, Oct 12, 2026 10:10 AM, Amrit Sarkar <[email protected]>
> > > > wrote:
> > > >
> > > >> Resending, the first copy seems stuck in moderation.
> > > >>
> > > >> Hi all,
> > > >>
> > > >> I've been away from Solr for a while. Coming back, I spent some time
> > > >> digging into the observability side, both what Solr exposes today and
> > > how
> > > >> other search engines handle it, to rebuild my understanding before
> > > >> proposing anything. One gap stood out, and I'd like to pick up
> > > SOLR-11994
> > > >> <
> > >
> > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E
> > > >
> > > >> ("Expose a slow query API", open since 2018) to close it. I want to
> > > check
> > > >> the shape with you before writing code.
> > > >>
> > > >> *The problem.* When someone asks "which queries are hurting this
> > cluster
> > > >> right now?", Solr has no direct answer:
> > > >>
> > > >> - slowQueryThresholdMillis
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E
> > > >
> > > >> writes to solr_slow_requests.log, which you have to ship somewhere
> > > >> and grep.
> > > >> - Request handler metrics
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E
> > > >
> > > >> give latency per handler, not per query.
> > > >> - Tracing
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> shows individual requests, if you run a backend and sample the
> > right
> > > ones.
> > > >> - /tasks/list
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> shows only queries that are still running, and only those sent with
> > > >> canCancel=true.
> > > >>
> > > >> None of them gives a ranked list of the most expensive queries.
> > > >>
> > > >> OpenSearch solved this with its Query Insights
> > > >> <
> > >
> > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> plugin (2.12+). Each node keeps an in-memory top-N of the searches it
> > > >> coordinated
> > > >> <
> > >
> > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E
> > > >,
> > > >> ranked by latency, CPU or memory, over a rolling window. One API call
> > > >> returns them from all nodes. It runs inside the OpenSearch process
> > > >> <
> > >
> > https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E
> > > >,
> > > >> with no sidecar.
> > > >>
> > > >> *The proposal.* A new module, solr/modules/query-insights, modelled on
> > > >> that plugin:
> > > >>
> > > >> - *Off by default, two steps to turn on.* List the module in
> > > >> SOLR_MODULES
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is
> > > >> loaded.
> > > >> - *Small hook in core.* A SearchRequestListener interface, called
> > at
> > > >> the end of SearchHandler
> > > >> <
> > >
> > https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> request handling on the coordinator only (shard sub-requests are
> > > skipped).
> > > >> With no listener registered, the cost is one isEmpty() check per
> > > >> search.
> > > >> - *Recording.* Per node, a top-N per metric (latency, CPU) for the
> > > >> current and previous clock-aligned window. Defaults follow
> > > OpenSearch: N=10
> > > >> (max 100), 5-minute windows. A request that doesn't beat the
> > current
> > > Nth
> > > >> value costs a field read and nothing else.
> > > >> - *Records.* Collection, handler, params (redactable, truncated at
> > 8
> > > >> KB), hits, shard count, user, latency, CPU, and the trace id
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E
> > > >.
> > > >> The trace id is something OpenSearch doesn't have: it links each
> > > entry to
> > > >> the request log and to traces.
> > > >> - *API.* GET /api/node/query-insights/top-queries?type=latency|cpu&
> > > >> nodes=all, needs the metrics-read
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> permission. Merges across nodes and lists any node that didn't
> > > answer.
> > > >> - *Settings.* System properties for defaults, ext.queryInsights.*
> > > cluster
> > > >> properties
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> for runtime changes.
> > > >> - *Not in the first PR.* Grouping similar queries by shape
> > > >> <
> > >
> > https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E
> > > >,
> > > >> memory as a metric, and stored history. History would come later as
> > > a Time
> > > >> Routed Alias
> > > >> <
> > >
> > https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> with autoDeleteAge, which gives us retention without a custom
> > cleanup
> > > >> job.
> > > >>
> > > >> *Why not tracing?* Tracing and this work together rather than compete.
> > > >> Tracing answers "what happened inside this request?" once you know
> > which
> > > >> one to look at. This answers "which requests should I look at?"
> > without
> > > a
> > > >> tracing backend, and the trace id in each record takes you straight to
> > > the
> > > >> trace.
> > > >>
> > > >> *Questions for you:*
> > > >>
> > > >> 1. Module plus a small core listener, or would you rather see this
> > in
> > > >> core?
> > > >> 2. Is metrics-read the right permission for data that includes
> > query
> > > >> text, given the redaction option?
> > > >> 3. Anyone already working on something similar? I found SOLR-1101
> > > >> <
> > >
> > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E
> > > >,
> > > >> SOLR-11994
> > > >> <
> > >
> > https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E
> > > >
> > > >> and Eric's User Behaviour Insights PR (#2452)
> > > >> <
> > >
> > https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E
> > > >.
> > > >> I'd like to line up with UBI when we get to stored history.
> > > >>
> > > >> I've written up a fuller design covering the core hook, recording,
> > > >> settings, security and tests, and I'll share it on the JIRA. Feedback
> > on
> > > >> the direction here first would help me get it right before I start.
> > > >>
> > > >> Thanks,
> > > >> Amrit
> > > >>
> > > >
> > >
> >
>
I'm actually hoping that Amrit thinks bigger and more holisticially about the
problem of running Solr in a fixed resource environment where load goes up and
down.
We're seeing more and more delegation to agents to craft the queries our search
engines respond to. At one of our clients the load goes up and down very
spikely, and it's driven by the whims of the agents. Those agents are built by
teams outside of search, and very much have a "move fast and break things"
philosopy. They are delegating to external customers access to MCP servers
that then talk to search! And tehn we wonder why search falls over?
Especially because the elasticity of the server resources is never quite what
we want.
Query Insights is something I'm familiar with, it was one of the first plugins
on OpenSearch that I worked on, and indeed, the User Behavior Insights plugin
(and standard!) was named that to match up with the Query Insights name.
Here is the thing. Query Insights is only half the battle. And honestly,
half the battle was what OpenSearch fought. It missed/never did, the second
half of the battle. I hope Solr does. Once you have near real time
understanding of query performance, then we can start thinking about how to
protect the cluster when we start seeing problems. Query Insights can help us
understand what aspects of our queries are causing the most load, and help us
dial things back when the cluster is under pressure. I would rather turn off
vectors than have my entire cluster crash. I would rather have a simple query
run then a rich query over having no query run. I would rather disable
certain capablities temporarily if an Agent is just crushing my server... "I
know you asked for X,Y, and Z, but you are killing me, so I ran your query just
with Y"...
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
