Hi Eric, Thanks, that's an interesting idea. I hadn't thought about it that way.
It's further out than what I'm planning for the first version, which is just seeing the slow queries. I want to understand the vector controls and the load side better before I say much. But I can see this becoming the framework for it over time. Would love to hear Alessandro's thoughts too. Amrit On Tue, Oct 13, 2026 03:23 PM, Eric Pugh via dev <[email protected]> wrote: > I would love to not just see the query insights, but be able to use them > to shape things. > > I pitched Alessandro earlier today after his talk on new dense vector > controls. Many of the controls are query time settings that let you trade > speed for quality. > > It would be interesting to see a vision of where Solr could trade quality > for query speed based on the overall load on the cluster. Could Query > insights help us understand overall performance of the cluster and > potentially let us decide to have simpler faster queries till cluster > performance returned? > > Maybe that is all a bit further away then what you are thinking about > Query Insights as a tool for understanding your slow queries? > > > > On Oct 12, 2026, at 6:10 PM, Amrit Sarkar <[email protected]> > wrote: > > > > Resending, the first copy seems stuck in moderation. > > > > Hi all, > > > > I've been away from Solr for a while. Coming back, I spent some time > > digging into the observability side, both what Solr exposes today and how > > other search engines handle it, to rebuild my understanding before > > proposing anything. One gap stood out, and I'd like to pick up SOLR-11994 > > <https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E < > https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E>> > > ("Expose a slow query API", open since 2018) to close it. I want to check > > the shape with you before writing code. > > > > *The problem.* When someone asks "which queries are hurting this cluster > > right now?", Solr has no direct answer: > > > > - slowQueryThresholdMillis > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/configuring-logging.html%23logg > ing-slow-queries&source=gmail&ust=1791911415632000&sa=E < > https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/configuring-logging.html%23logg > ing-slow-queries&source=gmail&ust=1791911415632000&sa=E>> > > writes to solr_slow_requests.log, which you have to ship somewhere and > > grep. > > - Request handler metrics > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/performance-statistics-referenc > e.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E < > https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/performance-statistics-referenc > e.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E>> > > give latency per handler, not per query. > > - Tracing > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/distributed-tracing.html&source > =gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://solr.apache.org/guide/solr/latest/deployment-guide/d > istributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>> > > shows individual requests, if you run a backend and sample the right > ones. > > - /tasks/list > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/task-management.html%23listallt > asks&source=gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://solr.apache.org/guide/solr/latest/deployment-guide/t > ask-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E > >> > > shows only queries that are still running, and only those sent with > > canCancel=true. > > > > None of them gives a ranked list of the most expensive queries. > > > > OpenSearch solved this with its Query Insights > > <https://www.google.com/url?q=https://docs.opensearch.org/la > test/observing-your-data/query-insights/index/&source=gmail& > ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://docs.opensearch.org/latest/observing-your-data/query > -insights/index/&source=gmail&ust=1791911415633000&sa=E>> > > plugin (2.12+). Each node keeps an in-memory top-N of the searches it > > coordinated > > <https://www.google.com/url?q=https://docs.opensearch.org/la > test/observing-your-data/query-insights/top-n-queries/&sourc > e=gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://docs.opensearch.org/latest/observing-your-data/query > -insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E>>, > > ranked by latency, CPU or memory, over a rolling window. One API call > > returns them from all nodes. It runs inside the OpenSearch process > > <https://www.google.com/url?q=https://github.com/opensearch- > project/query-insights/blob/5be9003098e482345f65fe2c00197ea5 > b323a940/src/main/java/org/opensearch/plugin/insights/QueryI > nsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://github.com/opensearch- > project/query-insights/blob/5be9003098e482345f65fe2c00197ea5 > b323a940/src/main/java/org/opensearch/plugin/insights/QueryI > nsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E>>, > > with no sidecar. > > > > *The proposal.* A new module, solr/modules/query-insights, modelled on > that > > plugin: > > > > - *Off by default, two steps to turn on.* List the module in SOLR_MODULES > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/configuration-guide/solr-modules.html&source= > gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://solr.apache.org/guide/solr/latest/configuration-guid > e/solr-modules.html&source=gmail&ust=1791911415633000&sa=E>> > > and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is > loaded. > > - *Small hook in core.* A SearchRequestListener interface, called at the > > end of SearchHandler > > <https://www.google.com/url?q=https://github.com/apache/solr > /blob/main/solr/core/src/java/org/apache/solr/handler/compon > ent/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://github.com/apache/solr > /blob/main/solr/core/src/java/org/apache/solr/handler/compon > ent/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E>> > > request handling on the coordinator only (shard sub-requests are > skipped). > > With no listener registered, the cost is one isEmpty() check per search. > > - *Recording.* Per node, a top-N per metric (latency, CPU) for the > > current and previous clock-aligned window. Defaults follow OpenSearch: > N=10 > > (max 100), 5-minute windows. A request that doesn't beat the current Nth > > value costs a field read and nothing else. > > - *Records.* Collection, handler, params (redactable, truncated at 8 > > KB), hits, shard count, user, latency, CPU, and the trace id > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/distributed-tracing.html&source > =gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://solr.apache.org/guide/solr/latest/deployment-guide/d > istributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>>. > > The trace id is something OpenSearch doesn't have: it links each entry to > > the request log and to traces. > > - *API.* GET > > /api/node/query-insights/top-queries?type=latency|cpu&nodes=all, needs > > the metrics-read > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/rule-based-authorization-plugin > .html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/rule-based-authorization-plugin > .html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E>> > > permission. Merges across nodes and lists any node that didn't answer. > > - *Settings.* System properties for defaults, ext.queryInsights.* cluster > > properties > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/cluster-node-management.html%23 > clusterprop&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/cluster-node-management.html%23 > clusterprop&source=gmail&ust=1791911415633000&sa=E>> > > for runtime changes. > > - *Not in the first PR.* Grouping similar queries by shape > > <https://www.google.com/url?q=https://docs.opensearch.org/la > test/observing-your-data/query-insights/grouping-top-n-queri > es/&source=gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://docs.opensearch.org/latest/observing-your-data/query > -insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E > >>, > > memory as a metric, and stored history. History would come later as a > Time > > Routed Alias > > <https://www.google.com/url?q=https://solr.apache.org/guide/ > solr/latest/deployment-guide/aliases.html%23time-routed-alia > ses&source=gmail&ust=1791911415633000&sa=E <https://www.google.com/url?q= > https://solr.apache.org/guide/solr/latest/deployment-guide/a > liases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E>> > > with autoDeleteAge, which gives us retention without a custom cleanup > > job. > > > > *Why not tracing?* Tracing and this work together rather than compete. > > Tracing answers "what happened inside this request?" once you know which > > one to look at. This answers "which requests should I look at?" without a > > tracing backend, and the trace id in each record takes you straight to > the > > trace. > > > > *Questions for you:* > > > > 1. Module plus a small core listener, or would you rather see this in > > core? > > 2. Is metrics-read the right permission for data that includes query > > text, given the redaction option? > > 3. Anyone already working on something similar? I found SOLR-1101 > > <https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E>>, > > SOLR-11994 > > <https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://issues.apache.org/jira > /browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E>> > > and Eric's User Behaviour Insights PR (#2452) > > <https://www.google.com/url?q=https://github.com/apache/solr > /pull/2452&source=gmail&ust=1791911415633000&sa=E < > https://www.google.com/url?q=https://github.com/apache/solr > /pull/2452&source=gmail&ust=1791911415633000&sa=E>>. > > I'd like to line up with UBI when we get to stored history. > > > > I've written up a fuller design covering the core hook, recording, > > settings, security and tests, and I'll share it on the JIRA. Feedback on > > the direction here first would help me get it right before I start. > > > > Thanks, > > Amrit > > Disclaimer > > The information contained in this communication from the sender is > confidential. It is intended solely for use by the recipient and others > authorized to receive it. If you are not the recipient, you are hereby > notified that any disclosure, copying, distribution or taking action in > relation of the contents of this information is strictly prohibited and may > be unlawful. > > This email has been scanned for viruses and malware, and may have been > automatically archived by Mimecast, a leader in email security and cyber > resilience. Mimecast integrates email defenses with brand protection, > security awareness training, web security, compliance and other essential > capabilities. Mimecast helps protect large and small organizations from > malicious activity, human error and technology failure; and to lead the > movement toward building a more resilient world. To find out more, visit > our website. >
