I would love to not just see the query insights, but be able to use them to shape things.
I pitched Alessandro earlier today after his talk on new dense vector controls. Many of the controls are query time settings that let you trade speed for quality. It would be interesting to see a vision of where Solr could trade quality for query speed based on the overall load on the cluster. Could Query insights help us understand overall performance of the cluster and potentially let us decide to have simpler faster queries till cluster performance returned? Maybe that is all a bit further away then what you are thinking about Query Insights as a tool for understanding your slow queries? > On Oct 12, 2026, at 6:10 PM, Amrit Sarkar <[email protected]> wrote: > > Resending, the first copy seems stuck in moderation. > > Hi all, > > I've been away from Solr for a while. Coming back, I spent some time > digging into the observability side, both what Solr exposes today and how > other search engines handle it, to rebuild my understanding before > proposing anything. One gap stood out, and I'd like to pick up SOLR-11994 > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E>> > ("Expose a slow query API", open since 2018) to close it. I want to check > the shape with you before writing code. > > *The problem.* When someone asks "which queries are hurting this cluster > right now?", Solr has no direct answer: > > - slowQueryThresholdMillis > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E>> > writes to solr_slow_requests.log, which you have to ship somewhere and > grep. > - Request handler metrics > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E>> > give latency per handler, not per query. > - Tracing > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>> > shows individual requests, if you run a backend and sample the right ones. > - /tasks/list > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E>> > shows only queries that are still running, and only those sent with > canCancel=true. > > None of them gives a ranked list of the most expensive queries. > > OpenSearch solved this with its Query Insights > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E>> > plugin (2.12+). Each node keeps an in-memory top-N of the searches it > coordinated > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E>>, > ranked by latency, CPU or memory, over a rolling window. One API call > returns them from all nodes. It runs inside the OpenSearch process > <https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E>>, > with no sidecar. > > *The proposal.* A new module, solr/modules/query-insights, modelled on that > plugin: > > - *Off by default, two steps to turn on.* List the module in SOLR_MODULES > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E>> > and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is loaded. > - *Small hook in core.* A SearchRequestListener interface, called at the > end of SearchHandler > <https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E>> > request handling on the coordinator only (shard sub-requests are skipped). > With no listener registered, the cost is one isEmpty() check per search. > - *Recording.* Per node, a top-N per metric (latency, CPU) for the > current and previous clock-aligned window. Defaults follow OpenSearch: N=10 > (max 100), 5-minute windows. A request that doesn't beat the current Nth > value costs a field read and nothing else. > - *Records.* Collection, handler, params (redactable, truncated at 8 > KB), hits, shard count, user, latency, CPU, and the trace id > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>>. > The trace id is something OpenSearch doesn't have: it links each entry to > the request log and to traces. > - *API.* GET > /api/node/query-insights/top-queries?type=latency|cpu&nodes=all, needs > the metrics-read > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E>> > permission. Merges across nodes and lists any node that didn't answer. > - *Settings.* System properties for defaults, ext.queryInsights.* cluster > properties > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E>> > for runtime changes. > - *Not in the first PR.* Grouping similar queries by shape > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E>>, > memory as a metric, and stored history. History would come later as a Time > Routed Alias > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E>> > with autoDeleteAge, which gives us retention without a custom cleanup > job. > > *Why not tracing?* Tracing and this work together rather than compete. > Tracing answers "what happened inside this request?" once you know which > one to look at. This answers "which requests should I look at?" without a > tracing backend, and the trace id in each record takes you straight to the > trace. > > *Questions for you:* > > 1. Module plus a small core listener, or would you rather see this in > core? > 2. Is metrics-read the right permission for data that includes query > text, given the redaction option? > 3. Anyone already working on something similar? I found SOLR-1101 > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E>>, > SOLR-11994 > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E>> > and Eric's User Behaviour Insights PR (#2452) > <https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E > > <https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E>>. > I'd like to line up with UBI when we get to stored history. > > I've written up a fuller design covering the core hook, recording, > settings, security and tests, and I'll share it on the JIRA. Feedback on > the direction here first would help me get it right before I start. > > Thanks, > Amrit Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.
