Resending, the first copy seems stuck in moderation. Hi all,
I've been away from Solr for a while. Coming back, I spent some time digging into the observability side, both what Solr exposes today and how other search engines handle it, to rebuild my understanding before proposing anything. One gap stood out, and I'd like to pick up SOLR-11994 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415632000&sa=E> ("Expose a slow query API", open since 2018) to close it. I want to check the shape with you before writing code. *The problem.* When someone asks "which queries are hurting this cluster right now?", Solr has no direct answer: - slowQueryThresholdMillis <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1791911415632000&sa=E> writes to solr_slow_requests.log, which you have to ship somewhere and grep. - Request handler metrics <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/performance-statistics-reference.html%23search-request-handler&source=gmail&ust=1791911415632000&sa=E> give latency per handler, not per query. - Tracing <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E> shows individual requests, if you run a backend and sample the right ones. - /tasks/list <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/task-management.html%23listalltasks&source=gmail&ust=1791911415633000&sa=E> shows only queries that are still running, and only those sent with canCancel=true. None of them gives a ranked list of the most expensive queries. OpenSearch solved this with its Query Insights <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/index/&source=gmail&ust=1791911415633000&sa=E> plugin (2.12+). Each node keeps an in-memory top-N of the searches it coordinated <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, ranked by latency, CPU or memory, over a rolling window. One API call returns them from all nodes. It runs inside the OpenSearch process <https://www.google.com/url?q=https://github.com/opensearch-project/query-insights/blob/5be9003098e482345f65fe2c00197ea5b323a940/src/main/java/org/opensearch/plugin/insights/QueryInsightsPlugin.java&source=gmail&ust=1791911415633000&sa=E>, with no sidecar. *The proposal.* A new module, solr/modules/query-insights, modelled on that plugin: - *Off by default, two steps to turn on.* List the module in SOLR_MODULES <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/configuration-guide/solr-modules.html&source=gmail&ust=1791911415633000&sa=E> and set SOLR_QUERYINSIGHTS_ENABLED=true. Without both, nothing is loaded. - *Small hook in core.* A SearchRequestListener interface, called at the end of SearchHandler <https://www.google.com/url?q=https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java&source=gmail&ust=1791911415633000&sa=E> request handling on the coordinator only (shard sub-requests are skipped). With no listener registered, the cost is one isEmpty() check per search. - *Recording.* Per node, a top-N per metric (latency, CPU) for the current and previous clock-aligned window. Defaults follow OpenSearch: N=10 (max 100), 5-minute windows. A request that doesn't beat the current Nth value costs a field read and nothing else. - *Records.* Collection, handler, params (redactable, truncated at 8 KB), hits, shard count, user, latency, CPU, and the trace id <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1791911415633000&sa=E>. The trace id is something OpenSearch doesn't have: it links each entry to the request log and to traces. - *API.* GET /api/node/query-insights/top-queries?type=latency|cpu&nodes=all, needs the metrics-read <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/rule-based-authorization-plugin.html%23predefined-permissions&source=gmail&ust=1791911415633000&sa=E> permission. Merges across nodes and lists any node that didn't answer. - *Settings.* System properties for defaults, ext.queryInsights.* cluster properties <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/cluster-node-management.html%23clusterprop&source=gmail&ust=1791911415633000&sa=E> for runtime changes. - *Not in the first PR.* Grouping similar queries by shape <https://www.google.com/url?q=https://docs.opensearch.org/latest/observing-your-data/query-insights/grouping-top-n-queries/&source=gmail&ust=1791911415633000&sa=E>, memory as a metric, and stored history. History would come later as a Time Routed Alias <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/aliases.html%23time-routed-aliases&source=gmail&ust=1791911415633000&sa=E> with autoDeleteAge, which gives us retention without a custom cleanup job. *Why not tracing?* Tracing and this work together rather than compete. Tracing answers "what happened inside this request?" once you know which one to look at. This answers "which requests should I look at?" without a tracing backend, and the trace id in each record takes you straight to the trace. *Questions for you:* 1. Module plus a small core listener, or would you rather see this in core? 2. Is metrics-read the right permission for data that includes query text, given the redaction option? 3. Anyone already working on something similar? I found SOLR-1101 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-1101&source=gmail&ust=1791911415633000&sa=E>, SOLR-11994 <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-11994&source=gmail&ust=1791911415633000&sa=E> and Eric's User Behaviour Insights PR (#2452) <https://www.google.com/url?q=https://github.com/apache/solr/pull/2452&source=gmail&ust=1791911415633000&sa=E>. I'd like to line up with UBI when we get to stored history. I've written up a fuller design covering the core hook, recording, settings, security and tests, and I'll share it on the JIRA. Feedback on the direction here first would help me get it right before I start. Thanks, Amrit
