On Thu, Oct 29, 2026 at 4:32 PM Amrit Sarkar <[email protected]> wrote:
> Hi David, > > Fair challenge, and you're right that a tracing backend can sort by > duration. I've worked on distributed tracing for the last two years, and > the catch I keep running into is that *it can only rank what it was sent.* > > Where tracing falls short here: > > - *Sampling decides before anyone knows the request is slow.* Our tracing > page > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html%23sampling&source=gmail&ust=1793392173181000&sa=E> > says tracing every request is rarely affordable and shows a 10% ratio, and > Solr follows the caller's decision > > <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/modules/opentelemetry/src/java/org/apache/solr/opentelemetry/OtelTracerConfigurator.java%23L68&source=gmail&ust=1793392173181000&sa=E>. > If the gateway samples 1%, a bad query shape can hurt the cluster for > minutes and leave no trace at all. > > True (by default)... so configure OTEL_TRACES_SAMPLER=always_on, ignoring the caller's sampling decision (if the caller sent a trace ID). Granted, I don't pretend to think the volume & cost works for everyone. I believe we're doing this where I work but I couldn't confirm it before sending this. I think we could consider shipping Solr with a small opt-in OTEL Sampler impl that makes a "drop" decision for any non-"server" span. The idea is, in production, you decide you only want one span for each request a Solr node receives. No "client" spans (somewhat redundant with a paired "server" recipient) and no "internal" spans, which are likely to grow more numerous as we add more spans. I've heard that some projects have developed a custom log-level like configuration for spans, making it easy to choose the desired detail level at deployment time. > > - *Tail sampling costs full volume.* Every span gets exported, and the > collectors > need each whole trace on one instance > > <https://www.google.com/url?q=https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor%23scaling-collectors-with-the-tail-sampling-processor&source=gmail&ust=1793392173181000&sa=E>, > held in memory until they decide. You ship 100% of traffic to keep the slow > 1%. > - *A search span is thin.* One span per request, no CPU time, no hit > count. http.params only comes from the URL query string > > <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/tracing/TraceUtils.java%23L159-L161&source=gmail&ust=1793392173181000&sa=E>, > so POSTed and JSON DSL > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/query-guide/json-query-dsl.html&source=gmail&ust=1793392173181000&sa=E> > queries never reach it. > > Is this point a complaint that Solr's spans currently lack valuable information? I agree; we can work on fixing that! I see you have volunteered -- wonderful; I'm eager to help as well on all-things tracing. > > - *Duration points at the victim.* On a saturated node, the longest > traces are the cheap queries that waited, not the one burning CPU. > - *The slow request log > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/configuring-logging.html%23logging-slow-queries&source=gmail&ust=1793392173181000&sa=E> > has the same blind spot.* One QTime threshold, a line per core on each > node, no CPU ranking. > > What Query Insights adds: > > - *Every request, ranked.* Top N by latency and by CPU (from > ThreadCpuTimer > > <https://www.google.com/url?q=https://github.com/apache/solr/blob/0d6b2e32f59/solr/core/src/java/org/apache/solr/util/ThreadCpuTimer.java&source=gmail&ust=1793392173181000&sa=E>) > per time window. One call, GET > /api/node/query-insights/top-queries?type=cpu&nodes=all, merges every node > and lists any that didn't answer. > - *Cheap and bounded.* About 0.35 MB per node at defaults, no > background threads. A request that doesn't make the list costs one > comparison. > - *Works with tracing.* Each record carries the trace ID > > <https://www.google.com/url?q=https://solr.apache.org/guide/solr/latest/deployment-guide/distributed-tracing.html&source=gmail&ust=1793392173181000&sa=E>, > so the list tells you which traces to open. > - *Little to maintain.* Two lines in solr.in.sh > > <https://www.google.com/url?q=http://solr.in.sh&source=gmail&ust=1793392173181000&sa=E> > turn it on. In Solr, it's a small listener in SearchHandler (#5238 > > <https://www.google.com/url?q=https://github.com/apache/solr/pull/5238&source=gmail&ust=1793392173181000&sa=E>) > plus a module that's off by default. > > Okay. I'll review the code to assess it myself. > > > On making tracing better, agreed, and I'll take two pieces: > > - A JIRA to add CPU time and hit count to the search span. > - SOLR-13732 > > <https://www.google.com/url?q=https://issues.apache.org/jira/browse/SOLR-13732&source=gmail&ust=1793392173181000&sa=E>, > Dat's issue from 2019: always trace requests that carry a configured > header. It pairs well with this. Find the query in Query Insights, replay > it with the header, and you get its full trace even at 1% sampling. > > Awesome. I'm hoping we can augment the server search span with interesting info without adding bespoke code just for tracing to accomplish it. In my view, we work on adding more response header info to Solr, and then we generically add all response headers as span attributes. This is more broadly useful (caller sees the info) and easier to maintain. We can discuss variations on this; there are alternative ways to conceptualize achieving the low-maintenance goal.
