This is an automated email from the ASF dual-hosted git repository.
epugh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/solr-mcp.git
The following commit(s) were added to refs/heads/main by this push:
new 997a57b9 docs: correct docker-compose auto-start guidance and expand
LGTM observability docs (#230)
997a57b9 is described below
commit 997a57b907a281707bdfa831680676ad42d22a38
Author: Aditya Parikh <[email protected]>
AuthorDate: Tue Oct 6 14:33:29 2026 -0400
docs: correct docker-compose auto-start guidance and expand LGTM
observability docs (#230)
* docs(dev): correct docker-compose auto-start guidance for bootRun
DEVELOPMENT.md and AGENTS.md told developers to run `docker compose up
-d` before `bootRun`, then contradicted that a few lines later by
saying Spring Boot Docker Compose starts the services automatically.
The developmentOnly `spring-boot-docker-compose` /
`spring-ai-spring-boot-docker-compose` starters mean `bootRun` really
does auto-start (and stop) compose.yaml's `solr` and `zoo` services,
so the manual step was redundant there. It's still required when
running the packaged JAR, Docker image, or native image directly,
since those artifacts exclude developmentOnly dependencies. The old
text also claimed LGTM starts automatically, which is wrong: it's
excluded via `org.springframework.boot.ignore: "true"` in
compose.yaml and always needs a manual `docker compose up -d lgtm`.
Signed-off-by: Aditya Parikh <[email protected]>
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: Aditya Parikh <[email protected]>
* docs(dev): scope docker-compose auto-start claim to HTTP mode only
Code review on PR #230 caught that the previous commit's claim —
"bootRun auto-starts compose.yaml" — was placed on the default STDIO
path, but application-stdio.properties sets
spring.docker.compose.enabled=false specifically to avoid startup
delay and container-lifecycle noise on the JSON-RPC transport. Only
application-http.properties enables it. Move the manual
`docker compose up -d` step back to the STDIO path (and the JAR path,
unchanged) and attach the auto-start explanation to HTTP mode only,
where it is actually true.
Signed-off-by: Aditya Parikh <[email protected]>
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: Aditya Parikh <[email protected]>
* docs: expand LGTM observability guidance with lifecycle notes and a
walkthrough
The observability docs described starting the LGTM stack with a plain
docker compose up -d but didn't call out that lgtm carries
org.springframework.boot.ignore: "true" in compose.yaml, so bootRun never
starts or stops it in any transport mode (unlike solr/zoo in HTTP mode).
Documents the manual docker compose up -d lgtm / stop lgtm commands and
the in-memory-only storage.
Also adds a runnable walkthrough (a curl against /mcp, or any MCP client)
so there's something to look at, a "Pivoting Between the Three" section
showing how a trace's "Logs for this span" link jumps to the correlated
Loki lines, and a troubleshooting table for the common no-data cases.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Signed-off-by: Aditya Parikh <[email protected]>
* docs(observability): describe the signals the server actually exports
Measured against grafana/otel-lgtm, the page promised three things the
http profile does not do:
- Metrics: only the OpenTelemetry HTTP server metric
(http_server_request_duration_seconds_*) is exported over OTLP. The
PromQL examples queried Micrometer names (http_server_requests_seconds_*,
jvm_memory_used_bytes), which exist only at /actuator/prometheus and
return nothing in Grafana. Replace them with queries that return data.
- Outgoing Solr calls are not traced as separate spans; their time is part
of the tool span (e.g. search-service#search).
- Trace-to-log linking works, but a successful tool call writes no log
lines, so "Logs for this span" is empty for it. Add a failing
check-health call whose WARN shows the link, a LogQL filter for
request-scoped lines, and troubleshooting rows for both cases.
Also note that OTEL_TRACES_URL sets the endpoint for all three signals and
how to reach a host LGTM from a container, and that traces and metrics
take up to a minute to appear.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Aditya Parikh <[email protected]>
---------
Signed-off-by: Aditya Parikh <[email protected]>
Co-authored-by: Claude Sonnet 5 <[email protected]>
---
AGENTS.md | 9 +-
dev-docs/DEVELOPMENT.md | 41 ++++++---
docs/observability.md | 132 ++++++++++++++++++++++++---
docs/site/content/pages/mcp/observability.md | 132 ++++++++++++++++++++++++---
4 files changed, 273 insertions(+), 41 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index cbe4cd91..275d25f2 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -43,8 +43,13 @@ Solr MCP Server is a Spring AI Model Context Protocol (MCP)
server that enables
./gradlew nativeCompile -Pnative # Compile native binary (host OS
only)
./gradlew nativeTest -Pnative # Run tests as native image
-# Run locally (requires `docker compose up -d` for Solr)
-./gradlew bootRun # STDIO mode (default)
+# Run locally
+# STDIO disables Docker Compose auto-start
(spring.docker.compose.enabled=false in
+# application-stdio.properties) — run `docker compose up -d` first, same as
for the
+# packaged jar/Docker/native image or to start the boot-ignored `lgtm` service.
+docker compose up -d && ./gradlew bootRun # STDIO mode (default)
+# HTTP mode enables it (application-http.properties), so bootRun
auto-starts/stops
+# compose.yaml's solr/zoo via the developmentOnly spring-boot-docker-compose
starter.
PROFILES=http ./gradlew bootRun # HTTP mode
```
diff --git a/dev-docs/DEVELOPMENT.md b/dev-docs/DEVELOPMENT.md
index 88b5afa6..2bd6eb92 100644
--- a/dev-docs/DEVELOPMENT.md
+++ b/dev-docs/DEVELOPMENT.md
@@ -95,27 +95,28 @@ grype sbom:application.cdx.json
## Running Locally
-### Start Solr
+### Run the Server
+
+#### STDIO Mode (Default)
+
+`application-stdio.properties` sets `spring.docker.compose.enabled=false`
(Docker Compose
+management is only useful for local HTTP development; in STDIO it would add
startup delay and
+unexpected container lifecycle noise on the JSON-RPC transport), so start Solr
by hand first:
```bash
docker compose up -d
+./gradlew bootRun
```
-This starts a Solr instance in SolrCloud mode with ZooKeeper and creates two
sample collections:
+This starts Solr in SolrCloud mode with ZooKeeper and creates two sample
collections:
- `books` - Created empty. The books.csv download and post are commented out in
`init-solr.sh`, so use it as a scratch collection or uncomment those lines.
- `films` - Collection populated with Solr's sample film data
-### Run the Server
-
-#### STDIO Mode (Default)
-
-```bash
-./gradlew bootRun
-```
-
-Or using the JAR:
+Or using the JAR — same manual step, since the `developmentOnly`
docker-compose starter isn't on
+the packaged jar's runtime classpath either:
```bash
+docker compose up -d
java -jar build/libs/solr-mcp-1.0.0-SNAPSHOT.jar
```
@@ -125,11 +126,25 @@ java -jar build/libs/solr-mcp-1.0.0-SNAPSHOT.jar
PROFILES=http ./gradlew bootRun
```
-Spring Boot Docker Compose will automatically start the services declared in
`compose.yaml`
-(Solr, ZooKeeper, and optionally LGTM for observability) before the
application starts.
+`application-http.properties` sets `spring.docker.compose.enabled=true`, so
this mode's
+`bootRun` picks up the `spring-boot-docker-compose` /
`spring-ai-spring-boot-docker-compose`
+`developmentOnly` dependencies and auto-starts (then stops) the `solr` and
`zoo` services
+declared in the root `compose.yaml` — no manual `docker compose up` needed
here. Running the
+built JAR/Docker/native image in HTTP mode still needs the manual step, since
those artifacts
+exclude `developmentOnly` dependencies.
The server will start on http://localhost:8080
+#### Observability (optional)
+
+The `lgtm` service in `compose.yaml` carries `org.springframework.boot.ignore:
"true"`, which
+opts it out of Spring Boot's Docker Compose lifecycle management. Start it by
hand if you want
+the Grafana/Loki/Tempo/Mimir stack, regardless of run mode:
+
+```bash
+docker compose up -d lgtm
+```
+
### Environment Variables
- `SOLR_URL`: Solr instance URL (default: `http://localhost:8983/solr/`)
diff --git a/docs/observability.md b/docs/observability.md
index b3c17671..42bdcb1c 100644
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -6,11 +6,17 @@ When running in **HTTP mode**, the Solr MCP Server exports
telemetry data via Op
| Signal | Backend | What it shows |
|--------|---------|---------------|
-| **Traces** | Tempo | Distributed traces for every MCP tool invocation, Solr
query, and HTTP request |
-| **Metrics** | Mimir/Prometheus | JVM stats, HTTP request rates, Solr query
latencies, cache hit ratios |
-| **Logs** | Loki | Structured application logs correlated with trace IDs |
+| **Traces** | Tempo | A trace per HTTP request, with a span for the MCP tool
it invoked |
+| **Metrics** | Mimir/Prometheus | HTTP server request count and duration |
+| **Logs** | Loki | Application logs, each tagged with the trace and span it
was written under |
-Every MCP tool invocation creates a trace span: search, indexing (JSON, CSV,
XML), collection operations (list, stats, health, create), and schema
retrieval. All incoming HTTP requests and outgoing Solr calls are automatically
traced.
+Every MCP tool invocation creates a span named after the service and tool,
such as
+`search-service#search` or `collection-service#check-health`, inside the trace
of the HTTP
+request that carried it. The call to Solr is not a separate span; its time is
part of the
+tool span.
+
+JVM, Tomcat and other Micrometer metrics are not exported over OTLP. They are
served for
+scraping at `/actuator/prometheus` (see **Actuator Endpoints** below).
***
@@ -32,6 +38,28 @@ This starts:
| OTLP gRPC | localhost:4317 | Trace/metric/log ingestion (gRPC) |
| OTLP HTTP | localhost:4318 | Trace/metric/log ingestion (HTTP) |
+**LGTM is never auto-started.** The `lgtm` service carries
+`org.springframework.boot.ignore: "true"` in `compose.yaml`, which opts it out
of Spring
+Boot's Docker Compose lifecycle management. This is independent of transport
mode: even
+in HTTP mode, where `PROFILES=http ./gradlew bootRun` auto-starts (and stops)
the `solr`
+and `zoo` services, LGTM is untouched. Start it explicitly, either with the
+`docker compose up -d` above (which starts everything) or, if `solr`/`zoo` are
already
+running via `bootRun`'s auto-start, on its own:
+
+```bash
+docker compose up -d lgtm
+```
+
+Because it's Boot-ignored, `bootRun` won't stop it either—bring it down
by hand
+when you're done:
+
+```bash
+docker compose stop lgtm
+```
+
+The `grafana/otel-lgtm` container stores everything in memory with no
persistent volume,
+so a restart discards all traces, metrics, and logs.
+
### Run the Server with Observability ###
```bash
@@ -46,6 +74,46 @@ otel.exporter.otlp.endpoint=http://localhost:4317
otel.exporter.otlp.protocol=grpc
```
+`otel.exporter.otlp.endpoint` is the endpoint for all three signals, so
`OTEL_TRACES_URL`
+moves logs and metrics too, despite its name. When the server runs in a
container and LGTM
+on the host, `localhost` is the container itself; set
+`OTEL_TRACES_URL=http://host.docker.internal:4317` (plus
+`--add-host=host.docker.internal:host-gateway` on Linux).
+
+### Generate Some Activity ###
+
+Grafana has nothing to show until at least one tool call runs. Any MCP client
works
+(the [Quick Start](tutorial.md) prompts are enough), or call the HTTP endpoint
directly.
+`HTTP_SECURITY_ENABLED=false` skips the OAuth2 setup for this local
check—see
+[Security](security/http.md) to keep it on:
+
+```bash
+HTTP_SECURITY_ENABLED=false PROFILES=http ./gradlew bootRun
+```
+
+```bash
+curl -s -X POST http://localhost:8080/mcp \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json, text/event-stream" \
+ -d
'{"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name":"list-collections","arguments":{}}}'
+```
+
+Run it a few times—each call is one trace and one more request in the
metrics.
+
+A successful call writes no log lines, so it has nothing to show in Loki. To
see a trace
+with a log attached, make one call that fails; a health check on a collection
that does not
+exist logs a `WARN` inside the request:
+
+```bash
+curl -s -X POST http://localhost:8080/mcp \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json, text/event-stream" \
+ -d
'{"jsonrpc":"2.0","method":"tools/call","id":2,"params":{"name":"check-health","arguments":{"collection":"no-such-collection"}}}'
+```
+
+Traces take up to a minute to become searchable in Tempo, and metrics are
exported once
+a minute, so an empty Grafana right after the calls is expected.
+
***
## Grafana ##
@@ -59,30 +127,53 @@ Open [http://localhost:3000](http://localhost:3000) and
click **Explore** in the
{.service.name="solr-mcp"}
-3. Click on a trace to see the span waterfall—each MCP tool invocation,
Solr query, and HTTP request is a separate span
+3. Click on an `http post /mcp` trace to see the span waterfall: the security
filter
+ chain, then one span for the tool (`search-service#search`,
+ `collection-service#check-health`, …) with its duration
### View Logs (Loki) ###
1. Select **Loki** as the data source
2. Use LogQL to search:
- {service_name="solr-mcp"} |= "search"
+ {service_name="solr-mcp"} | trace_id != ""
-3. Logs are automatically correlated with trace IDs—click a log line to
jump to its trace
+ That keeps only lines written during a request; drop the filter to include
startup
+ and lifecycle lines, which have no trace.
+3. Expand a line and follow its **Trace** link to open the request in Tempo.
### View Metrics (Prometheus) ###
1. Select **Prometheus** as the data source
2. Example queries:
- # HTTP request rate
- rate(http_server_requests_seconds_count[5m])
-
- # JVM memory usage
- jvm_memory_used_bytes
+ # HTTP request rate, by method and status
+ sum by (http_request_method, http_response_status_code)
(rate(http_server_request_duration_seconds_count{job="solr-mcp"}[5m]))
# Request latency (p99)
- histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
+ histogram_quantile(0.99, sum by (le)
(rate(http_server_request_duration_seconds_bucket{job="solr-mcp"}[5m])))
+
+ These are the OpenTelemetry HTTP server metrics, the only application
metrics exported
+ over OTLP. Micrometer's names (`http_server_requests_seconds_*`, `jvm_*`)
return nothing
+ here; read them from `/actuator/prometheus` instead.
+
+### Pivoting Between the Three ###
+
+The fastest way through all three signals for one request:
+
+1. Find the trace in **Tempo** (TraceQL query above) and open it.
+2. Each span carries a **Logs for this span** link—click it to jump to
the
+ matching lines in **Loki**, filtered by trace ID. This works because the
OTEL logback
+ appender (`logback-spring.xml`) tags every log line with the active trace
and span ID.
+ The link filters on the trace, so it finds the same lines from any span in
it.
+3. The link comes up empty for a trace that logged nothing, which is every
successful
+ tool call: the services log only on failure. The `check-health` call above
is the one
+ to follow; its `WARN` sits under the `collection-service#check-health` span.
+4. From a log line, the **Trace** link takes you back to its trace.
+5. **Metrics** aren't per-request the same way—there's no single
span/log ↔
+ metric-sample link—but the PromQL queries above will show the
aggregate effect
+ (e.g. a rise in `http_server_request_duration_seconds_count`) of whatever
activity you
+ just generated.
***
@@ -100,6 +191,21 @@ curl http://localhost:8080/actuator/loggers # Logger
levels
***
+## Troubleshooting ##
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| No traces/metrics/logs show up in Grafana at all | LGTM was never
started—`bootRun` does not start it in either mode | `docker compose up
-d lgtm` |
+| Tempo finds nothing right after the calls | Traces take up to a minute to
become searchable; metrics are exported once a minute | Wait and re-run the
query |
+| **Logs for this span** is empty | The request logged nothing; successful
tool calls never do | Expected; follow a failing call such as `check-health` on
a missing collection |
+| `jvm_*` or `http_server_requests_seconds_*` returns nothing in Grafana |
Those are Micrometer metrics, served only at `/actuator/prometheus` | Query
`http_server_request_duration_seconds_*` in Grafana, or curl the actuator |
+| Traces appear but stop after a restart | The `otel-lgtm` container has no
persistent volume | Expected; re-run your workload after restarting `lgtm` |
+| `otel.exporter.otlp.endpoint` connection refused | Running the server
outside the `search` Docker network (e.g. inside its own container) while LGTM
is on the host | Point `OTEL_TRACES_URL` at a reachable host, or join the same
Docker network |
+| Traces are sparse or missing under load |
`management.tracing.sampling.probability` is below 1.0 | Raise it for the
session you're debugging; keep it low in production |
+| No data in **STDIO** mode | Tracing/metrics export is an HTTP-mode
feature—STDIO has no servlet layer to instrument | Run `PROFILES=http
./gradlew bootRun` instead |
+
+***
+
## Production Configuration ##
For production, reduce the sampling rate and configure the OTLP endpoint for
your collector:
diff --git a/docs/site/content/pages/mcp/observability.md
b/docs/site/content/pages/mcp/observability.md
index 46da40be..ba8015f2 100644
--- a/docs/site/content/pages/mcp/observability.md
+++ b/docs/site/content/pages/mcp/observability.md
@@ -9,11 +9,17 @@ When running in **HTTP mode**, the Solr MCP Server exports
telemetry data via Op
| Signal | Backend | What it shows |
|--------|---------|---------------|
-| **Traces** | Tempo | Distributed traces for every MCP tool invocation, Solr
query, and HTTP request |
-| **Metrics** | Mimir/Prometheus | JVM stats, HTTP request rates, Solr query
latencies, cache hit ratios |
-| **Logs** | Loki | Structured application logs correlated with trace IDs |
+| **Traces** | Tempo | A trace per HTTP request, with a span for the MCP tool
it invoked |
+| **Metrics** | Mimir/Prometheus | HTTP server request count and duration |
+| **Logs** | Loki | Application logs, each tagged with the trace and span it
was written under |
-Every MCP tool invocation creates a trace span: search, indexing (JSON, CSV,
XML), collection operations (list, stats, health, create), and schema
retrieval. All incoming HTTP requests and outgoing Solr calls are automatically
traced.
+Every MCP tool invocation creates a span named after the service and tool,
such as
+`search-service#search` or `collection-service#check-health`, inside the trace
of the HTTP
+request that carried it. The call to Solr is not a separate span; its time is
part of the
+tool span.
+
+JVM, Tomcat and other Micrometer metrics are not exported over OTLP. They are
served for
+scraping at `/actuator/prometheus` (see **Actuator Endpoints** below).
***
@@ -35,6 +41,28 @@ This starts:
| OTLP gRPC | localhost:4317 | Trace/metric/log ingestion (gRPC) |
| OTLP HTTP | localhost:4318 | Trace/metric/log ingestion (HTTP) |
+**LGTM is never auto-started.** The `lgtm` service carries
+`org.springframework.boot.ignore: "true"` in `compose.yaml`, which opts it out
of Spring
+Boot's Docker Compose lifecycle management. This is independent of transport
mode: even
+in HTTP mode, where `PROFILES=http ./gradlew bootRun` auto-starts (and stops)
the `solr`
+and `zoo` services, LGTM is untouched. Start it explicitly, either with the
+`docker compose up -d` above (which starts everything) or, if `solr`/`zoo` are
already
+running via `bootRun`'s auto-start, on its own:
+
+```bash
+docker compose up -d lgtm
+```
+
+Because it's Boot-ignored, `bootRun` won't stop it either—bring it down
by hand
+when you're done:
+
+```bash
+docker compose stop lgtm
+```
+
+The `grafana/otel-lgtm` container stores everything in memory with no
persistent
+volume, so a restart discards all traces, metrics, and logs.
+
### Run the Server with Observability ###
```bash
@@ -49,6 +77,46 @@ otel.exporter.otlp.endpoint=http://localhost:4317
otel.exporter.otlp.protocol=grpc
```
+`otel.exporter.otlp.endpoint` is the endpoint for all three signals, so
`OTEL_TRACES_URL`
+moves logs and metrics too, despite its name. When the server runs in a
container and LGTM
+on the host, `localhost` is the container itself; set
+`OTEL_TRACES_URL=http://host.docker.internal:4317` (plus
+`--add-host=host.docker.internal:host-gateway` on Linux).
+
+### Generate Some Activity ###
+
+Grafana has nothing to show until at least one tool call runs. Any MCP client
works
+(the [Quick Start](/mcp/quick-start.html) prompts are enough), or call the HTTP
+endpoint directly. `HTTP_SECURITY_ENABLED=false` skips the OAuth2 setup for
this local
+check—see [Security](/mcp/security.html) to keep it on:
+
+```bash
+HTTP_SECURITY_ENABLED=false PROFILES=http ./gradlew bootRun
+```
+
+```bash
+curl -s -X POST http://localhost:8080/mcp \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json, text/event-stream" \
+ -d
'{"jsonrpc":"2.0","method":"tools/call","id":1,"params":{"name":"list-collections","arguments":{}}}'
+```
+
+Run it a few times—each call is one trace and one more request in the
metrics.
+
+A successful call writes no log lines, so it has nothing to show in Loki. To
see a trace
+with a log attached, make one call that fails; a health check on a collection
that does not
+exist logs a `WARN` inside the request:
+
+```bash
+curl -s -X POST http://localhost:8080/mcp \
+ -H "Content-Type: application/json" \
+ -H "Accept: application/json, text/event-stream" \
+ -d
'{"jsonrpc":"2.0","method":"tools/call","id":2,"params":{"name":"check-health","arguments":{"collection":"no-such-collection"}}}'
+```
+
+Traces take up to a minute to become searchable in Tempo, and metrics are
exported once
+a minute, so an empty Grafana right after the calls is expected.
+
***
## Grafana ##
@@ -62,30 +130,53 @@ Open [http://localhost:3000](http://localhost:3000) and
click **Explore** in the
{.service.name="solr-mcp"}
-3. Click on a trace to see the span waterfall—each MCP tool invocation,
Solr query, and HTTP request is a separate span
+3. Click on an `http post /mcp` trace to see the span waterfall: the security
filter
+ chain, then one span for the tool (`search-service#search`,
+ `collection-service#check-health`, …) with its duration
### View Logs (Loki) ###
1. Select **Loki** as the data source
2. Use LogQL to search:
- {service_name="solr-mcp"} |= "search"
+ {service_name="solr-mcp"} | trace_id != ""
-3. Logs are automatically correlated with trace IDs—click a log line to
jump to its trace
+ That keeps only lines written during a request; drop the filter to include
startup
+ and lifecycle lines, which have no trace.
+3. Expand a line and follow its **Trace** link to open the request in Tempo.
### View Metrics (Prometheus) ###
1. Select **Prometheus** as the data source
2. Example queries:
- # HTTP request rate
- rate(http_server_requests_seconds_count[5m])
-
- # JVM memory usage
- jvm_memory_used_bytes
+ # HTTP request rate, by method and status
+ sum by (http_request_method, http_response_status_code)
(rate(http_server_request_duration_seconds_count{job="solr-mcp"}[5m]))
# Request latency (p99)
- histogram_quantile(0.99, rate(http_server_requests_seconds_bucket[5m]))
+ histogram_quantile(0.99, sum by (le)
(rate(http_server_request_duration_seconds_bucket{job="solr-mcp"}[5m])))
+
+ These are the OpenTelemetry HTTP server metrics, the only application
metrics exported
+ over OTLP. Micrometer's names (`http_server_requests_seconds_*`, `jvm_*`)
return nothing
+ here; read them from `/actuator/prometheus` instead.
+
+### Pivoting Between the Three ###
+
+The fastest way through all three signals for one request:
+
+1. Find the trace in **Tempo** (TraceQL query above) and open it.
+2. Each span carries a **Logs for this span** link—click it to jump to
the
+ matching lines in **Loki**, filtered by trace ID. This works because the
OTEL logback
+ appender (`logback-spring.xml`) tags every log line with the active trace
and span ID.
+ The link filters on the trace, so it finds the same lines from any span in
it.
+3. The link comes up empty for a trace that logged nothing, which is every
successful
+ tool call: the services log only on failure. The `check-health` call above
is the one
+ to follow; its `WARN` sits under the `collection-service#check-health` span.
+4. From a log line, the **Trace** link takes you back to its trace.
+5. **Metrics** aren't per-request the same way—there's no single
span/log ↔
+ metric-sample link—but the PromQL queries above will show the
aggregate effect
+ (e.g. a rise in `http_server_request_duration_seconds_count`) of whatever
activity you
+ just generated.
***
@@ -103,6 +194,21 @@ curl http://localhost:8080/actuator/loggers # Logger
levels
***
+## Troubleshooting ##
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| No traces/metrics/logs show up in Grafana at all | LGTM was never
started—`bootRun` does not start it in either mode | `docker compose up
-d lgtm` |
+| Tempo finds nothing right after the calls | Traces take up to a minute to
become searchable; metrics are exported once a minute | Wait and re-run the
query |
+| **Logs for this span** is empty | The request logged nothing; successful
tool calls never do | Expected; follow a failing call such as `check-health` on
a missing collection |
+| `jvm_*` or `http_server_requests_seconds_*` returns nothing in Grafana |
Those are Micrometer metrics, served only at `/actuator/prometheus` | Query
`http_server_request_duration_seconds_*` in Grafana, or curl the actuator |
+| Traces appear but stop after a restart | The `otel-lgtm` container has no
persistent volume | Expected; re-run your workload after restarting `lgtm` |
+| `otel.exporter.otlp.endpoint` connection refused | Running the server
outside the `search` Docker network (e.g. inside its own container) while LGTM
is on the host | Point `OTEL_TRACES_URL` at a reachable host, or join the same
Docker network |
+| Traces are sparse or missing under load |
`management.tracing.sampling.probability` is below 1.0 | Raise it for the
session you're debugging; keep it low in production |
+| No data in **STDIO** mode | Tracing/metrics export is an HTTP-mode
feature—STDIO has no servlet layer to instrument | Run `PROFILES=http
./gradlew bootRun` instead |
+
+***
+
## Production Configuration ##
For production, reduce the sampling rate and configure the OTLP endpoint for
your collector:
