Claus Ibsen created CAMEL-23708:
-----------------------------------
Summary: OpenTelemetry2 - Missing RECEIVED spans for routes
consuming from stub/seda in nested async paths
Key: CAMEL-23708
URL: https://issues.apache.org/jira/browse/CAMEL-23708
Project: Camel
Issue Type: Bug
Components: camel-opentelemetry
Reporter: Claus Ibsen
When using the stub component to simulate Kafka (e.g. with {{--observe}} / dev
profile), route consumer RECEIVED spans are not exported for routes that
consume from stub endpoints when the producer side is inside a multicast on a
seda consumer thread.
h3. Steps to reproduce
Run the route-topology example with OpenTelemetry dev mode:
{code}
camel run route-topology.camel.yaml --observe
{code}
The example has this flow:
{code}
timer -> order-generator -> direct:process-order -> process-order ->
kafka:orders
-> order-dispatcher (multicast)
-> kafka:fulfillment -> fulfillment route -> kafka:warehouse-shipments
-> kafka:notifications -> notification route -> kafka:email-outbox
{code}
h3. Expected behavior
Each trace should have 27 spans including EVENT_RECEIVED spans for all 6 routes:
- timer (order-generator) - OK
- direct:process-order (process-order) - OK
- direct:validate-order (validate-order) - OK
- kafka:orders via stub (order-dispatcher) - OK
- kafka:fulfillment via stub (fulfillment) - MISSING
- kafka:notifications via stub (notification) - MISSING
h3. Actual behavior
Each trace has exactly 25 spans. The EVENT_RECEIVED spans for the fulfillment
and notification routes are never exported to the DevSpanExporter.
The RECEIVED spans ARE created (processor spans log4, to7, log5, to8 correctly
reference them as parentSpanId), but {{span.end()}} is apparently never called,
so {{SimpleSpanProcessor.onEnd()}} never fires and they never reach the
exporter.
h3. Analysis
The issue is specific to nested async routing: the fulfillment and notification
routes consume from stub endpoints whose producers are inside a multicast
within the order-dispatcher route, which is itself on a seda consumer thread.
The {{onExchangeDone}} route policy callback ({{TracingRoutePolicy}} in
{{camel-telemetry}}) does not fire for these exchanges, so the RECEIVED span is
started but never ended/exported.
The kafka:orders RECEIVED span (order-dispatcher route) works correctly - this
is a direct stub producer to seda consumer path without the nested multicast.
Key code path:
- {{camel-telemetry}}: {{Tracer.TracingRoutePolicy.onExchangeDone()}} ->
{{endEventSpan()}} -> {{spanLifecycleManager.deactivate(span)}} ->
{{otelSpan.end()}}
- {{camel-seda}}: {{SedaProducer.process()}} (InOnly path) ->
{{addToQueue(exchange, true)}} -> async consumer pickup
- {{camel-stub}}: extends seda, same behavior
h3. Impact
- Orphan spans in trace visualization (Jaeger, TUI waterfall) - processor spans
reference non-existent parent
- Missing routeId propagation for orphaned spans (dev console cannot walk
parent chain)
- Affects any scenario with stub/seda consumers inside multicast or other async
EIPs
h3. How this was discovered
Using the TUI MCP {{tui_get_spans}} tool to fetch raw span JSON data from a
running Camel TUI session, analyzing parent chain integrity across 20
consecutive traces (500 spans). Every trace showed the identical orphan pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
