dpol1 opened a new pull request, #9154: URL: https://github.com/apache/storm/pull/9154
## What is the purpose of the change Part of #9016. With `topology.tracing.enabled: true`, Storm carries an OpenTelemetry trace context with each tuple, so a trace follows a tuple tree across bolts, threads and workers; a join starts a new trace linked to its inputs. Spans that bolt code or instrumented clients create inside `execute()` belong to the same trace. Tracing is off by default; when off, Storm creates no spans and tuples serialize to the same bytes as today. Storm only calls the OpenTelemetry API. The SDK registered as the global instance, usually the Java agent on the workers, samples and exports the spans. Each spout emit starts a trace, each `execute()` on a traced tuple runs in a child span, and an emit takes its parent from its anchors, on any thread. For reliable spouts, a span under the root, started and ended at once, records the ack, fail or timeout. `docs/Tracing.md` has the details and the limits. The context travels after the tuple values, inside the compressed frame (26 bytes without tracestate), so the current reader, which stops after the values, ignores it. `opentelemetry-api` 1.66.0 becomes a compile dependency of storm-client, managed through `opentelemetry-bom`. The only new public method is `TupleUtils.traceContext(Tuple)`. Keeping every failure at 100% sampling is out of scope; I'll open a separate issue for it. ### Cost On a topology whose bolts do no work, with tracing off I saw no regression against master beyond the variation between runs (no agent, two runs each). With tracing on, against the Java agent with tracing off, an async bolt on one worker loses about 29% of its throughput at ratio 0, 34% at 1% and 57% at 100%. Half to two thirds of the ratio-0 cost is the agent's API bridge (open-telemetry/opentelemetry-java-instrumentation#20340). With the agent's default settings its executors instrumentation adds more: ratio 0 then loses 67%; `docs/Tracing.md` says when to turn it off. At 100% the SDK exports 5–11% of the spans and drops the rest. <details><summary>Numbers</summary> storm-perf `ConstSpout` → `IdBolt` → `DevNullBolt`, LocalCluster on a laptop, agent 2.31.1 with its executors instrumentation off, two runs per cell, before the rebase on current master. Acks/s counted at the spout, whole-JVM CPU per ack. | | async, 1 worker | async, 2 workers | sync, 1 worker | |---|---|---|---| | agent, tracing off | 530–598 k/s, 7.3–7.5 µs | 213–215 k/s, 19.1–19.4 µs | 250–258 k/s, 10.5–10.8 µs | | ratio 0 | 393–406 k/s, 10.8–11.2 µs | 171–174 k/s, 26.0–27.3 µs | 215–221 k/s, 13.4–13.7 µs | | 1% | 365–379 k/s, 11.7–12.2 µs | 167–168 k/s, 27.0–28.0 µs | – | | 100% | 205–275 k/s, 17.0–19.6 µs | 137–175 k/s, 33.2–37.4 µs | 140–160 k/s, 20.3–22.8 µs | Not measured: latency, several hosts, a real backend, failed or timed-out tuples. </details> ## How was the change tested - `TopologyTracingTest` (new, 15 tests) runs topologies on a two-worker LocalCluster and reads the spans through `OpenTelemetryExtension`: root, execute, emit and outcome spans, joins, unanchored and delayed emits, unsampled contexts, tracing off. - `KryoTupleSerializerDeserializerTest` (6 new tests): round trips, unchanged bytes without a context, the old reader on new bytes, truncated and unknown extensions. - storm-client, storm-server, storm-webapp and storm-core pass with the CI test command, plus RAT (storm-core without `-Pnative`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
