dpol1 commented on PR #9154: URL: https://github.com/apache/storm/pull/9154#issuecomment-6083158451
Thanks @GGraziadei. Before touching storm-client I tried something smaller in the tracer: when the sampler drops the root span of a spout emit, the tuple now carries no trace context, so Storm skips `startExecute`, `boltFail` and `spoutOutcome` for that tree. On my laptop this removes most of the throughput loss. On one worker, ratio 0 goes from about 40% below tracing off on the current head to 12% or less, so it's not yet within the noise you were hoping for. On two workers it's level with tracing off (head is 21 to 26% below). The numbers are below, please take them as a rough guide. It does change the traces in two ways, and I'd like your opinion on them. A client instrumented by the agent and called in `execute()` for such a tuple now starts its own trace, decided by the root sampler, so at 1% some of these calls show up as sampled traces with no link to the tuple. An emit anchored to a traced and an untraced tuple now continues the traced tree, with no link to the untraced one. Would that be ok as the basis for this optimization, with RECORD_ONLY roots still passing their context on? I think your storm-client fast path would have the same two effects, so I'd rather agree on this first and measure any storm-client change on its own afterwards. I haven't pushed it yet, and I'm happy to run other setups if useful. <details> <summary>Numbers (laptop)</summary> Laptop, LocalCluster with all workers in one JVM, agent 2.31.1 with its executors instrumentation off, the async variant from the Cost section of the PR description. IPv6 is disabled on this machine, so ZooKeeper clients retried `[::1]` in every run, sometimes during a measurement window. It affects all configurations. It also limits how precisely I can attribute the remaining cost. Head is 42e1430cc. "This change" swaps only the `storm-opentelemetry` jar. Throughput with the spout at full speed, against "agent, tracing off". One worker: four rounds, each against its own control run, because the control drifted between rounds. Two workers: two runs. | | 1 worker | 2 workers | |---|---|---| | ratio 0, head | -46 to -37% | -26 to -21% | | ratio 0, this change | -12 to -4% in three rounds, -1 to +6% in one | -0.3 to +3.6% | | 1%, head | -60 to -44% | -25 to -23% | | 1%, this change | -23 to -4% | -5 to -1% | The head range at ratio 0 on one worker includes a run that the harness rejected for its exit code at shutdown, after both windows. Without it the range is -46 to -40%. In round 2, head at 1% ran at 223-232k tuples/s against 313-336k in the other rounds, and I haven't looked into why. Without that round the range is -49 to -44%. At 1% on one worker, three of the four runs with this change dropped spans from a full export queue, up to 2,400 per run. Head dropped none. This change acked 1.5 to 2.3 times as many tuples per second as head in the same round, so the SDK received more spans per second. Complete latency at 50k tuples/s on one worker, p99 against tracing off, which itself varies by ±0.14 ms. Ratio 0, head +0.83 to +1.05 ms and this change -0.15 to +0.28 ms. 1%, head +0.18 to +1.03 ms and this change -0.05 to +0.19 ms. At the same rate, CPU per ack at ratio 0 is 44.5 to 55.2 µs with this change, against 44.5 to 47.5 µs on head and 40.1 to 43.6 µs with tracing off. The upper end with this change is above head's range, and I haven't looked into why. The SDK still starts one dropped root span per spout emit, and each bolt emit still calls the tracer, which returns at once. Your storm-client changes, sampling at the spout and skipping untraced emits, would remove those calls. I haven't profiled where the remaining cost goes. Tuples of dropped trees now cross workers without trace bytes, so encode and decode do not run for them. With a parent-based sampler, unsampled contexts still cross workers for trees under a RECORD_ONLY root, and for tuples of an emit whose anchors come from different traces, when the sampler drops the new root that emit starts. Both cases are intended, so decode keeps unsampled contexts instead of returning null when the sampled bit is clear. Next to measure: the 50% and 75% load points with longer runs, a network between workers, real client libraries and a RECORD_ONLY sampler. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
