dpol1 opened a new issue, #9155: URL: https://github.com/apache/storm/issues/9155
With tracing at 100% (#9154), the OpenTelemetry SDK's batch span processor can't keep up on a fast topology: in my runs it exported 5–11% of the spans and dropped the rest, and tuning it raised that only to 13%. It drops spans whatever the outcome, so failures and timeouts go too. @GGraziadei suggested sending the spans down a Storm stream to a collector bolt that keeps failures, timeouts and a sample of the rest. The bolt would also need to know when the work for a tuple ends in an async bolt; otherwise it can't tell where a timed-out tuple stopped. Open questions: what the bolt holds and for how long, what it does under overload, and whether reducing near each worker beats one central bolt. I'd prototype it first and measure it against OTLP. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
