alamb opened a new issue, #9415: URL: https://github.com/apache/arrow-datafusion/issues/9415
### Is your feature request related to a problem or challenge? "Tracing" is a visualization technique for understanding how potentially concurrent operations happen, and is common in distributed systems. You can also visualize DataFusion executions using traces. Here is an example trace we have at InfluxData that is integrated into the rest of our system and visualized using https://www.jaegertracing.io/  This visualization shows when each operator started and stopped (and the operators are annotated with how much CPU time is spent, etc). These spans are integrated into the overall trace of a request through our cloud service, which allows us to understand where a request's time is spent, both across services as well as within our DataFusion based engine. For more background on tracing, this blog seems to give a reasonable overview: https://signoz.io/blog/distributed-tracing-span/ ### Describe the solution you'd like I would like to make it easy for people to add DataFusion ExecutionPlan level tracing to their systems as well. This came up twice recently, so I wanted to get it filed into a ticket ### Describe alternatives you've considered _No response_ ### Additional context @simonvandel noted in [Discord](https://discord.com/channels/885562378132000778/1166447479609376850/1213154589537341510) > I have been playing a bit with https://github.com/wolfpld/tracy and https://docs.rs/tracing-tracy/latest/tracing_tracy/ and thought it would be nice to have some visibility into ExecutionPlans being executed. It also came up in slack: https://the-asf.slack.com/archives/C04RJ0C85UZ/p1709051125059619 > But I want to see traces from which I want to calculate how much time spent in executing the queries and how much time got spent into data transfer from a target storage system, It would be good, if you can guide me how can I get these metrics -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
