Hi everyone, Following up on the public feature request <https://github.com/apache/beam/issues/33176> and discussions I've decided to try to implement open telemetry tracing into Beam SDK, starting with Java SDK and two IOs - Kafka and PubSub.
I've been working on this and the results can be reviewed in this PR <https://github.com/apache/beam/pull/34544>. Where it is helpful - user can track latency for sampled messages across multiple services e.g: pubsub publisher --> dataflow with bigtable enrichment --> kafkaIO --> kafka consumer Support in this PR is limited to: - Trace context propagation from/to KafkaIO and from PubSubIO and in the future to PubSubIO as well as long as both IOs are running in the same stage. - Cloud Bigtable's client and other libraries with opencensus tracing are automatically supported - Default support for Cloud Trace when Dataflow is used What is not working: - Trace context is not propagated yet beyond the stage/thread in which it was initiated - Other SDKs are not yet supported Looking for your feedback, let me know if you like it or you find this implementation limiting your use cases. Radek
