dttung2905 opened a new issue, #17115: URL: https://github.com/apache/iceberg/issues/17115
### Feature Request / Improvement ### Summary Add optional OpenTelemetry (OTEL) distributed tracing to the Iceberg Kafka Connect sink connector. When enabled, the connector should create spans for the two main pipeline stages and continue upstream traces by extracting the W3C `traceparent` header from Kafka Connect record headers. This is **span tracing** for the Connect sink pipeline — distinct from existing OTEL **metrics** work in the core library ([#16169](https://github.com/apache/iceberg/issues/16169)) and JMX **metrics** for the commit pipeline ([#17025](https://github.com/apache/iceberg/pull/17025)). ### Motivation / use case End-to-end CDC pipelines commonly look like: **Source DB → Debezium (or similar) → Kafka → Iceberg Kafka Connect sink → Iceberg tables** Debezium already supports distributed tracing via `ActivateTracingSpan` (standard Connect sources) and connector-native tracing (e.g. [debezium-connector-cassandra#173](https://github.com/debezium/debezium-connector-cassandra/pull/173)), injecting `traceparent` into Kafka/Connect record headers. Today the Iceberg sink does not consume that context, so traces break at the sink boundary and operators cannot answer: - What is the lag from source CDC event to Iceberg ingest? - What is the lag from ingest (buffer) to catalog commit (durability)? - Which records/commits failed, and with what error? The Iceberg Kafka Connect sink also has a **two-phase** write path (buffer to Parquet/ORC files on `put()`, then async coordinator commit to the catalog). Without connector-level spans, generic worker instrumentation alone cannot distinguish ingest latency from commit latency. ### Query engine None ### Willingness to contribute - [ ] I can contribute this improvement/feature independently - [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
