Hi Brian, Thank you for bringing this up. I see value here for NiFi deployments in larger (enterprise) environments, where NiFi runs alongside other integration and data management solutions that also support (or plan to support) OpenTelemetry. Moving data between on-prem legacy applications and cloud data lakes is a good example. It currently takes a lot of work to correlate NiFi logging with that of Azure Data Factory (and friends), so in practice it often doesn't get done.
I don't have the technical depth to really talk implementation, but from a functional perspective a generic wrapper that leverages existing flowfile/provenance data would make sense and I only would want the wrapper to look for an "opentelemetry.label" attribute to easily identify a specific flow and correlate it with the same "flow" in other systems. Regards, Isha -----Oorspronkelijk bericht----- Van: Brian Putt <puttbr...@gmail.com> Verzonden: woensdag 15 juni 2022 21:57 Aan: dev@nifi.apache.org Onderwerp: [DISCUSS] Distributed tracing using OpenTelemetry Hello Apache NiFi, I'd like to discuss implementing NIFI-10110 which adds OpenTelemetry integration into NiFi. Tracing will provide a way to identify bottlenecks within various flows and propagate trace information to downstream systems (whether they're another NiFi cluster or otherwise). I wanted to get the community's feedback to see if this makes sense and if it's something that you'd find valuable. If so, would anyone be interested in discussing implementation details? We'd certainly be willing to implement this feature, but feel it will need feedback from the community as we add the integration. Some key areas around implementation will involve where it makes sense to add the OpenTelemetry wrapper because ideally, processors get tracing 'for free' and this then leads to other questions as to sampling and how to expose flowfile attributes into span tags that get emitted. OpenTelemetry offers a Java library that's licensed under Apache 2.
