Hi, Monitoring in NiFi is one of my favorite topics and I do think OpenTelemetry would be of great value. There is never too much observability in a software like NiFi when it comes to integrating anything with anything. I'm more than happy to be part of the discussion in terms of requirements/implementation, etc. Getting better insights from the provenance data of NiFi is probably where I'd start.
Pierre Le jeu. 16 juin 2022 à 10:44, Isha Lamboo <isha.lam...@virtualsciences.nl> a écrit : > Hi Brian, > > Thank you for bringing this up. I see value here for NiFi deployments in > larger (enterprise) environments, where NiFi runs alongside other > integration and data management solutions that also support (or plan to > support) OpenTelemetry. Moving data between on-prem legacy applications and > cloud data lakes is a good example. It currently takes a lot of work to > correlate NiFi logging with that of Azure Data Factory (and friends), so in > practice it often doesn't get done. > > I don't have the technical depth to really talk implementation, but from a > functional perspective a generic wrapper that leverages existing > flowfile/provenance data would make sense and I only would want the wrapper > to look for an "opentelemetry.label" attribute to easily identify a > specific flow and correlate it with the same "flow" in other systems. > > Regards, > > Isha > > -----Oorspronkelijk bericht----- > Van: Brian Putt <puttbr...@gmail.com> > Verzonden: woensdag 15 juni 2022 21:57 > Aan: dev@nifi.apache.org > Onderwerp: [DISCUSS] Distributed tracing using OpenTelemetry > > Hello Apache NiFi, > > I'd like to discuss implementing NIFI-10110 which adds OpenTelemetry > integration into NiFi. Tracing will provide a way to identify bottlenecks > within various flows and propagate trace information to downstream systems > (whether they're another NiFi cluster or otherwise). > > I wanted to get the community's feedback to see if this makes sense and if > it's something that you'd find valuable. If so, would anyone be interested > in discussing implementation details? We'd certainly be willing to > implement this feature, but feel it will need feedback from the community > as we add the integration. > > Some key areas around implementation will involve where it makes sense to > add the OpenTelemetry wrapper because ideally, processors get tracing 'for > free' and this then leads to other questions as to sampling and how to > expose flowfile attributes into span tags that get emitted. > > OpenTelemetry offers a Java library that's licensed under Apache 2. >
